In 1995, there were about 1,000 books with the word “terrorism” in their titles. By 2011, that had multiplied tenfold and the proliferation continues this autumn, with an onslaught of authors focusing on the spreading stain of violence in the Middle East. Little wonder that many of us have succumbed to “threat fatigue”.

Contrary to the warnings of the intelligence services, however, this may not be a bad thing. While religiously motivated terrorist attacks in the West have constituted personal tragedies, in terms of scale, they have been mosquito pricks not sledgehammer blows.

A survey of the years from September 11, 2001, suggests not so much the great success of terrorist groups in the land of the infidel as its abject failure. Mass-casualty attacks have been few and far between, although it seems reasonable to expect low-level, low-tech, “lone wolf” attacks to continue sporadically, such as the recent train attack in France.

The fact is that the Muslim world has been by far the greatest casualty, with sectarian conflict between Sunnis and Shia the defining fault line of terrorist activity right across North Africa and the Middle East.

Jason Burke, a British journalist, is the author of a trio of related books (Al Qaeda, On the Road to Kandahar and The 9/11 Wars), with experience from two decades of front-line reporting from across the Muslim world. He knows only too well that this region is bearing the brunt of extremist violence that is tearing up national borders and turning old realities on their head.

His analysis has the weight of considerable on-the-ground experience, rather than the lofty theorising of an armchair academic. The New Threat from Islamic Militancy is a scrupulously researched study based on calm and cogent analysis, compellingly told.

While he is addressing policymakers in Europe and Washington, Burke’s book is also aimed squarely at the Muslim world, not least because it deconstructs the very parochial western perspective of the threat. Hence the telling observation in the introduction that few in Europe or the United States are aware of the second-most murderous terrorist attack of recent times: the multiple suicide bombings perpetrated by an earlier incarnation of ISIL that killed more than 800 Yazidis in northern Iraq in 2007.

Burke’s focus is on Al Qaeda, ISIL (which Burke, writing for an English-speaking audience, calls Islamic State), their affiliates and then the much broader movement of fellow travellers and sympathisers united by their anti-western, anti-Semitic and homophobic rhetoric learnt from gangs, on streets, in prisons, mosques and community centres worldwide.

Understanding the Muslim world and its history is essential to demystify the contemporary crop of jihadists and their world view. “The West has enormous difficulty comprehending the sense of loss in the Muslim world,” says Burke. “We need to keep the natural revulsion, but also look coldly and analytically as to why some people might see IS as the panacea for all the ills of the umma.

“That is fantastically important. You have to understand the rule of the Ottomans, the Mughals, the golden age of the Abbasids, Umayyads and Almohads – this great loss of the power and glory of Islam. Without understanding that, it’s very difficult to understand their caliphate project and the demand for respect and dignity that seems so at odds with the systematic abuse of human rights.”

For much, if not most, of the past 14 centuries, the ascendancy in global power and prestige has belonged to the Islamic world, emphatically not the West, as today’s jihadists like to point out. The early history of Islam, one of their defining sources of inspiration, could hardly be further removed from its Jewish and Christian counterparts.

“While for Jews the collective memory of the earliest experience of believers is repeated exile, and for Christians it is persecution, for Sunni Muslims it is one of the most successful military and political campaigns in history,” writes Burke. Within decades of the Prophet Mohammed’s death, the Islamic Empire stretched from the shores of the Atlantic in the west to the mountains of Central Asia in the east. Yet, the golden years of Islamic civilisation, under first the Umayyads (661-750) then more progressively with the pluralist, intellectually inquisitive Abbasids (750-1258), completely shame the sterile and reductive vision of education offered by ISIL, which has excised foreign languages, literature, art, music, mathematics, social sciences and national history from the curriculum.

Burke rightly emphasises the lack of religious knowledge of the average jihadist. Selective quotations from holy texts are nothing new in any religion and it’s no different with Islam. Contrast, for example, some of the early Quranic verses (“There is no compulsion in religion, right guidance has been distinguished from error”), delivered to the Prophet when the Muslim community was small and weak, with the later, more belligerent “sword verses” received when the Prophet Mohammed was approaching the height of his power (“Kill the polytheists wherever you find them, arrest them, imprison them, besiege them and lie in wait for them”). Early Muslim dynasties held the later verses on jihad to abrogate the earlier, a view wholeheartedly accepted by contemporary Islamists.

Burke is at his best when he unpicks lazy, commonly held opinion. He condemns British home secretary Theresa May for her “extraordinary and misleading” statement about ISIL in 2014 that “the threat we face is now more dangerous than at any time before or since 9/11”, a patent absurdity that does a disservice to the antiterrorism cause. He is also correct to brand the “global war on terror” a “monumentally misconceived strategy” that is partly to blame for the spread of religious extremism, militancy and terrorism during the past decade.

Much attention has been given to ISIL’s grasp of digital communications, an aspect of its operations that makes Ayman Al Zawahiri’s Al Qaeda look distinctly analogue. Yet, even here, the common obsession with the media “spectacular”, shared by both organisations, is only treading familiar ground. As the German anarchist Johannes Most wrote in his tract Philosophy of the Bomb, published in 1880: “Outrageous violence will seize the imagination of the public and awaken its audience to political issues.” Adept it may be, but ISIL has no monopoly on understanding – or manipulating – the media.

Burke is a good guide to the key texts and figures within the militant movement, from the Egyptian Muslim Brotherhood member Sayyid Qutb, author of Milestones (1964), a foundational text for modern militancy; to another jihad-preaching Egyptian, Abdel Salam Faraj; and Abdullah Azzam, the Palestinian author of the clunky pamphlet Defending the Lands of the Muslims is Each Man’s Most Important Duty.

The yawning gap between rhetoric and reality can be discerned in Qutb’s writings at the outset. “After annihilating the tyrannical force, whether political or a racial tyranny, or domination of one class over the other within the same race, Islam establishes a new social and economic political system, in which all men enjoy real freedom,” he wrote. In fact, freedom has palpably failed to emerge under any such project – ISIL being merely the latest chapter in a history of severe repression.

When it comes to apportioning blame for the wave of violence and intolerance, many point the finger at the spread of Salafism. “None of this militancy was an historical inevitability,” Burke says. “One of the strands that has now set in is the spread of a rigorous, conservative and intolerant observance, which was once restricted to a minority but is now much more widely followed. The impact of these austere doctrines is that even if they are not violent themselves they encourage a view of other Muslims and other faiths which is at the very least intolerant if not violent.”

Foreign bogeymen and conspiracy theories are often attractive ways of explaining political reversals, yet, sometimes one need not look so far from home to discover the real reasons for change. As the Syrian historian and journalist Sami Moubayed emphasises in Under the Black Flag: At the Frontier of the New Jihad, Syria and Iraq have had one thing in common for decades: the Baath party. Once seen as the answer to the challenges facing the postcolonial order in the Middle East, by the turn of the century, the Baath in Assad’s Syria and Saddam’s Iraq had become synonymous with “cronyism, nepotism, influence peddling, collusion, extortion and bribery”.

When the state fails its people for several generations, is it so surprising that many Iraqis and Syrians hope for something better from – and in desperation pledge allegiance to – the latest project promising a happier future for everyone? One by one Arab nationalism, pan-Arabism, Nasserism, socialism and secularism have proved inadequate to meet the basic needs and expectations of most Arab populations, from peace, education and health care to employment and economic growth. ISIL, in other words, is just the latest taxi at the rank.

How to judge the impact of ISIL, then, this latest Islamist revival movement? As Burke notes, even within the Islamic world, the expansion of the so-called caliphate, with its self-*declared “emirates” or governorates emerging from Saudi Arabia and Egypt to Libya, Yemen and Algeria, has failed to usher in a series of “miniature Islamic states” or anything close to them. Again, there is the vast chasm between aspiration and reality, a real-*world reflection of the delusion inherent in utopianist terrorism – religious or otherwise. This is an important point the media frequently omits to report while publicising the terrorists’ bombastic, self-aggrandising statements. Popular consent to government by the Islamists tends to be noticeable by its absence.

Moubayed’s insider account brings to bear the experience of two decades of analysing Syria and the Middle East. He writes of Al Qaeda’s Syrian franchise, Jabhat Al Nusra’s record of government under its “extremely delusional” leader Abu Mohammed Al Golani. A system where public services include administering schools, mosques and courts, providing water and electricity, collecting garbage and fixing prices of bread and oil – not to mention closing down hairdressers, removing mannequins from shop windows, forcing residents to pray, whipping those who don’t and so on. Reading this dirigiste nonsense in 2015, many readers may turn their eyes towards the heavens and think, God help the Syrian and Iraqi people.

The motivations for joining Al Qaeda or ISIL vary enormously from one individual to another, but some generalisations are nevertheless possible. “Sexual opportunity, status and adventure” loom large for young, alienated, bored, underpaid and under-*sexed men in the West, writes Burke. The narcissism of “Al Britani Brigade Bangladeshi Bad Boys” from Portsmouth, England, is evident in the name. A hamburger-flipping non*entity can become, at a stroke, a self-*important “Abu Sayyaf Al Britani”, named after the Prophet’s sword-bearer.

On a broader scale, the recent years of discredited US-led interventions in Iraq and Afghanistan, together with the implosion of Syria and Libya and the trials and tribulations of Egypt, have had damaging consequences on how the Muslim and western worlds see each other.

In a survey from 2011 to 2014, the US’s Pew Research Center revealed trends that were both heartening and deeply discouraging. On the one hand, an extensive 2013 poll of Muslims worldwide revealed that support for suicide bombing and Al Qaeda was limited and concerns over extremism were high. On the other hand, while the proportion of people in the US, UK and France who had favourable views of Muslims remained steady at about two-thirds, the views of Christians and Jews within the Islamic world declined dramatically, especially in Turkey, where favourable views of Christians declined from 31 per cent in 2003 to 6 per cent in 2010. By 2011, favourable views towards Jews stood at just 2 to 3 per cent in most Muslim countries. This is a tragic polarisation from which no good will come.

“What worries me most now,” says Burke, “is not the spread of Islamic militancy, but the spread of some of those elements that are necessary but not sufficient for extremist violence – anti-*Semitism, homophobia, seeing the West as decadent, the sense of righteous betrayal and victimhood, of being under attack.”

For a Syrian like Moubayed, who has seen his country destroyed in recent years, optimism is inevitably in short supply. He acknowledges that ISIL is not going anywhere soon, pointing to the self-proclaimed caliph Abu Bakr Al Baghdadi’s ability to create affiliates and provinces “without firing a bullet or spending a penny”. He worries about its spread into Libya and the potential for it to win more followers among Europe’s disaffected Muslims, a clear and present danger for social peace. Yet Syria, he argues, will rise from the ashes. “Politicians make mistakes. Regimes miscalculate. But history doesn’t. History always gets it right.” Ascribing agency and morality to the impersonal force of history may be overdoing it – time will remove saints and sinners alike – but hope is a precious commodity and cannot, or must not, be crushed.

ISIL, like Al Qaeda, has sworn to eliminate what it calls the “grey zone”, the area between good and evil, belief and unbelief. Yet this, as Burke eloquently argues, is where all that is best about our world – “diversity, tolerance, understanding, discussion and debate” – resides. The majority of people, Muslims and non-*Muslims alike, want to decide for themselves how to live their lives, not be dictated to by repressive bullies. For all its braggadocio, this is where ISIL, like other terrorist groups before it, will be doomed to failure, defeated by the far greater proven strength of human decency.

Justin Marozzi’s latest book is Baghdad: City of Peace, City of Blood, winner of the 2015 Ondaatje Prize.