Last week the government claimed it was succeeding in stamping out gangs and gang culture, in a review of current Home Office policy. But if you scratch below the rhetoric, argue Jon Shute and Juanjo Medina, you will find these claims are based on evaluation methods barely worthy of an undergraduate thesis.

Let us be clear; the Home Office review of its Ending Gang and Youth Violence policy is utterly appalling.

In the era of austerity, £10 million of taxpayers’ money has been wasted on initiatives that have not been described or evaluated. Grandiose success claims are made despite precisely no evidence of understanding or achievement.

Seen in these terms, the Coalition’s use of the term ‘gang’ can only be seen as a convenient rhetorical label for inciting public fear, scapegoating structural abandonment of, and justifying increased controls over, marginal populations – and for further stigmatising entire communities.

Despite – or perhaps because of – recent declines in youth violence, recent policy responses have begun to rely increasingly on the spectre of ‘the gang’ as a trope for representing serious youth crime, invoking moral panic, and justifying greater police powers in socially marginalised  communities.

The cynical disconnect between this and the growing weight of critical, empirical research on British youth gangs strains belief and exposes the unreason at the heart of Coalition policy.

Last year we criticised the policy on three grounds. The first was evidential: it could not define and operationalize the ‘gangs’ it declared as its policy object. Nor did it seem aware of the analysis commissioned and published by an institutional co-author of the report – the Home Office – that both offered a definition and challenged its own simplistic elision of guns, gangs, and knife violence. It also seemed oblivious to the general lack of a ‘what works’ evidence base for gang reduction, despite a century’s worth of well-funded US research.

The second  related to the indiscriminate blunderbuss nature of the framed policy response: four government bills and sixty varied policy initiatives were adduced as being relevant, effective or promising on the basis of weak, anecdotal or non-existent evidence; and no commitment to robust evaluation was given.

Finally, intellectual/moral incoherence was evident as the paper simultaneously wielded a large and enhanced criminal justice ‘stick’ in one metaphorical hand while proffering a somewhat wrinkled ‘carrot’ of pre-announced and austerity-compromised welfare support in the other.

Two years on, our worst fears have been confirmed. The main ‘Annual Report 2013’ repeats the confused and confusing blunderbuss approach: 49 pages, 60 action points, dozens of initiatives – many of which are existing generic DWP or Home Office programmes – and, in lieu of real evidence of impact, six glib text boxes describing a small range initiatives from the twenty-nine initial pilot sites.

The review uses the terms ‘gang’ no less than 266 times, which also features in many further initiatives, including – incredibly – the intention to provide gang ‘fact packs’ and ‘warning sign’ training documentation to police and a range of educational, community and criminal justice organisations.

Yet we know that:

• clumsy imposition of the ‘gang’ label tends to increase group cohesion, eliciting the very problems, attitudes and processes one was supposed to be guarding against

• in practice agencies apply this label in a problematic manner and on the basis of always (at best) approximate intelligence.

• not even official government-sponsored surveys can agree on what a gang is (compare The Offending, Crime and Justice Survey, The British Crime Survey for England and Wales 2010/11 and the one from 2011/12).

An accompanying ‘Review 2012-13’ document published simultaneously by the Home Office promises further detail of ‘achievement’ and ‘success’ (it studiously stays clear of the word ‘evaluation’) but only offers the following information:

• that the only effort at direct evaluation was from the funders themselves (the Home Office), which amounted to (i) two online surveys of ‘local contacts’ – mostly community safety managers as opposed to service providers – and (ii) up to three telephone interviews with the same contacts. Only 10/29 (34%) completed both surveys, and 13/29 (44.8%) provided an interview. Six trial areas ‘did not contribute to the research in any way’ (p6).

• that the statutory and voluntary organisations that were possibly heavily dependent on the Ending Gangs and Youth Violence funding were reported to perceive the experience of being funded broadly positive. However, the initial procurement and assessment arrangements took so long that little time was available to use the money, and there were also fears concerning the sustainability of funding. No individual or organisational measures of gang-involvement or offending behaviour were recorded.

• that due to the perceived impossibility of an eminently-achievable mixed methods (quasi-) experimental evaluation of project inputs, processes and outcomes, the Home Office elected to examine police recorded youth crime in project areas over 2012-13. It found that most forms of youth violence fell to the same extent that they had fallen in the year preceding EGYV and, indeed, in most other communities in England & Wales.

To be clear, the weaknesses of the ‘evaluation design’ are such that if it were an undergraduate research methods project, it would barely pass, and would certainly fail at postgraduate level.

Despite this, both Home Secretary Teresa May and Work and Pensions Minister Iain Duncan Smith make Orwellian claims of success in the ministerial foreword to the main document: “The initiative is working, the crimes that the programme aims to tackle are diminishing… the programme has led to more effective leadership and a greater sense of strategic direction. That has helped those on the front-line increase the effectiveness of their work. And that has contributed to the drop in youth violence” (p6).

The first and last clause of this statement are in no way supported by the available data, and in that sense, should be seen as grossly misleading and unsubstantiated.

We don’t discount the need to think about gangs in the context of youth offending prevention. But we need a more evidence-based, systematic and honest approach to it.