The New Researcher Network hosted a breakfast session during Policy Week to investigate how much evidence is ‘enough’ prior to policy implementation and how to strike the balance between ‘ideal evidence’ and pragmatic decision making. The event was chaired by Dr Kieron Flanagan, whilst Dr Julian Simpson of the University of Manchester and Dr Kathryn Oliver  of the University of Oxford offered their critical reflections on the topic. Here Laura ffrench-constant recaps the session.

The panel was posed the specific question how much evidence is enough, but also the associated questions: which types of evidence are preferable and how can they be analysed for rigour.

Julian Simpson began the session by offering a historical perspective to the debate. He emphasised how historical perspectives are often lacking in policy processes, yet they have a lot to add in terms of evaluating the starting point of a policy process. Simpson noted that historical perspectives of policy can offer direct evidence but also add a valuable mode of thought which is open to complexity and a range of evidence.

Simpson utilised the example of the NHS and General Practice to illustrate how historical examples can inform policy processes. He made the point that history can offer unexpected insights which might complement other types of evidence, for example, the history of the NHS reveals it to be reformist whilst surveys of staff perceptions typically show attitudes resistant to reform. The final point Simpson made was that we should be less focused on quantity but focus instead on the quality of evidence when asking the question how much evidence is ‘enough’ to inform policy.

Next, Kathryn Oliver considered how much evidence is ‘enough’ with the caveat that models intended to depict the policy cycle never work as predicted in reality. Oliver pointed out that evidence within a policy cycle is meant to reduce political ambiguity and not decrease scientific uncertainty. She used the analogy of a rugby match and specifically the 2003 World Cup Final when England beat Australia 20-17. She explained that the evidence based decision in the final minutes of the game would have been for England to push for a try because typically more points are scored that way in a rugby match but instead a drop goal was the decision that ultimately won the game.

Oliver added that producing more evidence can only address a rational route for decision making, and it should be remembered that not all policy decisions are made rationally with some decisions based instead on personal experiences and emotions. In answer to the question, how much evidence is ‘enough’, Oliver responded that it depends on both the context of the policy decision and also the means by which the researcher chooses to feed the evidence into a policy decision; highlighting a number of different ways to submit evidence to policy processes beyond submitting evidence to committees.

During discussion at the end of the session, Flanagan raised the paradox that there is concerted effort by the UK government to build an evidence base via funding for research and direct employment of researchers yet a lot of evidence produced is potentially never used. Other questions submitted by the audience covered whether there is a hierarchy of evidence and whether researchers should be stealth advocates or honest brokers of their research. When asked how evidence can be involved in irrational policy processes, Oliver responded that it would have to play an informative as opposed to an arbitrative role.

Further reading:

Paul Cairney’s Blog ‘Politics and Public Policy’ on EBPM