The opening ceremony and plenary session at the Making Impact Evaluation Matter conference in Manila made clear that impact evidence – in the form of single evaluations and syntheses of rigorous evidence – do indeed matter. Two key themes were (1) strong evidence about the causal effects of programmes and policies matter to making decisions that improve the welfare of people living in low- and middle-income countries and (2) that, to make impact evaluation matter more, we need to continue to make efforts to build capacity to generate, understand, and use such evidence in those same countries.
Impact Evaluations do matter for decision-making
As this conference was hosted in Manila – the first conference focused on impact evaluation in Asia – it is no surprise that a repeated example of evidence-informed decision-making related to the Philippines conditional-cash transfer programme, known as the ‘4P’s. As noted both by the Philippine socioeconomic planning secretary Arsenio Balisacan and Chair of 3ie’s Board Richard Manning in the opening ceremony of the conference, impact evaluation evidence has been crucial in maintaining and expanding the 4Ps programme. It will ideally also be crucial in insulating 4Ps from party politics and changes in the administration. Moreover, the impact evidence led to an important modification of programme coverage, expanding eligibility to poor students at the secondary level as well as those at the primary level.
This is one example of how high-quality evidence can inform decision making and, ultimately, make a difference to people’s lives. But for impact evaluations to matter they need to be available and address key policy questions. As Richard Manning noted, while the volume of impact evaluations and evidence syntheses produced in recent years has grown quickly – with about 300 new studies produced every year – only 2-3 per cent of global development spending is subjected to evaluation. What about the rest? Without evidence we cannot offer clear guidance to policymakers about which programmes are most effective in improving lives.
According to Bindu Lohani of the Asian Development Bank (ADB), the evidence gap is a more urgent concern in some sectors than others. He highlighted infrastructure – a focal investment area for the ADB – as an example of where impact evaluation evidence is lacking. As the effects of a changing climate grow in magnitude, the need to build an evidence base on interventions for climate resilient infrastructure keeps increasing.
Building capacity and ‘working with’
Richard Manning highlighted that the past ten years has seen a shift, from an initial focus on evaluating the effects of aid programmes, to a broader focus on development effectiveness. Rather than being researcher- or donor-led, the demand for evidence increasingly comes from national and sub-national governments in low- and middle-income countries. Policymakers in these countries have limited resources and want to ensure their fair allocation and efficient use. The Philippines exemplifies this. With support from the Australian government, the Philippine government has worked towards generating and using evidence to inform their social programming.
With this shift, it becomes all the more important that researchers and decision-makers gain capacities in setting priorities and asking key questions, generating evidence, and then understanding and making use of this rigorous evidence. Bindu Lohani stressed that to ensure more widespread use of evidence, we should work to build the capacity in all countries to produce evidence that meets high standards.
In this way – and through early-and-often engagement (as highlighted by 3ie in its two days of pre-conference workshops and again by Paul Gertler in the opening plenary) between researchers, decision-makers and local stakeholders – more evidence can be produced that is rigorous and useful. This is a fundamental step in making impact evaluation matter. International donors will continue to play an important role in this. Richard Manning highlighted the example of the Philippines-Australia collaboration, where a bi-lateral funder supports a country in building the evidence across a range of programmes, as good model for others to follow.
The Philippine secretary of social welfare and development Corazon Juliano Soliman noted that impact evidence can be lost in translation and therefore not used or used most beneficially. She implored impact evaluators to speak plain English to users, if they want their evidence used. More work is required to ask questions that matter to decision-makers and then to convey the results in a jargon-free language that decision-makers can understand
The plenaries and debates in Manila last week highlighted that impact evaluation matters. It is also clear that impact evaluation can matter more for improving the welfare of people living in low- and middle-income countries. One way to make it matter more is to better understand how impact evidence has been and can be used in decision- and policy-making. Another way is to build capacity to generate, understand, and use rigorous evidence in all countries.