Launching a campaign- 2020 hindsight: what works in development?

Today we are launching, at 3ie, a yearlong social media campaign called ‘2020 Hindsight: What Works in Development.’  For our non-American readership, let me briefly explain where the idiom ‘Hindsight is 20/20’ comes from. 20/20 vision is a term used to express that you can see clearly at 20 feet what should normally be seen at that distance. 20/20 hindsight means that one is able to evaluate past choices more clearly than at the time of the choice – indeed with the perfect clarity of vision that one did not have at the time.

To explain why we are launching this campaign and what we are hoping to accomplish, join me first on a little experiment of travelling back in time. In March 2006, the Evaluation Gap Working Group launched a report suitably titled When Will We Ever Learn? Improving Lives Through Impact Evaluation, which was one of the fundamental reasons why 3ie was established in 2008. The report concluded grimly and correctly that:

‘Each year billions of dollars are spent on thousands of programs to improve health, education and other social sector outcomes in the developing world. But very few programs benefit from studies that could determine whether or not they actually made a difference. This absence of evidence is an urgent problem: it not only wastes money but denies poor people crucial support to improve their lives.’

3ie was established to help address this evidence gap by supporting the production of new and rigorous evidence of what works, where, why, for whom and at what cost. Several other important players also came into existence during the first decade of the new millennium: Innovations for Poverty Action,  the Center for Effective Global Action  and the Abdul Latif Jameel Poverty Action Lab , two of whose founders won the 2019 Nobel Prize in economics.

Fast forward to 2020 and thousands of development effectiveness studies have been implemented in low- and middle-income countries (L&MICs), examining what works, for whom, why, and at what cost. 3ie was an important contributor to this achievement. Most of these studies, which now constitute a large public good of evidence, are available in 3ie’s searchable repositories of impact evaluations.

So what are the 10 things that work? Surely, 3ie should have something to say about this given its role in addressing the evidence gap described more than a decade ago?

Yes, but...not so fast! The studies we are referring to here are individual impact evaluations. These can inform us about whether a certain intervention or development program worked, but their findings may be of limited applicability for significantly different countries, contexts, time-periods, or implementation approaches.

The good news is that 3ie has gone a step further. We have produced or supported more than 50 systematic reviews thanks to the foresight of our founding executive director, Howard White. And since then 3ie has curated the world’s largest repository of systematic reviews of development-related questions.

Systematic reviews use standardized methods to collect secondary data, critically appraise research studies, and synthesize findings qualitatively or quantitatively. They are designed to provide a complete and exhaustive summary of current rigorous evidence relevant to a research question. What this means is that you can rely on all of the existing relevant studies for a certain topic, rather than just one. As long as this is properly done, most people will agree that several data points are better than one!

What this means is that you can ask 3ie —can a conditional cash transfer program increase the rate at which people go to get health check-ups? —and we can answer affirmatively with some amount of certainty based on available evidence. We can say this because we have systematic reviews (see 1 and 2) on the topic. Without a systematic review of the evidence, we would more likely say ‘well this worked here and the other worked there.'

While the evidence gaps are still large, for example in sectors like climate change, or in contexts of fragility and conflict, we need to hold ourselves to account. What have we learnt 14 years after the report asked ‘when will we ever learn?’ A back-of-the-envelope calculation suggests that since then, about three billion US dollars have been spent on the impact evaluations and systematic reviews captured in the 3ie repositories* . This is not pocket change, but only a very small fraction of the amount spent on development interventions over the same timeframe, which may or may not work. Nevertheless, policymakers and donors who funded this research deserve answers about what works. More importantly, people in L&MICs deserve development interventions that are based on the best evidence available to achieve their intended outcomes.

We therefore embark on a yearlong social media campaign, 2020 Hindsight: What Works in Development. We will present insights from programmes and policies that work based mainly on evidence from systematic reviews.

My team has warned me of the challenges. (i) A lot of the impact evaluations included in systematic reviews are of relatively low quality, hence we should be careful about the conclusions. (ii) A lot of the systematic reviews find conflicting evidence due to a lack of evidence, or moderating factors like context, implementation and intervention design. (iii) There may be evidence that interventions mostly do not work. (iv) And, even when programs create change, they may not achieve the desired development goals. Elsewhere, we have also blogged about the fact that impact evidence alone is not enough - cost evidence is essential for deciding how scarce resources can best be used and that this is still largely missing.

Nonetheless, the year 2020 will be our year to reflect on what the increasing body of evidence taught us. We hope that our partners at Campbell, EPPI, GESI, ACE, Cochrane and other evidence producers, as well as you, dear reader, will join us!

*This is of course only a sub-sample, as many studies have not passed the inclusion threshold for these repositories. There is also a wider range of evaluations out there, including performance evaluations, programmatic evaluations, corporate evaluations, process evaluations, formative evaluations, qualitative assessments etc, none of which are captured in the repositories, and on which the largest amount of resources are being spent.