A little less than a year ago, at the start of what has become an infamous year, we launched a yearlong social media campaign called ‘2020 Hindsight: What Works in Development.’  Introducing the campaign, we explained that 20/20 vision is a term used to express that you can see clearly at 20 feet what should normally be seen at that distance. From there, the term 20/20 hindsight means one is able to evaluate past choices more clearly than at the time of the choice – indeed with the perfect clarity of vision that one did not have at the time.

Given the 2020 hindsight we now have about this past year, it is tempting to write a blog about how much we should have prepared for beforehand, knowing full well that pandemics can occur. But we will resist this path because it would take away from the blog's main purpose: to reflect on what we learnt from our synthesis work.

3ie was established in 2008 to help address the gap in quality development evidence that existed at the time. Since then, we have supported the production of new and rigorous evidence of what works, where, why, for whom and at what cost. Fast forward to 2020, and thousands of development effectiveness studies have been implemented in low- and middle-income countries by 3ie and many others, and hundreds of systematic reviews have synthesized the bodies of rigorous evidence relevant to development policy research questions. In addition, 3ie has curated the world’s largest searchable development evidence portal.

While the evidence gaps are still large in sectors like climate change or in contexts of fragility and conflict, we need to hold ourselves to account. What have we learnt 14 years after the report asked ‘when will we ever learn?’ We were aware of the possible challenges. As we noted in our launch post:

(i) A lot of the impact evaluations included in systematic reviews are of relatively low-quality, hence we should be careful about the conclusions. (ii) A lot of the systematic reviews find conflicting evidence due to a lack of evidence, or moderating factors like context, implementation and intervention design. (iii) There may be evidence that interventions mostly do not work. (iv) And, even when programs create change, they may not achieve the desired development goals. Elsewhere, we have also blogged about the fact that impact evidence alone is not enough - cost evidence is essential for deciding how scarce resources can best be used and that this is still largely missing.

As for the first of these concerns, the newly-updated development evidence portal made it an easier problem to manage. Since all of the systematic reviews in the portal have a quality appraisal rating, it was easy to weed out the low-quality reviews.
The remaining points all turned out to be more-or-less true. Nevertheless, we did find sufficient clear, interesting, actionable findings of positive effects to fill up the year, though their cost-effectiveness was missing in action.

The COVID-19 pandemic did make us reflect on how existing evidence could help us with something unprecedented. Looking at health and sanitation research took on a new importance, leading us to write about handwashing, vaccination campaigns, and community health workers.

Since it's the holiday season, gifts are on many of our minds. For us, this campaign has been a year of gifts, in a sense, with each piece of evidence representing a gift of knowledge. Here are a few of our favorites:

• How to get people to wash their hands: Only in 2020 could a blog post about hand washing have attracted so much attention and become our favorite post.
• What works to improve vaccination rates in low- and middle-income countries: We wrote this post back in April, hoping that it would be useful before too long. Now, the topic is even more relevant, as the distribution of COVID-19 vaccines begins.
• In humanitarian emergencies, cash-based social assistance is cheaper than food distribution, and it works just as well: With countries around the world initiating or expanding cash transfer programs to help manage the economic effects of the pandemic, we were glad that the evidence showed it to be an effective approach.
• What works to help students learn? Teach the teachers. Feed the students: The most effective solutions are not always high tech. While this blog was written before the pandemic had been declared, structured pedagogy programs (the combination of new instructional materials and teacher training) adapted to an online teaching reality are likely to only increase in importance!
• What works to get firms hiring? Support small businesses — but not the smallest ones: This post is not quite what we expected when we set to write about support for small businesses. Rather than findings about which types of programming were most effective, we found evidence about what size of businesses should be targeted for support, if the goal is to increase hiring.
• Shifting forest management to local control may reduce deforestation, without payments to anyone: Unlike other posts, this finding is about a policy change, not a program funded by a government or a donor. In fact, it would not necessarily cost anything!

We hope that our readers have found this series not just interesting, but useful. To share stories of how our campaign led to concrete changes in development policies, programs of research, or other examples of evidence use, we invite anyone to contact us at influence@3ieimpact.org. The best examples we receive will be recognized on our website with blog posts in next year's social media campaign, which will focus on evidence impact and use. We look forward to hearing from you!

This blog is part of our campaign 2020 Hindsight: What works in Development. Learn more about the campaign and read past blogs here.

Interventions that raise agricultural productivity in poor regions have the potential to offer two benefits at once: increasing the food supply and providing income for farmers, who make up a majority of the world's poor. Increases in yields of cereal crops have closely tracked with reductions in poverty in Sub-Saharan Africa, according to the World Bank. Improved agricultural technologies can improve yields, but only if people know how to use them.

Farmer field schools – intensive group trainings in new agricultural technologies – have been effective at increasing yields, incomes and profits for farmers, in addition to reducing the use of harmful pesticides. That said, these programs seem to provide the most benefits for farmers who have relatively more land and better education than their peers do. So they may produce plenty of positive effects, but they also should not be viewed as a solution for the poorest of the poor.

This evidence comes from a systematic review, which brings together evidence from 92 studies on 71 separate farmer field school programs in Africa, Asia and Latin America. By synthesizing the evidence from so many separate studies, the review offers a more comprehensive look at the field schools' effectiveness than any evaluation of a single program.

The farmer field schools included in the review focused on different topics and crops, but all featured intensive, facilitated group training. These trainings often took place at weekly meetings throughout the growing season, conducted in combination with the use of control plots farmed using standard practices. A majority of the studies took place in Sub-Saharan Africa and South Asia.

Common curricula included integrated pest management (IPM) and integrated production and pest management (IPPM), both of which aim to improve production while minimizing the use of toxic pesticides.

The field schools have been shown, on average, to increase knowledge of new farming practices, increase the adoption of those practices, increase yields, and increase revenues. The increases in revenues for farmers were larger than the increases in yields, likely because farmers were also spending less money on pesticides.

It appears, however, that it is primarily more educated farmers with more land who benefit from these programs. They have the education to understand the programs, and they have enough land that they can employ a new technique on some of it. Although data limitations make it hard to say how much land or education is necessary, the review's analysis found that each additional year of schooling, on average, increased farmers' yields and reduced their pesticide use after participating in FFS programs. These findings suggest that programs may have larger effects if they target more educated farmers with more land. At the same time, this evidence suggests that other approaches may be needed to help less educated, poorer farmers.

Also, for farmer field schools that included both men and women and those that targeted women specifically, the programs only benefited women who had the social standing to utilize the training. For example, in places where women primarily grow subsistence crops, farmer field schools about cash crops did not benefit them. In other cases, programs did not produce effects because they targeted women who did not have decision-making power in their households, the review found, making them unable to lead the adoption of a new technology. However, in some cases where programs were appropriately targeted, as in a study in Tanzania, women benefited more from the farmer field schools than men.

In general, the programs seemed to only affect participants' farming practices, not their neighbors. Despite hopes that farmer field schools might lead to knowledge diffusion, the review's findings suggest that such knowledge transfers do not generally happen. There may be a few cases where knowledge diffusion occurred, in programs that targeted more educated farmers about simple topics like reducing pesticide usage. The main barrier to diffusion identified in the review was the complexity and experiential nature of FFS learning.

The other issue faced by farmer field school programs is scale. Virtually all of this evidence comes from evaluations of smaller-scale programs. The review found that the training of facilitators in participatory leaning methods is key to program success, and this type of training can be hard to scale up.

Despite all these caveats, the evidence shows that well-targeted farmer field school programs led by well-trained facilitators can produce several positive outcomes at once: higher agricultural yields, higher incomes for farmers, and reduced pesticide use.

The full systematic review, as well as a short summary, can be found here. Beyond that review, hundreds more systematic reviews and thousands of impact evaluations on all kinds of development topics are available at through our Development Evidence Portal.

This blog is part of our campaign 2020 Hindsight: What works in Development. Learn more about the campaign and read past blogs here.
 

A majority of the world's poor live in rural areas and work in agriculture, according to the World Bank. Furthermore, hundreds of millions of people, including some of those same farmers, do not have enough nutritious food to eat. So it is clear why raising agricultural income and productivity in low- and middle-income countries is a priority for governments and development organizations.

In this case, one straightforward approach seems to be effective. Subsidizing agricultural inputs – seeds or fertilizer – has been shown to increase both agricultural productivity and farmer income. That said, proper implementation and timing are extremely important for such programs, because farmers need to have the subsidized inputs in time to meet nature's schedules.

These findings come from a 3ie systematic review which draws together evidence from 15 separate studies assessing how agricultural input studies affect yields, farm incomes, and the adoption of new agricultural technologies. Since they synthesize findings from multiple studies, systematic reviews provide stronger evidence than individual case studies, where one-off issues can drive results.

All but one of the studies included in this review were conducted in Sub-Saharan Africa, including seven in Malawi alone. The remaining study was from India. Ten of the interventions subsidized both seeds and fertilizer, three only subsidized seeds, and two only subsidized fertilizer. Many programs targeted rice or maize production. The amounts of the subsidies ranged from 22 to 100 percent. Subsidies were often provided through vouchers.

The review found positive effects on farmers' incomes, productivity per hectare of land, and rates of adoption of new seeds or fertilizer. There did not appear to be a significant relationship between the size of the subsidy and the outcomes, but that absence of a finding could be a result of contextual differences between the countries where studies took place.

Although the findings were generally positive, the review identified implementation issues which prevented subsidy programs from reaching as many farmers as they could have. In some cases, late voucher distribution and shortages of subsidized inputs reduced programs' effects. In other cases, prices remained so high even after subsidies that farmers did not use their vouchers. And several studies found that some farmers were reselling vouchers or fertilizer.

The review also does not have data on the cost efficiency of the programs, which is a common problem among impact evaluations, as we've written about here and here.

At this point, some readers may be asking: Wait, how might such subsidies affect the larger economy, considering fiscal policy issues or price distortions? The review does its best to provide an answer: the subsidies probably have small positive effects on consumer welfare and economic growth. But because these estimates are drawn from a fundamentally different type of analysis, they are much more tentative. [In other words: If this wasn't your question, don't worry about the stuff below – it gets technical and is far more uncertain than the findings we reported above.]

To consider macro effects, the review also includes 16 modeling studies, which are separate from the impact evaluations discussed above. These studies are country-level simulations of how the whole economy may be affected by changes in subsidy policies, based on macro-level economic data and other indicators.

I'll quote the review's authors to explain the issues with this process:

"The functional relationships specified between sectors, agents, goods and prices in models are in most cases approximations. How these relationships are specified can influence the output of models to a very great degree. For instance, a number of modeling studies simulate broadly similar changes in fertilizer subsidies in Malawi over a similar time-period with, in some cases, quite different results."

Nonetheless, as noted, above, these modeling studies generally show positive effects on consumer welfare and economic growth. Most of the models show that subsidizing inputs leads to a reduced price of the crop once it is grown, alongside an increase in consumption of that crop, as well as an increase in demand for agricultural labor. All of the studies which modeled incomes predict that subsidies increase real incomes.

Some of these models also suggest that the source of funding matters for outcomes, finding that subsidies financed through taxation yield positive effects, but subsidies financed through reductions in rural infrastructure spending may yield negative effects.

For more details, the full review is available here. Hundreds more systematic reviews and thousands more impact evaluations on a range of development topics are available on our Development Evidence Portal.

This blog is part of our campaign 2020 Hindsight: What works in Development. Learn more about the campaign and read past blogs here.
 

The importance of electricity for poverty reduction is highlighted in the seventh Sustainable Development Goal: Ensure access to affordable, reliable, sustainable and modern energy for all. Nonetheless, more than 800 million people worldwide did not have access to electricity as of 2018. Tremendous investments from a range of sources, including the Asian Development Bank, have been made to close this gap – over $30 billion per year in 2015 and 2016. So what social outcomes does this type of investment yield?

A recent systematic review produced by the Asian Development Bank's Independent Evaluation Department (IED) and The International Initiative for Impact Evaluation (3ie) suggests that programs to increase access to electricity led to modest gains on education measures and household income. The largest effect was the increase in the time students spent studying at night, while other educational and income measures rose slightly. The interventions also resulted in slightly higher levels of total household income, lower levels of household poverty, and higher levels of business productivity. All these effects were statistically significant but small, except for the change in night study time, which was larger.

The review combines the results from 126 impact evaluation studies published since 2004 evaluating interventions to expand electricity access implemented by different organizations all over the world. This synthesis offers stronger evidence than an individual evaluation, where results can be driven by one-off factors.

While the number of studies has increased since 2013, with 82% of studies included in the review published since then, rigorous studies on the impact of electricity are still limited. Impact evaluation studies in energy – in which electricity is a subsector – account for less than 1% of all impact evaluations published from 2000 to 2015.

Most of the interventions in the review were expansions of electric grids into rural or peri-urban areas, while a few evaluated smaller-scale, off-grid interventions to expand access. An even smaller number of evaluations investigated demand-side interventions, such as reduced electric rates. The interventions were implemented in Asia, Africa, Latin America and the Middle East.

In general, the quality and reliability of electricity service varied widely across the studies, before and after the interventions – but each intervention represented an expansion of electricity access.
The review notes that one key mechanism by which electrification increases incomes is by reducing the time women spend on housework, allowing them to earn other income. However, in some places this mechanism was blocked by gender norms against women working outside the home.

In addition to modest effects, the review also found considerable heterogeneity, or variation across studies, that resulted partly from study characteristics, such as study design, the way variables were measured, location, or length of follow-up after interventions took place. With a high degree of heterogeneity observed across studies, understanding how context influences the use of electricity infrastructure is key for producing positive changes in social outcomes. Without appropriately considering these factors, individuals may not alter their routines to utilize the new infrastructure.

Challenges still exist that prevent universal access to electricity, including high investment and operation costs; limited demand given market conditions; implementation and maintenance issues; and regulatory or political issues. Even with expanded access, other factors like connection quality and unequal access may have limited the development impacts. Important evidence gaps remain for future studies to address. Understanding what works and does not is important for policy planning and decision-making.  

Given the findings and our understanding of the evidence base, the systematic review calls for: (1) more funding for rigorous research to fill evidence gaps; (2) adjustments to the expectations for the magnitude of project outcomes; and 3i) interventions targeting multiple constraints, such as availability, reliability, and affordability.

The greatest needs for new research focus on: (1) understanding the effect of demand-side interventions, (2) conducting theory-based mixed-method evaluations, and (3) measuring outcomes related to quality of life and social experience.

The report is the first systematic review conducted by IED. Reflecting a continued effort to expand and diversify IED evaluation products, methodologies, and knowledge, IED intends to produce more systematic reviews as part of its regular work program.

For more details, the full report is available here. A version of this blog is also available on the Asian Development Bank's website here. 

This blog is part of our campaign 2020 Hindsight: What works in Development. Learn more about the campaign and read past blogs here.