Opening the expert panel which convened last week in Washington to mark 3ie's 15th birthday, Inter-American Development Bank President Ilan Goldfajn made a pitch that would be echoed throughout the morning:
"We need an effectiveness culture."
Right now, one metric is still dominant across development institutions: the amount of funding. These panelists wanted to change that.
"The number one indicator still, in our system, is 'How much money did you pay out last year?'" said Håvard Mokleiv Nygård, director of knowledge at the Norwegian Agency for Development Cooperation. "That is always creating perverse incentives."
While the importance of dollar figures will not go away, other incentives are needed to ensure that the money is spent well.
"Officers in the field are motivated to close deals," Goldfajn said. "But we want them to close effective deals that will have an impact."
When 3ie was founded, it made less sense to talk about evidence-informed decision-making, because almost no impact evaluations had been conducted. But that’s not the case anymore, said 3ie Executive Director Marie Gaarder.
"The body of evidence that can inform decisions has increased dramatically," Gaarder said.
The production of thousands of rigorous impact evaluations worldwide by the development community is a remarkable achievement, said Amanda Glassman, executive vice president of the Center for Global Development, which co-hosted the event with 3ie.
"The diffusion of that information and the potential of rigorous evidence to improve impact was a development innovation in itself," Glassman said.
Still, too much evidence sits unused at decision time.
"The good and timely use of evidence continues to be really challenging," Gaarder said. "Evidence use needs to be planned for and woven into the institutional culture. I believe we can do better, and it is incumbent on us to do so."
There are plenty of challenges. Other priorities can always win out – like politics, said Nathanael Bevan, deputy director of research in the Research and Evidence Directorate of the UK Foreign, Commonwealth and Development Office. Champions of evidence need to recognize that they won't always win.
"Success is about being really clear that whenever we're making a decision, we're bringing the best available evidence," Bevan said. "It's really about persistence and incremental change."
Also, even when politics and evidence align, a policymaker has to know how to turn findings into actions.
"It's not easy or straightforward to take a global evidence base and know whether or not it applies in your context as a practitioner or policymaker, or if it does, how to apply it to your lived real-world context," said Anne Healy, senior advisor to the chief economist of USAID. "It's not something that you should expect every single colleague to be an expert in doing."
This challenge can be particularly acute in governments and agencies based in low- and middle-income countries.
"Capacity is a key challenge," said Sekhar Bonu, a 3ie board member and the former director general of the development monitoring and evaluation office of India’s NITI Aayog. "Capacity within the government, and capacity across the broader ecosystem … building those capabilities in the external ecosystem is really critical."
The capacity can come from "evidence translators who can sit between and bridge the space between the evidence producers and the evidence users," Healy said. But"it is a unique skill set. It is not in large enough supply."
Panelists also pushed the evaluation community to reflect on the limitations of the methodologies which have been used so far and to strive to overcome them.
While many individual programs have been evaluated and shown to be effective, sometimes the combined effect can be lost because of "perverse incentives" in a political system, said Santiago Levy, a 3ie board member and non-resident senior fellow at the Brookings Institution.
"We never evaluated how the different pieces of the puzzle fit together," Levy said. "We looked at individual trees; we never saw the forest. The individual trees, some of them were very pretty. The forest was ugly."
He called for evaluators to look for lenses that would allow them to step back.
"We need to develop methodologies to look at the whole," Levy said. "We need to look at the general equilibrium of the system."
Despite the challenges, the panel agreed that impact evaluations – and the use of rigorous evidence – need to be mainstreamed into the culture of development institutions.
"This is not the cherry on the cake," Arianna Legovini, Director of the Development Impact Evaluation (DIME), World Bank, said. "This is part of the ingredients of doing development financing differently … Yes, we need to take a little bit of a chunk of what goes into investments, but then [it will] make those investment so much better."
Watch the full panel discussion here.
