We are capping our 10th anniversary celebrations with a two-part blog from our founding and current executive directors. Their reflections capture 3ie's journey from starting up to taking off and cruising ahead.
Back in 2008, 3ie was just my laptop and me. It has come a long way since then. Although I bid farewell to the organisation in 2015, I have continued to watch how the organisation has grown. What has 3ie achieved in the last ten years?
My vision for 3ie, which I laid out when interviewed for the position of founding executive director, was that 3ie should be in a developing country, funding research by developing county researchers to answer policy questions of interest to developing country decision makers.
First, our main office was set up in Delhi with the majority of staff recruited locally. I have been told that Bill Gates once asked, “Where is 3ie based?”, and was appropriately impressed when told, “India”. There were some tricky times in the early days when our location in Delhi was questioned. Now, the Delhi office is well established, and as the largest of 3ie’s three offices, plays a key role in the organisation.
Second, all 3ie grants have required the substantive involvement of developing country researchers on the study teams. It is one of the scoring criteria for proposals, which was refined in various ways over time. We offered grants requiring the grant-holding institution be based in a developing country. We also designed the grant to ensure research teams developed engagement and evidence use plans. These plans focussed on engagement with implementing agencies and other key actors from the initial proposal stage and throughout the evaluation cycle.
Finally, policy relevance has been a central feature of 3ie’s grants, the emphasis on which has grown over time. Early on – I think ahead of the crowd – we moved from ex-post dissemination to ex-ante stakeholder engagement. By 2010, we had implemented the first version of our Policy Influence Plan, which we refined and evolved into the evidence-informed Stakeholder Engagement and Evidence Uptake Plan. By requiring a plan and regular engagement, which we jointly monitor with our grantees, we saw some type of evidence use in approximately half of our studies. By closely monitoring and robustly testing evidence uptake claims, we have established the types of impact studies tend to have. 3ie’s approach to stakeholder engagement and monitoring and measuring evidence use are considered by many to be standard setting.
Influence can also be systemic. One of the first awareness raising workshops I ran was in Kampala in November 2008. At that workshop, responsibility for impact evaluation was moved from the Ministry of Finance to the Office of the Prime Minister). Workshop participants made a list of priorities for impact evaluation. I later learned that that list was taken to the cabinet for approval, and became the foundation for the Government Evaluation Facility (GEF). 3ie committed to only fund evaluations approved by the monitoring and evaluation sub-committee to be part of GEF. Working with the Office of Prime Minister, we received DFID support to fund a number of GEF studies under the Uganda Evidence Programme. I was asked to present on the role of monitoring and evaluation to the entire cabinet – which led to a two-hour discussion with all other business being pushed to the next meeting. One of my proudest moments at 3ie was receiving the first ever award given by the Ugandan government for ‘Services to Evaluation’ in 2014.
3ie’s contribution to the evidence architecture and research transparency
3ie was conceived of by the founding donors as a global grant-making facility for impact evaluation. It has commissioned and assured the quality of 234 impact evaluations, 38 systematic reviews and 23 replications. This is a significant contribution to our global body of knowledge.
It has achieved so much more than this. The ‘much more’ 3ie has achieved are the investments that have been made in building evidence architecture. The 3ie evidence database was launched immediately after we had a website. It was first populated in a rather ad hoc manner as we worked to reach 100 included studies. The database now has nearly 5,000 impact evaluations and over 300 reviews and is a widely recognised, valued and used resource. Another early initiative was starting up the Journal of Development Effectiveness.
Conducting systematic reviews was part of the original vision of 3ie. They were virtually unknown in international development at the time outside of the health field. 3ie played an important part in their wider adoption in the development sector. We turned to the Campbell Collaboration for help with systematic reviews, and in 2010, established the Campbell International Development Coordinating Group, which 3ie manages from its London office. From being a novice in the review field, 3ie quickly became an innovator. In 2015, it published the largest education effectiveness review that examined the impact of 216 programmes in 52 low- and middle-income countries. 3ie staff have been associated with discussions of including a broader range of impact evaluation designs in reviews, qualitative synthesis and theory-based, mixed-methods reviews.
3ie has also been a pioneer. Starting in 2012, 3ie developed the methods for creating robust and effective evidence gap maps and the means to display the maps in interactive online platforms. These maps are increasingly in demand and a cost-effective way to identify where to look for quality evidence for decision-making or to set funding priorities to fill gaps.
There were also early plans to establish a replication programme, which finally got going after the Washington office took it on after a few years. The programme sparked off public conversations on replication, research quality and transparency. The office subsequently also started up the 3ie Registry for International Development Impact Evaluations. The Washington office also managed a series of rapid impact evaluations on a variety of topics, which remain a model of low-cost, relatively quick randomised evaluations.
We were active in other ways in getting the 3ie name out there. One such effort, was organising large events, starting in Cairo in March 2009 with the first ever truly global impact evaluation conference attended by over 700 people (by this time I had an assistant!). A lot of time was spent on the road delivering ‘awareness raising workshops’ and building relationships in countries such as Uganda and South Africa which have become important champions of evidence in Africa.
Where does 3ie go from here? We originally defined 3ie’s vision as ‘Improving lives through impact evaluation’. That vision is needed as much today as it was 10 years ago. I know it, the dedicated staff of 3ie know it, and the world will know it, as 3ie continues to live up to this vision in the next 10, 20 and 30 years. May the evidence be with you.
To read our current executive director, Emmanuel Jimenez's blog, click here.
