Impact Evaluation Methodological Briefs co-authored by 3ie experts and published by UNICEF's Office of Research. The topics covered under these briefs include randomised controlled trials, quasi-experimental design and methods and theory of change.
A 3ie-supported study on the impact evaluation of the Cheeranjivi programme, a maternal health scheme launched by the Government of Gujarat in India cited in the World Bank Future Development blog (20 November).
3ie deputy executive director Jyotsna Puri was cited in a article titled The logframe dilemma in Devex (20 November 2014). This article is based on the logical framework — or “logframe” — approach to planning, monitoring and evaluation of development programmes is gaining traction among the international aid community. But does this system truly work for beneficiaries?
Evaluation Review is inviting papers for a special issue on survey versus administrative data in impact evaluations. In particular, they invite papers that compare impacts estimated with survey-based data with those estimated with administrative-based data, and attempt to reconcile the differences. Deadline for submission of abstract: 2 January 2015.
The Global Development Network is calling for papers, posters and sessions for its 16th Annual Global Development Conference on Agriculture for sustainable growth: challenges and opportunities for a new ‘Green Revolution’, from 11-13 June 2015, at Ben Guerir, Morocco. Deadline for applications: 31 January, 2015.
This blog by Henrietta Miers in SciDev.Net (31/10/2014) questions whether impact evaluations can measure the impact of gender empowerment programmes. Empowerment is a transformational change that needs to be understood without the pressures of observing results in short, pre-determined timeframes, she says.
Ruth Levine, director of global development at Hewlett Foundation and member, 3ie board of commissioners, interviewed by SciDev.Net
"To stay relevant in a changing world, aid agencies must focus their resources on meeting low-income countries’ growing demand for data that can help them assess the impact of development activities," Ruth Levine, director of global development at Hewlett Foundation told SciDev.Net (29 October 2014) during the 3ie Evidence Week in London, 13-17 October.
Call for papers: International workshop on 'Assessing the social, economic and political effects of social protection'
The Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), the German Development Institute / Deutsches Institut für Entwicklungspolitik (DIE) and the Internationales Zentrum für Internationale Zusammenarbeit/International Center for Sustainable Development, (IZNE) at Bonn- Rhein-Sieg University of Applied Sciences are convening an international Workshop on “Assessing the social, economic and political effects of social protection: ways to overcome the challenges” from 15-17 April 2015 at the German Development Institute / Deutsches Institut für Entwicklungspolitik (DIE) in Bonn, Germany. Deadline for abstracts: 20 December 2014.
Howard White was honoured with an award of distinction by the Government of Benin in London on 15 October. Aristide Djidjoho, general director of evaluation at the ministry of public policy analysis presented the award to Howard White on behalf of the president of Benin. The award was presented during the one-day colloquium of 3ie's London Evidence Week.
Djidjoho spoke of White’s significant contribution towards Benin’s development as well as his role in creating a sustainable environment for undertaking impact evaluations of development programmes. He went on to add that impact evaluations are not an end, but a means for achieving effective policies. He hopes that Benin and the whole of Africa will continue to benefit from White’s experience in the field of impact evaluations and evidence-informed decision making.
Djidjoho went on to thank Richard Manning, chair, 3ie board of commissioners and lauded 3ie’s efforts in building effective evaluation systems in several countries. He concluded by saying that through these sustained efforts in trying to strengthen public institutions, we can aim to achieve our common goals of development effectiveness.
Developing country policymakers are increasingly using impact studies to assess development programmes, Howard White, tells SciDev.Net (14 October 2014).
3ie invites policymakers, programme managers, researchers, and other stakeholders working on peace- and state-building interventions in low- and middle-income countries to complete this stakeholder survey in support of the Evidence for Peace: Evaluating to Improve Results in Countries Impacted by Conflict, Fragility and Violence (e4p) initiative launched in association with the Innovations for Poverty Action (IPA) and the World Bank
The stakeholder survey is one component of a set of activities to identify gaps in the evidence base and understand which evaluation questions are most pressing to those engaged in peace- and state-building work. It will catalog this information according to the opinions of experts in the sector.
The Making Impact Evaluation Matter conference organised by 3ie in collaboration with the Asian Development Bank and Philippine Institute for Development Studies in Manila, 1-5 September, received wide media coverage.
These include articles in the Asian Journal (3 September), Philippine Daily Inquirer (3 September), InterAkyson.com (3 September), BusinessMirror (3 September), Devex, (3 September), Devex, (4 September), Devex, (4 September), Manila Bulletin (7,September).
Awards for the best presentation at the Making Impact Evaluation Matter conference are:
Best overall presentation:
1st Prashant Loyoka
2nd Abu Shonchoy
2nd Bekele Shiferaw
Best presentation by a young researcher
1st Abu Shonchoy
2nd Nigussie Abadi
Best presentation by a L&MIC researcher
1st Nigussie Abadi
2nd Altantsetseg Batchuluun
3rd Abu Hayat Md. Saiful Islam
3rd Evans Muchiri
1st MacCarthy Honu-Siabi
1st Desiree Manicom
2nd Drew Cameron, 3ie
3rd Martina Vojtkova, 3ie
The Making Impact Evaluation Matter conference in Manila organised by the Asian Development Bank, 3ie and Philippine Institute for Development Studies was featured on Devex (3 September 2014).
3ie-supported study on metering of electric tube wells in West Bengal was featured in a television programme
3ie-supported study on the metering of tube wells in rural West Bengal, titled Does marginal cost pricing of electricity affect the ground water pumping behaviour of farmers? Evidence from India was featured in a television programme Neb Kolkata (30 August 2014).
3ie-supported study on student evaluation in schools in the State of Haryana, India, titled A wide angle view of learning: evaluation of the CCE and LEP programmes in Haryana was featured in Business Standard (31 August 2014), Zee news (31 August 2014), Nagaland Post (31 August 2014), The Hindu (1 September 2014), Webindia123 (1 September 2014), India Today (31 August 2014), Veooz.com (31 August 2014), Samachar.com (31 August 2014), The News (31 August 2014).
The Millennium Challenge Corporation (MCC) invites papers based on data from the MCC Evaluation Catalogue. Deadline for submission of abstracts, 5.00 p.m. EST, 31 January 2015.
A paper titled, Better targeting of farmers as a channel for poverty reduction: a systematic review of Farmer Field Schools targeting, by 3ie researchers Daniel Phillips, Hugh Waddington and Howard White, has been published in Development Studies Research.
The paper is available for download.
A 3ie-supported study evaluating the impact of the vocational education and training programme in rural China has been featured The Economist (23 August).
3ie is happy to announce the winners of the Impact Evaluation Repository challenge. In the last two months, we received 97 study submissions from more than 25 people.
Of the 97 studies submitted, 63 were eligible impact evaluations and were immediately added to the Impact Evaluation Repository. Of these, 32 studies** (and one correction) were published before 2013 and should have been found in our last search (thus are eligible for our $10 Amazon.com gift certificate prizes).
These studies are invaluable in our efforts to improve our systematic search and screening protocol and provide the international development community with the most up-to-date, published impact evaluation evidence available.
We would like to sincerely thank the following people for sending their approved impact evaluation studies to the Impact Evaluation Repository. As always, we encourage you to send any published impact evaluations to email@example.com.
Evidence Aid will hold a two-day training programme on systematic reviews on 13-14 November at 10.00-16.30 at the Queen's University, Belfast. The course will place systematic reviews in the context of disasters and the provision of humanitarian aid, both medical and non-medical.
William Savedoff of the Center for Global Development blogs on 3ie's journey from being a start-up to maturity. The appointment of Emmanuel (Manny) Jimenez as the new executive director marks a transition in the organisation.
"With the transition to a new executive director, 3ie will be able to build on past success while drawing on the different qualities and depth of experience and expertise that Manny Jimenez will bring to the post," says Savedoff.
The announcement of the Emmanuel Jimenez as the next 3ie executive director was featured on Hindustan Times (23 July 2014), the Indian national daily.
The Chair of 3ie, Richard Manning, has announced that the 3ie Board of Commissioners has appointed Emmanuel Jimenez to succeed Howard White as the executive director of 3ie in early 2015. Dr Jimenez is currently director, public sector evaluations in the Independent Evaluation Group of the World Bank.
3ie evaluation specialists participated in the Journées Béninoises de l’Evaluation in held in Cotonou, Benin, from 30 June to 2 July. There were 300 participants from Argentina, Mexico, Canada, France, South Africa, Uganda, Togo, Niger and Benin. Representatives from institutions like the West African Development Bank, the African Development Bank, GIZ, CLEAR, UNDP, UNICEF, etc. were also present.
The event received wide media coverage.
This paper by Annette N. Brown, Eric W. Djimeu and Drew B. Cameron reviews the literature on a set of self-tests that share some characteristics with HIV self-tests to determine whether there is any evidence of harm.
The International Rescue Committee requests feed back on 10 gap maps that they have constructed based on systematic reviews. These gap maps are visual representation of existing evidence on topics like agricultural interventions, access to quality education, cash transfers, social and economic empowerment of women.
A 3ie-supported study on Paying for performance in China’s battle against anaemia was featured in The Economist (14 June 2014).
3ie is playing a leading role in the development and delivery of a London International Development Centre (LIDC) short course on Evaluating Development Programmes.
The course will run 10-14 November 2014 at LIDC. Course fee: 1500 GBP. Applications are being accepted online on a rolling basis.
The Haskayne School of Business at the University of Calgary, Alberta, Canada is offering a post-doctoral fellowship. Applicants must hold or will imminently receive a PhD in the area of I-O psychology, applied psychology, organisational behaviour, or a related field. Research will focus primarily though not exclusively on the following two funded research projects: synthetic validity and metaBUS.
3ie regrets to announce that the planned thematic window on demining has been cancelled. We do not plan to launch a window on demining in the near future.
The Journal of Development Effectiveness, volume 6, issue 2, 2014 is out. In addition to other papers this issue features a study on the impact of microcredit on women's control over household decisions, a review of William Easterly's latest book The Tyranny of Experts.
Nature (30 April 2014) has cited the 3ie systematic review summary on farmer field schools in an article on agricultural innovations to increase yields and protect the planet.
A three-day evidence symposium on what works to improve labour productivity was organised by the International Labour Organization in partnership with J-PAL, Silatech and the Arab Urban Development Institute at the Georgetown University, Doha, 6-8 March 2014.The event brought together policymakers, development practitioners, academics and researchers to present and discuss the evidence base for what works in increasing the employment and productivity of youth. Discussants shared recent findings of impact evaluations from the Middle East and North Africa and other developing regions. Participants were also introduced to evidence from systematic and literature reviews which are important for evidence-based policies.
This paper by Thomas de Hoop, Luuk van Kempen, Rik Linssen & Anouka van Eerdewijk in the Feminist Economist looks at the relationship between participation in women's self-help groups, women's autonomy and subjective well-being in Odisha, India.
The findings suggest that while self-help group membership results in positive effects on women's autonomy, there is also evidence of negative effects on women's subjective well-being in villages with relatively conservative gender norms. Qualitative research suggests that this negative effect is associated with social sanctions after the transgression of gender norms.
The Global Institute for Development Evidence (GIDE) formerly, the Advisory Research Group International LLC (argI) headquartered in Honolulu, Hawaii (USA), is a non-profit international initiative for research-based evidence with services in Africa, Asia and USA.
GIDE helps to improve the effectiveness of policy-making and delivery by providing policymakers, development agencies and practitioners with access to authoritative independent policy research and expert advice. GIDE thereby increases development effectiveness and improves lives through production, use and dissemination of reliable evidence from best development impact studies for policy design.
Howard White's presentation on Evidence-based policies to reduce poverty at the Independent Evaluation Group, Washington, DC, was featured in Devex (21 April, 2014). The article talks about the importance of systematic reviews in measuring the effectiveness of development programmes.
3ie is part of the Global Open Knowledge Hub, an open-access digital data initiative to help increase online accessibility of development research coming from the global South and boost their decision-making influence.
An article in Sci Dev Net on the Global Open Knowledge Hub mention's 3ie's participation in this initiative.
A 3ie-funded impact evaluation of the Cheeranjivi Yojana, a health insurance programme in the state of Gujarat, India, undertaken by Duke University, was cited in an article titled Gujarat experiments with public health insurance in India Ink, New York Times, 3 April 2014.
Journal of Development Effectiveness, vol 6, issue 1 is out. The issue features a paper on Conditional, unconditional and everything in between: a systematic review of the effects of cash transfer programmes on schooling outcomes by Sarah Baird, Francisco H.G. Ferreira, Berk Özler & Michael Woolcock.
There's a section on book reviews as well. Howard White reviews three books -- Power: why some people have it – and others don’t, by Jeffrey Pfeffer, Hard facts, dangerous half-truths and total nonsense, by Jeffrey Pfeffer and Robert I. Sutton and The signal and the noise: the art and science of prediction, by Nate Silver.
Agricultural land tenure reforms have been less effective in Africa than Latin America or Asia, says Steven Lawry and Cyrus Samii in a Guardian op-ed. This is based on a systematic review, The impact of land property rights interventions on investment and agricultural productivity in developing countries. 3ie provided the quality assurance support for this review.
A three-day international workshop on Evaluating Forest Conservation Initiatives, held in Barcelona in December 2013. The workshop brought together about 40 researchers, practitioners and policymakers to discuss the complexities involved in evaluating forest conservation initiatives. Philip Davies, 3ie Deputy Director - Systematic Reviews was as the workshop. Davies has been cited in this blog by the Center for International Forestry Research.
The Economic Policy Research Institute (EPRI), together with the African Institute for Health and Development (AIHD), is offering a two-week course between 23 June and 4 July, aimed at providing participants with an in-depth understanding of the conceptual and practical issues involved in the development of social protection programmes in Africa. Participants will acquire tools required for the appropriate identification and successful design and implementation of these programmes.
Two papers from the Journal of Development Effectiveness are among the top 25 most read Development Studies papers of 2013 from among a variety of Routledge Development Studies journals. These include, An introduction to the use of randomised control trials to evaluate development interventions by Howard White and How to do a good systematic review of effects in international development: a tool kit by Hugh Waddington, Howard White, Birte Snilstveit, Jorge Garcia Hombrados, Martina Vojtkova, Philip Davies, Ami Bhavsar, John Eyers, Tracey Perez Koehlmoos, Mark Petticrew, Jeffrey C. Valentine, Peter Tugwell.
"To err is human, but to use the word 'error' in a replication study is usually not divine." Guest post on the Development Impact blog by Annette Brown, Deputy Director, Advancement and Impact Evaluation Services, 3ie, and Benjamin Wood, Evaluation Specialist for Replication, 3ie.
The special issue of The European Journal of Development Research features a paper by Howard White titled Current Challenges in Impact Evaluation.
Bill Saveoff of Center from Global Development applauds the achievements of 3ie under the leadership of Howard White. He also says it's time to revisit the original visions for 3ie – that all foreign aid and multilateral agencies should contribute 0.01% of their annual disbursements to 3ie in support of impact evaluation.
3ie-supported impact evaluation of a maternal health programme in Gujarat, India, shows that the much-touted Chiranjeevi Yojana, launched in 2006 to reduce maternal and infant mortality in the state, hasn’t had any significant impact on institutional delivery rates or maternal health outcomes.
Gap maps enable policymakers and practitioners to explore the findings and quality of the existing evidence and facilitate informed judgment and evidence-based decision making in international development policy and practice. This paper by 3ie researchers provides an introduction to evidence-gap maps, outlines the gap-map methodology, and presents some examples.
An article by Prof Michael Greenstone of the Massachusetts Institute of Technology in the New York Times cites a 3ie-supported study on reforming the environmental audit system in industrial units in Gujarat.
Howard White was interviewed on the Story of Aid in the Rear Vision programme on ABC Radio.
Howard White writes about the lessons 3ie has learned through the experience of conducting and managing impact evaluations in the past decade in the Impact magazine published by Population Services International.
An article in The Hindu newspaper (dated 31/10/2013) on the 3ie-ASCI conference on Measuring Results.
Howard White, 3ie Executive Director, was part of a discussion on evaluation of social sector programmes on Rajya Sabha TV with Ajay Chhibber, Director General, Independent Evaluation Office, Government of India; Thoriq Ali Luthfee, Minister of Health, Maldives and Biraj Patnaik, Principal Advisor, Right to Food in the office of Commissioners to Supreme Court. Anchor: Girish Nikam
Two 3ie-supported studies cited in the Economist. These include Girl power: cash transfers and adolescent welfare. Evidence from a cluster-randomized experiment in Malawi, by Sarah J. Baird, Ephraim Chirwa, Jacobus de Hoop, Berk Özler and Relative effectiveness of conditional and unconditional cash transfers for schooling outcomes in developing countries: a systematic review, by Sarah Baird, Francisco H. G. Ferreira, Berk Özler, Michael Woolcock.
Girl power: cash transfers and adolescent welfare. evidence from a cluster-randomized experiment in Malawi
This 3ie-supported study now has a paper in the NBER working paper series.
This study summarises evidence from short-term impacts of a cash transfer programme on the empowerment of adolescent girls in Malawi during and immediately after the two-year intervention.
Getting children into school is only part of the education battle. We must also ensure they learn once they are there. Blog by Howard White in the Guardian
The University of East Anglia is offering a one-year MSc course in Impact Evaluation in International Development.
The course has been designed for students who are interested in designing and implementing development projects and programmes and/or in researching development effectiveness, and who need to develop and enhance their skills for undertaking high-quality rigorous impact evaluations.
3ie is offering a scholarship for this course.
This 3ie-supported study now has a paper in the NBER Working Paper Series.
This study evaluates the impact of a programme that provides pre-fabricated housing to members of extremely poor population groups in Latin America. Findings show that a positive effect on general well-being of the people. There is also evidence of improvement in children's health in two countries.
A 3ie study on the impact of daycare in Brazil shows that it has the potential to improve household welfare especially for the poor. Household income was went up (8%), as did the labour supply of the carer (usually the mother). Children also fared better in terms of cognitive and anthropometric outcomes.
This paper by Marie Gaarder and Jeannie Annan argues that it is both possible and important to carry out impact evaluations even in settings of violent conflict, and it presents some examples from a collection of impact evaluations of conflict prevention and peacebuilding interventions. This Policy Research Working Paper is published by the Independent Evaluation Group, World Bank.
The Campbell Collaboration training videos were recorded at the May 2013 Campbell Colloquium. The introductory videos are intended for researchers conducting systematic reviews as well as for policymakers interested in evidence-based policies. The advanced methods / applied topics videos are for researchers carrying out systematic reviews.
Do conditions moderate the effects of cash transfer programs? Preliminary findings from a systematic review
The preliminary findings from the systematic review indicates that both conditional cash transfer (CCT) and unconditional cash transfer (UCT) programmes improve school enrolment: 23% in the case of UCT programmes and 41% for CCT programmes.
An impact evaluation of the loans offered by Compartamos, the largest microlender in Mexico shows that loan recipients grew their business revenues and expenses, were happier, more trusting, had greater household decision power, and were better able to manage liquidity and risk. However, there was little evidence that loans had an impact on building wealth like household income, business profits, or consumption.
The study adds to the evidence that microcredit is generally beneficial, but not necessarily transformative.
The Research Department of Agence Française de Développement (AFD) has published the Ex Post collection of the following papers: Insuring health or insuring wealth? An experimental evaluation of health insurance in rural Cambodia and Sky impact evaluation, Cambodia, 2010, village monographs
Ian Ramage and David Levine, co-authors of these studies will be speaking at the Health Financing in Developing Countries Conference (Financement de la santé dans les pays en développement) on 27 and 28 May at AFD.
This report gathers together examples which attempt to explain how and why change happens as a result of cash transfers (CTs). It first presents a selection of theories of change about how cash transfers are expected to work in general, drawn from academic literature, and then a selection of theories as used in a few specific cash transfer programmes.
Pocast of presentation by Jyotsna Puri at the 28th Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) meeting.
In a video interview, Phil Davies talks about the importance of timing for researchers wanting to engage and inform policymakers. He talks about 3ie GapMap as a visual and engaging tool for understanding what is known and what isn't.
In Part 2 of the video, he stresses the importance of physical access, encouraging efforts towards ensuring that policymakers are aware of, and can get hold of, the kinds of evidence that they need to make good decisions.
A DFID guidance note on the best ways to assess evidence in international development. It offers some rules on:
- understanding different types of empirical research evidence
- appreciating the principles of high quality evidence
- considering how the context of research findings affects the way that staff may use them
- understanding how to make sense of inconsistent or conflicting evidence
Findings from a 3ie-funded study on the impact of a conditional cash transfer programme in Nicaragua cited in this Inter-American Development Bank (IADB) blog.
Theme: What evidence-based development has to learn from evidence-based medicine
Speaker: Chris Whitty
Theme: What we have learned from 3ie's experience in evidence- based development
Speaker: Howard White
Howard White discusses different designs of randomized control trials and addresses criticisms of RCTs which are mostly argued to rest on misunderstandings of the approach.
The standard approach to policymaking and advice in economics implicitly or explicitly ignores politics and political economy, and maintains that if possible, any market failure should be rapidly removed. This NBER working paper by Daren Acemoglu and James A. Robinson argues why this conclusion may be incorrect.
This CGD Working Paper by Tessa Bold, Mwangi Kimenyi, Germano Mwabu, Alice Ng'ang'a, and Justin Sandefur looks at how a Kenyan education programme proven successful in a randomised controlled trial, failed to have similar outcomes when implemented.
See blog by Justin Sandefur Finding what works in development: what is the what?)
Esther Duflo discusses a 3ie-supported study on Gujarat pollution control in an interview to Yale Insights.
Markus Goldstein highlights the findings from two 3ie-supported studies -- a systematic review on the impact of daycare programmes and the impact evaluation of Save the Children's early childhood development programme in Mozambique.
"The purpose of this document is to advance the Foundation’s existing work so that our evaluation practices become more consistent across the organization.We hope to create more common understanding of our philosophy, purpose,
and expectations regarding evaluation as well as clarify staff roles and available support."
Howard White, 3ie Executive Director, discusses “closing the evidence gap”, randomised control trials, and the value of impact evaluation with Dereck Rooken-Smith of ODE, AusAID.
Louise Shaxson, Research Fellow, ODI on how "Pressure to demonstrate concrete impacts on public policy is encouraging researchers to make grand claims about what we/they are likely to achieve."
Rosalind Eyben, Research Fellow at IDS and Chris Roche, Associate Professor, La Trobe University kick off a discussion on the implications of evidence-based approaches.
Nina Cromeyer Dieke shares tips and lessons development community can learn from mainstream media. The 3ie policy impact toolkit finds a mention in the story.
Berk Ozler on the pros and cons of using surveys to measure impact.
"...So, evaluating a large government program using an unrelated routine government survey may be fine (although I suspect that they too will have biases depending on what the respondents think the survey is for, how large, important, and ‘in the news’ the intervention is, etc.), but evaluating your own experiment that aims to change some behavior by asking study participants whether they have changed that behavior is unacceptable."
How can systematic reviews contribute evidence for policy? Blogs on this page take up the debate on conducting and using systematic reviews.
Tracey Koehlmoos, adjunct professor at George Mason University, Washington DC, and adjunct scientist at ICDDR,B blogs on sessions at the Dhaka Colloquium on Systematic Reviews.
"...Perhaps the most controversial session that I have attended so far was provocatively named “Rapid reviews: opportunity or oxymoron?” 3ie deputy director, Phil Davies presented on “rapid evidence assessment” and their place in the pantheon of evidence synthesis efforts aimed at informing policy making. Serious questions remain about rapid reviews being biased compared to systematic reviews—and how the process would even allow the developers of these reviews to recognize any biases. However, Davies pointed out that “all evidence is probabilistic.” ..."
"This discussion paper aims to support appropriate and effective use of impact evaluations in AusAID by providing AusAID staff with information on impact evaluation. It provides staff who commission impact evaluations with a definition, guidance and minimum standards."
3ie recently participated at an IFAD learning event on impact evaluations for environmental and climate change interventions.
William Savedoff blogs on the challenge that impact evaluation poses for organisations. "...Other times, the concerns reflect an unwillingness to clearly state their goals, be explicit about their theories of change, or put their beliefs about what works to an objective test. Yet, this is exactly what is at stake with evaluation: are you willing to be proven wrong?"
"Potential biases can arise when collecting qualitative data, in deciding which questions are asked, in what order, how they are asked and how the replies of the respondents are recorded.
"There can also be biases in how the responses are interpreted and analyzed, and finally which results are chosen for presentation. Of course quantitative data and analysis is also prone to bias, such as sampling bias and selection bias. But methodologies have been developed to explicitly deal with these biases. Indeed evaluation designs are judged on precisely how well they deal with these biases."