For decades, development actors have recognized that good governance and solid country systems are essential for the effective delivery of well-designed, evidence-based interventions. A number of development organizations and initiatives have, therefore, emerged with explicit missions to strengthen the capacity and performance of publicly-mandated political and administrative institutional structures (for e.g., World Bank’s Global Governance practice and USAID Center for Democracy, Human Rights, and Governance). However, despite widespread investment in such policy and institutional reform (PIR) interventions, and their centrality to sustainable development outcomes, their impacts are not always measured or known.
PIR interventions exist in many forms, including establishing or improving laws or regulatory frameworks, capacity building or technical assistance, and development of information management systems—with the purpose of increasing institutional strength, improving processes, efficiency, quality, accountability, transparency, and management, among others. Evaluation of such interventions is inherently challenging for many reasons. The unit of observation(s) or intervention scale may be impossible to power statistically or have no identifiable counterfactual. Desired outcomes—such as increased accountability, transparency, good governance, role delineation—are often qualitative or descriptive, which makes measurement difficult. Theories of change may be challenging to map due to deviations in stated or unstated rules that govern behavior (e.g., mandates, incentives, contextual factors, competing interests, or policy cycles). Causal pathways may be multi-level, non-linear, distal, and/or unpredictable. Finally, change is often dependent on external factors beyond the direct control of an intervention.
Experts have been calling for complexity-responsive approaches to evaluation as traditional or single evaluation methods may insufficiently address the challenges above. Without proper accounting for these dynamics, important intermediate results and outcomes may be unmeasured and unaccounted for, leading to incorrect conclusions that a PIR intervention added low or no value.
MCC-3ie scoping review of policy and institutional reform methods
The Millennium Challenge Corporation (MCC) and 3ie have partnered to improve the design, monitoring, and evaluation of PIR interventions. 3ie’s mission is to improve understanding of what works, for whom, why, and at what cost to promote sustainable social and economic development. MCC’s PIR Working Group promotes development effectiveness and learning through MCC’s improved understanding and engagement with institutions.
Through this partnership, 3ie is conducting a scoping review that seeks to identify established methods for assessing attribution and plausible contribution of PIR intervention activities to outcomes, describe change and results, and/or measure impact. Findings from the scoping review will inform a methods menu tool that facilitates MCC’s understanding and adoption of appropriate evaluation methods. While the study’s focus is on summative evaluation, it will also seek to generate insights on prospective institutional root causes and intervention complexities to be accounted for during programmatic design and implementation.
Our scoping review research aims to:
- Identify the key social science research methods which have been used (or proposed) to evaluate causal effects and/or establish plausible contribution of policy and institutional reform intervention(s) to changes in outcomes
- Identify methods to evaluate interventions with complex elements
- Understand how these methods have been applied to PIR interventions
- Identify the types of indicators, outcomes, or data to be collected when evaluating a PIR activity
This research will inform MCC’s core evaluation questions by providing insight into plausible contribution of results and outcomes and increasing understanding of complex intervention elements and methods of assessing impact. Emergent findings from the scoping review will be published by June 2022 through 3ie’s Development Evidence Portal.
Call for additional resources and studies
As we are identifying the relevant literature for this review, we appreciate any suggestions, particularly those belonging to the grey literature, to ensure a comprehensive scoping study. We are seeking research papers that encompass the below criteria:
- The intervention pertains to policy and institutional reform: interventions that are focused on institutions that serve a public mandate
- Evaluation methods with explicit focus on understanding the causal mechanisms, merit or worth of the results of the intervention, either ex ante or ex post
- Empirical literature or operational guidance that demonstrates how method(s) have been robustly applied in low- and middle-income country contexts.
Please send your study suggestions to Zeba Siddiqui at zsiddiqui@3ieimpact.org. We will be accepting studies until 1 March 2022.
For more information about MCC’s PIR working group, please contact Julian Glucroft at glucroftjs [at] mcc [dot] gov.
