Mapping the evidence on the effects of rule of law interventions on justice outcomes

Mapping the evidence on the effects of rule of law interventions on justice outcomes

The systematic use of evidence can help inform decisions on what research to conduct, what interventions to implement and how to improve the effectiveness of programs. The development of an Evidence Gap Map (EGM), which collates the best available impact evaluation and systematic review evidence on a topic, is often the first step in this process.

This is why the US Agency for International Development (USAID)’s Center of Excellence on Democracy, Human Rights, and Governance (DRG Center) has commissioned 3ie and NORC at the University of Chicago to produce a series of six EGMs, one for each of the DRG Center’s program areas under their Foreign Assistance Framework. The EGMs are intended to support USAID in developing an evidence-informed understanding of effective approaches in these programmatic areas, and identify areas where further research is needed. Beyond providing a tool for informing policy and practice, the EGMs will inform the formulation of questions for the DRG Center's Learning Agenda that supports building the sector's knowledge and evidence base to inform USAID programming.

The six program areas are: rule of law; human rights; civil society; independent media; good governance; and political competition. 3ie will work to produce these maps over the course of two years. The timing of the EGMs has been designed to coincide with DRG Center policy processes, to better inform decision-making in each of these areas. The maps will help identify high-quality, relevant impact evaluations and systematic reviews that exist, and identify critical knowledge gaps where no rigorous evidence on the effects of these interventions exists. First up: mapping the evidence for interventions aiming to strengthen rule of law and access to justice.

Why do we need an EGM of rule of law interventions? Because rule of law is essential to promoting a just and inclusive society

Effective rule of law ensures that laws and the justice institutions that support them protect individual rights, and are responsive to and inclusive of, the needs of all people in society. It also plays an important role in contributing to economic, political and social stability, and is increasingly considered a key mechanism to ensuring that people access their rights and benefit from inclusive growth (OECD 2019).

An estimated 5.1 billion people - almost three quarters of the global population - have no access to effective justice (Task Force on Justice, 2019). These problems manifest in varied ways. For example, people may be unable to access justice due to a lack of legal identity. For others, structural inequalities may be reflected in and or made worse by lack of access to justice, such as for women, people with disabilities and displaced people (OECD 2019). Unfortunately, the global support for programs promoting rule of law and access to justice is falling, despite the continued need. Compared to other sectors, there are particularly large funding gaps in support for the justice sector, which receives only 1.8% of donor-funded aid from OECD DAC donors, and even this limited support has fallen by 40% in the past four years (Manuel et al., 2019).

By facilitating access to the rigorous evidence base for the justice sector, the rule of law EGM can help policymakers and practitioners strengthen the effectiveness of rule of law programming by applying an evidence-informed approach. Through enabling easy identification of relevant studies, the EGM can ensure new policies and programs are designed based on lessons and implications from existing evidence. The EGM will further identify gaps in the evidence base, where the effectiveness of key interventions is not well understood. Together, these contributions of the EGM can help make the case for new investments by enabling policymakers and practitioners to demonstrate that new policies and programs are informed by the best available evidence, building off of what is working, learning from what isn’t, and innovating and testing new approaches.

How do we find all of the evidence? Through an exhaustive search and screening process

We have already developed a plan for implementing the map, and we are in the process of searching for and screening potentially relevant literature. This process includes searching a number of academic databases as well as a range of grey literature sources, such as institutional websites. After systematically screening each study, we collect key information about all studies that meet the inclusion criteria and produce summary statistics to understand the evidence base in more detail. We will also map out gaps in the evidence. We plan to produce provisional results from the map by the end of October 2020, and final results will be made available on the 3ie Development Evidence Portal.

We need your help: Do you know of relevant studies that will make the EGM more comprehensive?

Table 1 (below) summarizes the inclusion criteria for the EGM. So far, we have identified over 500 relevant studies to include in the EGM - but we want more! To help us make sure our map is comprehensive, and to make sure our literature search includes all avenues, we would love your help.

Identifying relevant studies that exist in the ‘grey literature’ - everything published outside of academia or not published at all - can be especially difficult, yet that’s also where a lot of knowledge is held. If you’re aware of an impact evaluation or a systematic review focusing on rule of law or access to justice, please get in touch and join us in creating as comprehensive a picture of the evidence base as possible. Much of the existing evidence is from high income countries, so we are especially interested in any studies conducted in low or middle income countries.

As a thank you for your effort, anyone who identifies a study that meets our inclusion criteria, which we have not already picked up through our systematic search process, will be entered in a draw to win a £50 gift card.

Please send your submissions to our evaluation specialist, Ada Sonnenfeld, at asonnenfeld@3ieimpact.org. We’ll be accepting studies until 18 October, 2020.

Study characteristics Inclusion criteria
Population All populations are eligible for inclusion in this EGM of the global evidence base
Interventions

The EGM covers interventions (policies, programs, and projects) that aim to improve aspects of access to and execution of rule of law and justice. We include interventions across three primary domains:

  • Interventions that aim to create and/or strengthen formal and informal institutions of justice and frameworks of laws, including through the introduction of participatory processes, institutional accountability mechanisms, changing laws to meet international human rights standards, strengthening selection and certification processes and capacity-building for justice sector actors, or supporting linkages between informal and formal institutions.
  • Interventions that aim to strengthen the quality and effectiveness of legal and justice services at the point of delivery or interaction with members of society, including efforts to facilitate access, uphold the law and prevent crime, rehabilitate ex-offenders, or provide social services, support and protection to victims.
  • Interventions that aim to empower members of a society to strengthen rule of law and justice, such as through building knowledge of laws and rights, legal registration, building civil society and media capacity to support rule of law and access to justice, and community monitoring of justice sector actors.
Comparison We include impact evaluations that estimate effects by assessing outcomes for targeted individuals against a valid counterfactual, such as a comparison group that received no intervention, a different or placebo intervention, or those on a wait-list to receive the intervention.
Outcomes To be included in the EGM, studies must report at least one key outcome related to rule of law or access to justice, including prevention or solutions to justice problems, or attitudes, behaviours and coverage of justice institutions, services and individuals.
Study designs

We include two types of studies in the EGM: impact evaluations and systematic reviews.

  • Impact evaluations: We include primary studies that use rigorous methods to provide a quantitative estimate of the impact of an intervention.
  • Systematic reviews: We include effectiveness reviews that synthesize the available evidence for the effects of a particular intervention on outcomes of interest, based on an exhaustive search of the literature, clearly pre-defined criteria for inclusion, and strategies to critically appraise the evidence base to minimize errors in interpreting the results.


References

Domingo, Pilar (2016). Rule of Law, Politics and Development. The Politics of Rule of Law Reform. Overseas Development Institute: London. At: https://www.odi.org/publications/10376-rule-law-politics-and-development-politics-rule-law-reform

Manuel, M., Manuel, C. and Desai, H. (2019). Universal access to basic justice: Costing Sustainable Development Goal 16.3. ODI Working Paper 554. London: Overseas Development Institute. At: https://www.odi.org/publications/11347-universal-access-basic-justice-costing-sustainable-development-goal-163

OECD (2019). Equal Access to Justice for Inclusive Growth: Putting People at the Centre, OECD Publishing: Paris. At: https://www.oecd.org/gov/equal-access-to-justice-for-inclusive-growth-597f5b7f-en.htm

Task Force on Justice (2019). Justice for All. The Report of the Task Force on Justice. Pathfinders for Peaceful, Just and Inclusive Societies. New York: Center on International Cooperation. At: https://www.justice.sdg16.plus/

Add new comment

Authors

Ada Sonnenfeld Ada SonnenfeldFormer Evaluation specialist, 3ie
Birte Snilstveit Birte SnilstveitDirector – Synthesis & Reviews and Head of 3ie London Office
Lina Khan Lina KhanResearch assistant consultant, 3ie
Anna FoxFormer programme associate, 3ie

About

Evidence Matters is 3ie’s blog. It primarily features contributions from staff and board members. Guest blogs are by invitation.

3ie publishes blogs in the form received from the authors. Any errors or omissions are the sole responsibility of the authors. Views expressed are their own and do not represent the opinions of 3ie, its board of commissioners or supporters.

Archives

Authors

Ada Sonnenfeld Ada SonnenfeldFormer Evaluation specialist, 3ie
Birte Snilstveit Birte SnilstveitDirector – Synthesis & Reviews and Head of 3ie London Office
Lina Khan Lina KhanResearch assistant consultant, 3ie
Anna FoxFormer programme associate, 3ie

About

Evidence Matters is 3ie’s blog. It primarily features contributions from staff and board members. Guest blogs are by invitation.

3ie publishes blogs in the form received from the authors. Any errors or omissions are the sole responsibility of the authors. Views expressed are their own and do not represent the opinions of 3ie, its board of commissioners or supporters.

Archives