Six major upgrades to the evidence architecture: making evidence easily available and accessible

Easy access to evidence does not equal use. But if we are to realise the potential of evidence to inform policy and change lives, access to reliable sources of evidence is a necessary condition.

COVID has highlighted the challenges of the ‘infodemic’. Our inboxes, social media and the internet are flooded with information about the most recent research. Whether it is about COVID, immunization, livelihoods or nutrition, all evidence and information are not created equal, and for both researchers and decision-makers it is time consuming and difficult to separate the wheat from the chaff.

We at 3ie are tirelessly working to build an evidence architecture that makes it easier to access good and reliable evidence, which will help make better decisions for the planet, for the people. A key part of that architecture is the Development Evidence Portal. Those of you that follow our work know that since our inception we've hosted databases of impact evaluations and systematic reviews.

Over the last three years we've worked to upgrade these resources, bringing them together in a user-friendly interface where you can access all impact evaluations, systematic reviews and evidence gap maps (EGMs) focusing on the effects of interventions in low- and middle-income countries (L&MICs) across sectors. This gold mine of evidence now contains over 10,000 studies. In this first in a series of blogposts on the DEP, we outline the key features of our upgrade:

1)    Over 10,000 studies on the portal. Our efforts to update the DEP have reached a major landmark. With over 10,000 studies available, it is now the world’s largest collection of evidence on the effects of programs designed to improve lives in L&MICs. Till a few weeks ago, the DEP was current through 2016 and housed just around 2000 studies. This year, we implemented an extensive search of the academic and grey literature, sifting through 85,000 records to identify studies that meet our inclusion criteria. To learn more about where we search and what we include, you can access the search and screening protocol.

2)    Transitioning to a system of continuous evidence surveillance. Users will now benefit from  a ‘living’ Development Evidence Portal, updated as new studies become available. To date, the DEP has been updated through periodic searches every two-three years. Now we run searches every month to ensure the DEP remains current. Not only does this provide our users with access to the latest evidence, but it also supports our work in other ways. For example, we will be continually updating our Food Systems and Nutrition Evidence Gap Map as our first “living” EGM, a process facilitated in large part by the evidence surveillance we are conducting for DEP.

3)    Enhanced data extraction protocol, including data on equity, funders and the SDGs. The latest update to the portal includes the addition of detailed metadata for each study, allowing users to filter evidence across a number of categories, including sectors, regions, countries, and OECD-DAC codes. Our equity coding protocol captures whether and how studies pay attention to gender and equity considerations (spoiler alert: they very often don’t).

Note that while we have extracted all of the data for studies that existed in DEP before this year’s update, we are still in the process of doing this for the studies added this year. So you will notice that many of the studies published during 2017-2021 are “skeleton” records with just basic information. But we are hard at work extracting data from these studies, so we encourage you to check back frequently.

4)    New intervention-outcome taxonomy to filter evidence in a much more specific way. One of the new features we’re most excited about is our intervention-outcome taxonomy, which allows users to filter evidence by interventions and outcomes of interest. Through extensive consultations with experts in different sectors across the development space, we’ve developed a comprehensive classification system for the interventions evaluated and the outcomes measured in DEP studies. This allows users to find evidence that’s relevant to their questions in a much more specific way than has been possible in the past. For examples, if you’re interested in studies measuring the effects of access to microcredit on empowerment outcomes, you can use the filters to find exactly those studies (there are 19 of them at the moment, if you’re curious).

5) Methodological appraisals of impact evaluations. Another upcoming feature is methodological appraisals of impact evaluations. We’ve long conducted detailed critical appraisals of systematic reviews and made our confidence ratings (high, medium, or low) available to DEP users. The scale of the impact evaluation literature is such that we can’t conduct such detailed appraisals of those studies. But we are piloting a tool for making quick assessments of potentially major threats to the validity of a study’s findings. Once these assessments are available on the portal, they’ll provide an additional way for users to find the most relevant and the most trustworthy evidence.

6)    Supporting toolkits and guidelines. Because of the way it assembles the best and most recent evidence, the DEP provides the basis for decision-support products such as toolkits and evidence-informed guidelines. We are in the early stages of piloting our first product along these lines. Keep an eye on this blog for updates.

We consider this just the beginning. The DEP helps provide the foundation we need for an evidence architecture that helps facilitate evidence-informed decision making. For example, the DEP can provide the starting point for evidence brokering efforts such as rapid response services. As part of our West Africa Capacity-building and Impact Evaluation Program (WACIE), our helpdesk provides rapid evidence response and translation to help policymakers in West Africa answer targeted policy questions and design effective and cost-efficient development programs.

In future posts we will delve deeper into the key features of the DEP, offer analysis of what the DEP data tells us about the state of the evidence and share ideas about our plans and ambitions. In the meantime, please tell us what you think in the comments section! To know more about 3ie’s DEP, write to us at info@3ieimpact.org.