How to read a systematic review: A quick guide for policy practitioners
Here at 3ie, we're big supporters of systematic reviews, which combine evidence from multiple different studies addressing the same type of research question. This strategy provides stronger evidence than relying on a single case, where idiosyncratic issues can affect program outcomes. We've conducted dozens of systematic reviews on a broad range of international development topics, and our evidence repository contains hundreds more. When a good systematic review is available, it represents the best possible way to evaluate the evidence about a given question.
So how should they be used in practice? Let's consider a hypothetical example in which the ministry of education of a middle-income country is under pressure to raise students' test scores. (And let's assume its staff hasn't seen the helpful blog post we wrote last week.)
Where to start?
The first step is to find a relevant systematic review. 3ie's Evidence Hub is a great place to do that. Type in a few key words in the search field (e.g. "education test scores" or "improve learning."), filter the reviews by sector (e.g. "education"), or both. In this example, all these searches turn up dozens of reviews. This database is getting an overhaul with new features next month, so we'll have more about search functions then.
Next, read through the titles. Systematic review titles tend to be dry—but they often say exactly what's inside. In the search results here, we see a few that might be relevant. The first result:
That review might provide some useful information, but it is narrowly focused on cash transfers. Since the ministry's question is broader, keep scrolling:
This review is spot on. It compares the effects that different interventions have on learning outcomes (i.e. test scores), and it is focused on the right context: low- and middle-income countries.
We found one. What now?
The next step is quality control, because not all systematic reviews are equally good. One way to do this is to look at the source. At 3ie, we can vouch for all reviews we conduct ourselves. That's the case with the review in this example, so our imaginary ministry staff would skip down to the next section.
We also evaluate the quality of others' systematic reviews using this quality appraisal checklist and provide a summary of our appraisal. When we launch the newly updated repository site next month, the quality ratings of low, medium, or high will be also be provided – making them much easier to find.
To make a quick judgement for yourself, take a look at the 'Study Selection Criteria' and 'Included Studies' sections of the review's executive summary. A good review will have included a wide-ranging search for relevant articles, and should have found and included a sizeable number of separate studies — if the evidence exists, that is.
OK, we've got a relevant, high quality review — but its 891 pages!
Yes, systematic reviews can be very long, because they synthesize a lot of research.
Keep it simple! Reading the 'Results,' 'Conclusions,' and 'Policy Implications' sections of an executive summary should provide a good breakdown of any review's findings. We also write policy briefs based on some systematic reviews, as we did for the education review here, in addition to summaries of some reviews written by others. We'll be adding more of these summaries soon to our revamped repository.
Also, take note of the confidence the authors place in different findings, which often depends on how many studies they found evaluating each type of intervention.
Where does this tell us how big the effect will be?
In our example, this question would be: How much will test scores go up if we implement the pedagogy programs? Unfortunately, there isn't a simple answer.
Systematic reviews in the social sciences are good for identifying which types of interventions consistently produce measurable effects and which do not. Comparing the sizes of these effects is much more complicated, because the review analysis standardizes findings from disparate contexts. So unless you want to dive in to understanding standardized mean differences, just read the text of the executive summary and use the researchers' descriptions of their own findings. We also write policy briefs for some of our reviews, like this brief on the education review discussed above, that can provide a useful breakdown of results.
I couldn't find a systematic review that answers my question. Why is that?
Systematic reviews can't address all kinds of questions, because they require a solid base of individual studies to build on. For research questions that are specific to a particular location, or for questions about contexts where impact evaluations are seldom conducted, there is unlikely to be a systematic review. But as impact evaluations become more common, the base of individual studies grows. If there isn't yet a systematic evaluation of a particular question, that's all the more reason to support the development of impact evaluations, so one day there will be one.
This is the fourth blog in our campaign 2020 Hindsight: What works in Development. Learn more about the campaign and read past blogs here.