The chasm between policymakers and researchers is frequently observed but seldom addressed. A little over two weeks ago, 3ie organized a matchmaking market place to bridge this gap between research and policy. The setting for this was the Dhaka Colloquium for Systematic Reviews in International Development. The people who came together for this innovative matchmaking exercise were both ‘users’ and ‘doers’ of systematic reviews.
Systematic reviews are exhaustive, unbiased reviews of evidence in a specific area, that summarize, assess and present the state of the evidence from published/unpublished literature. Systematic reviews can help to explain what works, under what circumstances and why.
There are many attributes of systematic reviews that can be useful to policymakers or other users of systematic reviews such as governments, donor agencies, international development agencies and large NGOs. Since systematic reviews are exhaustive reviews of all evidence on a specific topic and are unbiased, they inform all aspects of an intervention including unintended consequences. In many cases where data is available, they use meta-analyses statistical techniques to pool data across all studies and provide a sense of the ‘net’ effect size. Good systematic reviews can thus be very good tools to analyze the effectiveness and cost-effectiveness of development interventions. They can help inform the policy emphasis of a government or a programmatic strategy within an organization.
But putting together systematic reviews is a skill intensive and time consuming task. Hence specialized teams are required to do these. A good systematic review team requires people who’ve done systematic reviews earlier, information specialists, sector experts and very importantly, research assistants who can spend hours and hours poring over papers from all sources (published, grey literature, project documents, design documents and anything else).
At the matchmaking clinic that 3ie organized, around 10 researchers (doers) and 30 possible policymakers (users) participated. They split up in groups, each with some users and some doers. Users presented possible questions that would be useful and relevant to them. Doers discussed steps to produce a policy relevant systematic review. Presentations by mixed teams at the workshop spanned a variety of areas ranging from interventions targeting better irrigation, environmental health, adaptation by farmers in the face of increased weather variability, micro-credit and schemes to increase farmer productivity in Malawi.
There are many steps involved in producing a systematic review. The first step is to identify the question and the population of interest/relevance. Knowing which interventions are going to be examined and who the main stakeholders of interest are, is key. Next, articulating the theory of change and eligible interventions is important. Since systematic reviews cover evidence of impact, understanding comparators, outcomes and the settings is the next step. Once these areas have been defined, a gap map can be put together. A gap map consolidates what we know about ‘what works’ in the sector of interest by drawing out evidence from systematic reviews and impact evaluations. (3ie’s London office has state of the art expertise in systematic reviews and has been working on systematic reviews and refining techniques for gap maps.)
Once the question has been identified, a fairly intensive and long review of the literature is undertaken. A 3ie supported systematic review that assesses the effectiveness of community driven interventions to reduce maternal and neo-natal mortality can provide a good example. In the first step, Haider, Lassi and Buta collected more than 30,000 studies using keyword searches in a variety of libraries. Inclusion criteria and exclusion criteria were used to select within these studies. Inclusion criteria included geographic area, target population, types of intervention and study methods. Studies included in this systematic review included interventions in Africa or Asia, included studies of interventions that were specifically community driven, were undertaken for women in situ (and not in a location outside of the community), and had to include members of the community. Included programmes needed to increase skills of community members in providing support to maternal and neo-natal care. The programme also needed to be a package of interventions that ranged from ante natal, natal and post-natal interventions. Therefore single interventions such as those that solely targeted resuscitation or Vitamin A provision were excluded. Studies that used robust identification strategies such as community based randomized, quasi-randomized and prospective time series were included.
From a population of more than 30,000 studies, 109 were selected for a detailed evaluation. An independent assessment of data quality and particularly bias, attrition, heterogeneity and sub-group analysis resulted in 27 studies being included finally in the systematic review.
The systematic review concluded that overall community based interventions (that included additional training for lady health workers, community mid-wives, village health workers, facilitators and traditional birth attendants) included in the study showed no significant impact on reducing maternal mortality. However it showed that community based intervention packages are associated with significant reductions in neonatal mortality. The systematic review showed that a combination of training in antenatal, natal and post natal care, preventive essential newborn care, breastfeeding counseling, management and referral of sick newborns, skills developing in behavior change communication and community mobilization to promote birth and newborn care preparedness, reduced neo-natal mortality by 27 per cent. This impact was 6 per cent higher in studies that included both therapeutic and preventive care, compared to those that just included preventive care.
Clearly these conclusions can be very useful for overly stretched health ministries that are frequently under-resourced and continually lack good health workers. In developing countries, nearly two-thirds of births occur at home and only half of all births are attended by trained birth attendants. Of the 136 million births every year, more than 7 million are still births or neo-natal births. 98 per cent of these are in developing countries. Community based interventions in developing countries that can reduce this percentage by more than one-fourth are clearly a cost-effective strategy and a potential win-win.
The Dhaka Colloquium featured many ideas for systematic reviews that have the potential for significant policy impact. The workshop concluded two main things: Many more matches between doers and users are needed, and, in many areas, many more impact evaluations (that go onto populate systematic reviews) are required. 3ie will continue to support both of these over the years.
(For those who are interested, 3ie’s newest thematic window on climate change and disaster risk reduction will soon put out a call for systematic reviews in the area of mitigation, adaptation and risk reduction. So watch this space.)