Replication research promotes open discourse

USAID

The just-released International Journal of Epidemiology (IJE) suite of publications reexamining the effectiveness of deworming in Kenya demonstrates the potential impact of replication research. The headline publication is a 3ie-funded replication study. The paper has been published alongside three additional commentaries: a synopsis of a systematic review of deworming evidence, a response from the original authors, and a response from the replication researchers. The publication of these papers in a respected journal puts the role of replication squarely where we think it needs and deserves to be to promote valuable public discourse around highly relevant evaluation evidence. We’re excited about these publications for a number of reasons.

First, replication research is being published! From 3ie’s perspective, this is an accomplishment onto itself. We designed the Replication Programme to help change incentives. In order to encourage more replication papers that will improve the evidence base for policymaking, we need to convince researchers that publication outlets exist for these time-intensive replication studies. The IJE arguably has the highest impact within the field of epidemiology (see information on IJE’s impact factor here). The editors’ decision to publish these papers is a testament to the value they see in replication research. This in turn helps change publication incentives.

Second, the replication studies have sparked a larger conversation around the existing deworming evidence. We’re discovering that a significant grey area exists regarding the ability of replication researchers to recreate the originally published results. However, these replication studies cannot be simply lumped into difficult to define successful or failure categories. We find that replication studies provide researchers with a valuable space to discuss analytical decisions and the robustness of publication results. We believe that these discussions improve the science around these evaluations, which in turn enhances the quality of the evidence on which policymakers rely to spend limited development funds.

Third, these conversations are public, which allows for scrutiny of the findings and a general discussion of the research. Miguel and Kremer helpfully provide their data for replication efforts here and a replication guide for their original paper here, both of which ease the process for replication researchers to reproduce their paper. The original authors have been very open with assisting researchers interested in reanalysing their results (as an example, here’s a report on GiveWell’s replication study). But most of the subsequent replication studies of their original deworming paper don’t appear to be widely circulated or publicly posted. That has now changed with the publishing of the replication results and these commentaries on the deworming evidence in the IJE. The discourse is now open for interpretation by everyone, from researchers and policymakers to funders and implementers. Regardless of which side of the deworming debate one falls, some facts remain irrevocable. The revised tables in the original author response to the replication study correct for agreed upon errors in the original publication. This clearly demonstrates the power of replication research.

Ultimately, 3ie, through our Replication Programme, seeks to change the incentive structure around replication research. If the reanalysis process becomes standardised and journals agree to publish this type of research, there will be a genuine opportunity for researchers to provide more robust evidence for policymaking. This open discourse around replication results will help normalise the replication process. We’re hoping to see more of these open discussions in the future.

 

Video: Benjamin DK Wood talk about how the publishing of this 3ie-funded study opens up the discourse on replications of impact evaluations.

 

Replication studies:

Re-analysis of health and educational impacts of a school-based deworming programme in western Kenya: a pure replication

Re-analysis of health and educational impacts of a school-based deworming programme in western Kenya: a statistical replication of a cluster quasi-randomized stepped-wedge trial

Original author’s reply:

Commentary: Deworming externalities and schooling impacts in Kenya: a comment on Aiken et al. (2015) and Davey et al. (2015)

Replication researchers’ response:

Authors’ Response to: Deworming externalities and school impacts in Kenya

Systematic review commentary:

Replication of influential trial helps international policy

Add new comment

About

Evidence Matters is 3ie’s blog. It primarily features contributions from staff and board members. Guest blogs are by invitation.

3ie publishes blogs in the form received from the authors. Any errors or omissions are the sole responsibility of the authors. Views expressed are their own and do not represent the opinions of 3ie, its board of commissioners or supporters.

Archives

About

Evidence Matters is 3ie’s blog. It primarily features contributions from staff and board members. Guest blogs are by invitation.

3ie publishes blogs in the form received from the authors. Any errors or omissions are the sole responsibility of the authors. Views expressed are their own and do not represent the opinions of 3ie, its board of commissioners or supporters.

Archives