Push-Button Replication

3ie’s push-button replication (PBR) is a project to confirm the validity of published results using both the original data and the programming code from a study. In addition to this project, 3ie also conducts push-button replications of all data sets emerging from 3ie-funded studies. To know more about this visit our page on research transparency. 

What is push button replication?

PBR research attempts to confirm the validity of published results using both the original data and the programming code from a study. The premise behind a PBR study is that the third party researcher should not need to make any significant adjustments, write new code or conduct additional analysis in order to arrive at the published results. Thus, a PBR is a step that precedes pure replication.
While a pure replication asks the question can we reproduce the published results using the original data and the methods described in the original study, a PBR tries to verify if we can use the original authors’ programming code with the original data to reproduce the published results. Through pure replication studies, researchers can potentially uncover errors where the programming code incorrectly implements the estimation methods described by the researchers of the original study. PBR studies, on the other hand, can help reveal instances where there are no or insufficient data and code available or where the programmes do not run.

Objectives

  • Establish procedures and standards for PBR so that original authors and replication researchers can better align their expectations and actions around this third-party verification process
  • Test whether development impact evaluations are generally push-button verifiable

How are PBR studies conducted?

  • Candidate studies were chosen from a set of articles selected from across all development impact evaluations published in the top ten journals for development impact evaluations in 2014. The sample size was established at 109 studies.
  • Researchers reached out to original authors to obtain data and code and inform them about the PBR project.
  • Researchers attempted to replicate the results using the original data and code, according to instructions given by the original author.
  • Researchers have written a report based on this project and a pre-print version is available on the Open Science Framework here.