3ie’s push-button replication (PBR) is a project to confirm the validity of published results using both the original data and the programming code from a study.
What is push button replication?
PBR research attempts to confirm the validity of published results using both the original data and the programming code from a study. The premise behind a PBR study is that the third party researcher should not need to make any significant adjustments, write new code or conduct additional analysis in order to arrive at the published results. Thus, a PBR is a step that precedes pure replication.
While a pure replication asks the question can we reproduce the published results using the original data and the methods described in the original study, a PBR tries to verify if we can use the original authors’ programming code with the original data to reproduce the published results. Through pure replication studies, researchers can potentially uncover errors where the programming code incorrectly implements the estimation methods described by the researchers of the original study. PBR studies, on the other hand, can help reveal instances where there are no or insufficient data and code available or where the programmes do not run.
- Establish procedures and standards for PBR so that original authors and replication researchers can better align their expectations and actions around this third-party verification process
- Test whether development impact evaluations are generally push-button verifiable
How are PBR studies conducted?
- Candidate studies are chosen from a set of articles selected from across all development impact evaluations published in the top ten journals for development impact evaluations in 2014. The sample size was established at 122 studies.
- Researchers reach out to original authors to obtain data and code as well as inform them about the PBR project.
- Researchers will attempt to replicate the results using the original data and code, according to instructions given by the original author.
- Researchers will write a report and classify the replication based on the outputs of the PBR. The PBR and the report will be available on Open Science Framework.
In May 2015, 3ie conducted a one-day consultation event in Washington DC, which was attended by supporters of replications, critics as well as people with a general interest in knowing more about replication research. One of the livelier discussions at the event centered on whether it is reasonable to expect that the vast majority of published empirical studies can be exactly reproduced. Researchers questioned whether it is fair to expect that original data and programming code or coding documentation from an article are available, which could then be used by a third party to easily reproduce the published results.
All present agreed that this kind of reproduction, where one can essentially push a button and recreate the published tables in an article is the most basic question in replication studies. Some argued that this expectation should be a given – that original authors always have the data and programming documentation to reproduce their work. Others expressed strong doubts about how frequently authors really can provide the required materials to reproduce the published findings. They argued that replication research should focus, either primarily or additionally, on this very first line of verification, at least until the relevant fields reach a point where we can take exact reproduction of published results for granted.