Replication research is often the space of junior researchers. As a well-cited research economist, with multiple international organisation affiliations, Paul Winters stands out in this space. I recently caught up with Paul to discuss his co-authored replication study of Galiani’s and Schargrodsky’s influential paper Property rights for the poor: Effects of land titling.
Galiani’s and Schargrodsky’s Property rights for the poor: Effects of land titling is considered an important paper in the development literature. Land titling programmes exist throughout the developing world. Whether it is through asset collateralisation or household investment, land titling theoretically combats poverty through numerous channels. Galiani’s and Schargrodsky’s heavily cited study shows that an Argentine land titling programme induced a range of positive effects, from expanding household investment to improving children’s education. Paul’s replication study set out to re-examine intermediate outcomes of this programme to better understand the robustness of the theory of change behind land titling interventions in general.
I was also interested in his perspective on 3ie’s replication programme. We recently held a replication consultation with researchers and multilateral organisation representatives, where some participants suggested we only focus our programme on attempting to exactly reproduce the published results (akin to 3ie’s pure replication section), and not include any robustness checks or explorations of the intervention’s theory of change. Paul’s replication study, like all 3ie-funded replication research, included both a pure replication section and robustness checks of the original research (although data limitations prevent it from fully exploring the intervention’s theory of change).
In our conversation, Paul reflected on his experience in replicating an influential paper and gave me his take on what he thinks about the replication process in general.
Ben: Congratulations on finishing up your replication paper. How would you summarise your replication study of Galiani and Schargrodsky’s Property rights for the poor: Effects of land titling impact evaluation?
Paul: Galiani and Schargrodsky (GS) is a widely cited paper, since they find that providing property rights has an impact on social outcomes. They are able to identify this impact using a natural experiment. In our paper, not only were we able to replicate the results found by GS, but robustness checks using alternative estimates of impact largely confirmed their results, suggesting the outcomes and conclusions presented by GS are reasonable.
Ben: Some of the participants at our replication consultation suggested that 3ie focus its replication programme only on studies that attempt to exactly reproduce the published analysis (similar to Chang and Li’s replication study in economics paper). You have argued against that and said that replication studies should continue to examine robustness checks beyond pure (or exact) replication. Would you please elaborate a bit on your thinking?
Paul: Journals like the American Economic Review (AER) only require researchers to provide the data and analysis files to exactly reproduce original studies and that is the minimum standard. But if you are delving into a data set and considering how variables are created and the analysis being conducted, it is critical to know the literature and to consider the theory of change. Additionally, Brian Nosek’s work that was presented at the replication consultation clearly highlights how decisions on variable creation and model specification can influence results.
Further, if you are interested in designing successful policy interventions linked to an evaluation, you need to understand the mechanisms by which impact occurs and ideally the type of beneficiary most likely to benefit from a programme (by exploring the heterogeneity of impact). A replication is an opportunity for carefully considering an intervention and moving beyond what is done in a paper.
Of course, if the objective is simply to ensure the integrity of the result the AER standard is fine. But 3ie seeks to use evidence to inform policy and programming. The replications you fund should be more critical and look beyond what is done in a paper by reconsidering the theory of change and checking the robustness of results.
Ben: Finally, a few researchers have expressed concerns about the usefulness of internal replication research. As a researcher who approached replication research without many expectations, what have you learned from your experience?
Paul: I believe it is a worthwhile undertaking. At a practical level, it would have been helpful to know what type of data were available for the replication right at the outset. We had hoped to use the data to explore the causal mechanisms of impact and heterogeneity of impact and this was in our replication plan. But those data were not available. It should be clear that GS did provide the data and what they provided is consistent with the AER standard. Further, if someone asked me for data on a previous project, especially a number of years later, as was the case for GS, I probably could not have provided much more. Of course, this points to a potential benefit of such replications. It could make us more careful about documenting all of the steps we take in our impact evaluations.
To download Paul’s co-authored replication study in 3ie’s Replication Paper Series, click here.
For highlights of 3ie’s replication consultation event, click here.
For more information on 3ie’s replication programme, click here and for the summary results of a recent systematic review on land titling click here.