The role and interpretation of pilot studies in impact evaluation research
Speaker: Shagun Sabarwal, associate director- training, CLEAR/J-PAL South Asia at IFMR
Chief discussant: Dr. Subrato Mondal, Monitoring, Evaluation and Learning Specialist, Health Office, USAID/India
Discussant: Purnima Menon, senior research fellow, Poverty, Health and Nutrition Division, IFPRI
Chair: Beryl Leach, deputy director, policy advocacy and communication, 3ie
Date: 3.30-5.00 p.m.,20 October 2016
Venue: PHD House, No. 4/2, Siri Institutional Area, August Kranti Marg, New Delhi, Delhi 110016
Download the brief for this seminar
Abstract
A pilot study is an important step when exploring a novel intervention or an innovative application of an intervention. Results from pilot studies can inform the feasibility and identify modifications needed when designing a larger and subsequent hypothesis-testing study. However, rigorous pilot studies (also referred to as formative studies) that are precursors to larger randomised evaluations of social programmes are not as frequently or as intensively conducted as they should be. The role of the pilot phase that precedes the randomised evaluation is often underestimated.
In this presentation, Shagun Sabarwal used the example of a recent pilot study done in collaboration with Government of Punjab as a precursor to a full-scale cluster randomised evaluation. The impact evaluation will test the effectiveness of a range of interventions aimed at increasing the motivation of and making the auxiliary nurse midwives (ANMs) at health sub-centres more attentive to high-quality service provision and patient satisfaction. The findings from the pilot study have informed the theory of change of the intervention and and has also informed the design of the randomised evaluation.In her talk,she discussed the challenge of communicating findings from pilot studies to implementation partners in in a way that minimises the risk of over-generalisation when making policies.
About the speaker
Shagun Sabarwal is associate director of training for CLEAR/J-PAL South Asia at IFMR, where she works on strengthening monitoring and evaluation capacity in the South Asia region. Shagun has a doctoral degree in public health from Harvard University. At J-PAL South Asia, Shagun supports the CLEAR Initiative, developing and delivering courses and technical advisory services in impact evaluation and measurement. She is also working on evaluating the impact of improving soft skills of frontline health workers in Punjab.