Mixed Methods | Impact Evaluation

Next 1 2 3 4 5 Previous

Posted 2010-08-24

Experimental approaches: What works?

In a recent issue of the Journal of Economic Perspectives, a number of leading applied researchers made original contributions to the debate on impact evaluation methods:

Design-driven approaches are often criticized for producing evidence that has little predictive value beyond the context of the original experiment. But Angrist and Pischke (2010) argue that generalisability can be increased by looking for extensive evidence on a particular issue from a range of contexts and that so called trivial questions address issues with far-reaching implications for policy, especially if the research methods are rigorous.

However. Leamer (2010) and Keane (2010) warn that given the role of assumptions in statistical work, one should be circumspect about the degree of evidence one is able to generate using controlled experiments. Leamer (2010) makes a case for the method of sensitivity analysis, which helps determine the actual impact a particular variable has if it differs from what was previously assumed. Keane (2010) on the other hand advocates for a theory based approach, one that takes into account all possible assumptions made during research. Sims (2010) also argues that for research to inform policy, econometricians must also be trained to face the complexities that arise in non-experimental inference.

Nevo and Whinston (2010) suggest that a serious empirical researcher should choose a method according to the specifics of the question being studied and the available data, and advocate that the method of structural modeling offers important advantages over the “design-driven” methods in dealing with issues of generalization and external validity. Stock (2010) proposes that other methods such as robust standard errors, methods for inference with weak instruments, nonparametric and semi-parametric regression must also be given due importance as theyminimize the impact of assumptions subsidiary to the main empirical question, but which nevertheless affect the variables of interest.

Posted 2010-06-29

Evaluating the impact of rural electrification

Electricity brings light to previously dark areas, a benefit greatly appreciated by low-income families around the world (IEG, 2008). But are there benefits other than better lighting and TV for the better off? A set of new rigorous impact studies suggests that there are.

Income increases have been found in South Africa from increased female employment, especially among communities that had been initially relying heavily on wood for cooking (Dinkelman, 2008). Broader changes were also found in both Bangladesh and Mozambique. In rural Bangladesh electrification had positive impacts on households’ income and educational outcomes, though rich households benefitted more than poor households (Khandker et al., 2009). And the study in Mozambique supported the argument of the IEG (2008) report that rural electrification can be commercially viable and cause structural transformation in a short period of time through its impact of electricity on households, education, agro-business, commerce, and the public sector in Mozambique (Mulder and Tembe, 2008).

Posted 2010-05-31

Evaluating the impact of school feeding programmes

A recent evaluation in rural Kenya shows that school feeding led to improved attendance rates among elementary school children (Omwami et al., 2010).

Similarly, school meals have led to significant gains in nutrition and learning in Andhra Pradesh, India; and have acted as a security net for young children by compensating for the lack of food in the context of drought (Singh, 2008).

However, an evaluation of two school feeding schemes in rural Burkina Faso showed that while girls’ enrollment increased by 5 to 6 percentage points, there was no significant impact on academic performance and attendance in the context of agriculture households without an active labour market (Kazianga et al., 2009).

Posted 2010-05-11

Evaluating the impact of Conditional Cash Transfers (CCTs) on education
Overall, evidence shows that Conditional Cash Transfers (CCTs) increase access to education but do not necessarily lead to improved learning outcomes.

In Nicaragua, the conditional cash transfer programme Red de Protección Social reduced dropout and repetition rates over a four year period with a positive effect on grade progression in areas with better equipped schools. The programme also effectively improved education supply in areas with poor initial supply (Maluccio et al. 2010).

A recent evaluation of Oportunidades in Mexico found that increasing the grant for secondary school children while eliminating it for primary school children had a substantially larger effect on the enrollment of the former (Attanasio et al., 2009).

However, the evaluation of the Colombian Familias en Acción programme revealed a negative effect on the school achievement of adolescents, possibly due to the poor quality of education caused by crowded classrooms or peer effects (Garcia and Hill, 2010).

Posted 2010-04-06

Evaluating the impact of Conditional Cash Transfers (CCTs) on maternal and neonatal health

A recent impact evaluation of Mexico’s flagship CCT programme Oportunidades shows an improvement in the quality of prenatal care for low-income rural women, with significant positive effects on birthweight, partly as the programme encourages women to become informed and active consumers of health care (Barber and Gertler, 2009).

However, in Nepal even though the Safe Delivery Incentive Programme has raised skilled birth attendance and reduced the chance of a woman delivering at home, the programme was found to have no effect on infant mortality and only 40 per cent of the eligible women benefitted from the programme - with distribution heavily skewed in favour of relatively better-off families. (Jackson et al, 2009).

Financial incentives to healthcare providers under the ‘Paying for Performance’ programme in Rwanda can improve both the use and the quality of health services (Basinga et al, 2010). The results indicate that an equal amount of financial resources without the incentives would not have achieved the same gain in outcomes.

Posted 2010-02-17

Using mixed methods for impact evaluation

Mixed methods are the explicit adoption of both quantitative and qualitative methods in the impact evaluation design (White, 2008).

The debate around ‘mixed methods’, also known as ‘Q-squared in Policy’ has stressed the complementarities of the two approaches. That is the depth from qualitative research and the statistical robustness from quantitative research often inform, correct and augment each other (Shaffer et al, 2009). Important aspects of this complementarity are that mixed methods for impact evaluation ensure that the research process is used as a way of increasing accountability and empowerment of the poor (Garbarino and Holland, 2009), and that participatory research offers qualitative insights and generates statistics for relevant dimensions that might otherwise be overlooked (Chambers, 2007).

Posted 2010-01-10

Updated wikipedia on impact evaluation. To view.

New glossary on impact evaluation now available. 3ie has developed a glossary of key terms in impact evaluation.

Posted 2009-12-18

Evaluating the US Coalition for Evidence-Based Policy

Recent congressional initiatives seek to focus federal funds on interventions that have been found effective in randomized experiments. A new report from the US Government Accountability Office examine the process used by the nonprofit Coalition for Evidence-Based Policy to identify social interventions meeting such standard, and concludes that requiring evidence from randomized studies as sole proof of effectiveness will likely exclude many potentially effective practices.

Posted 2009-12-07

Evaluating HIV/AIDS prevention programs

A meta-analysis showed HIV voluntary counselling and testing (VCT) recipients were significantly less likely to engage in unprotected sex when compared to behaviours before receiving VCT, or as compared to participants who had not received VCT (Denison JA et al, 2008).

However in Malawi, a program that offered door-to-door HIV testing was found to have no impact on preventive behaviour, as measured by condom purchases. Door-to-door voluntary counselling and testing (VCT) campaigns are expensive and not usually targeted at key populations at higher risk, thus lowering the effectiveness of such program. These costs must be weighed against small apparent benefits. Only some participants showed more preventative behaviour after learning their status, and even among them the effect was very small (Thornton R.L., 2008).

An earlier study showed that after two years, girls in schools in Kenya where teachers had been trained in the standard, national HIV/AIDS curriculum were no more likely to prevent pregnancies than girls in comparison schools. Reducing the cost of education by paying for school uniforms reduced dropout rates, teen marriage, and childbearing, and informing girls that men in their 20s, or older, are more likely to be HIV positive than young men led to a dramatic 65 percent drop in childrearing with older men, with no increase in pregnancies with younger men (Duflo et al, 2006, Dupas, 2009).

Posted 2009-11-17

Evaluating the impact of agriculture programs

Over 1 billion people go hungry in the world today. Learning from successes in agricultural development is now more urgent than ever. As heads of state gather in Rome for the World Summit on Food Security, a new book pulls together – for the first time – major successes in agricultural development that have brought millions of people out of hunger over the past 50 years.

"Millions Fed: Proven Successes in Agricultural Development", produced by the International Food Policy Research Institute (IFPRI) and commissioned by the Bill & Melinda Gates Foundation, determines what works in agriculture – what sorts of programs, policies, and investments have had a proven impact on hunger and food security.

It presents 20 case studies of large-scale initiatives that are documented to have substantial, long-term effects. These stories include achievements not just in increasing the yields and production of staple food crops, but also in expanding markets, developing better policies, conserving natural resources, and improving nutrition.