Mixed Methods | Impact Evaluation

Next 1 2 3 4 5 6 Previous

Posted 2012-01-31

Can conditional cash transfers stem the spread of HIV in Africa?

Conditional cash transfers (CCT) are being promoted for an increasing range of objectives. Can they help in the prevention of HIV/AIDS? Recent impact evaluations show that financial rewards have a positive impact on women and a negative impact on men in this respect. In rural Malawi, the cash transfer encouraged women to reduce their risky sexual behavior in the short-run (Kohler and Thornton 2011). The Zomba Cash Transfer Program in Malawi which provided school fees and cash transfers to girls also led to significant reduction in teenage pregnancy and self-reported sexual activity (Baird, Chirwa, and McIntosh 2010). This gives strength to the theory that education can be a ‘social vaccine’ in changing sexual behavior and the spread of HIV (Jukes et al., 2008). On the other hand, financial rewards may encourage men to ‘purchase’ risky sex, unless cash is supplemented with counselling. For the men in the rural Malawi study, the cash transfer led to increased risky sexual behavior in the short-run. (Kohler and Thornton 2011). However, when conditional cash transfers were combined with individual and group counseling, they reduced the incidence of curable sexually transmitted infections among both young men and women in Tanzania (Dow et al. 2010).

Posted 2011-12-21

Do School Feeding Programs improve School Participation?

School feeding programs are a popular form of conditional transfers where children, in exchange for regular school attendance, are fed a meal during school hours or given food rations to take home. While evaluations have found that such programs significantly improve school attendance (Kristjansson et.al 2006), the relative impact of a cooked meal in the school vs a take-home ration has been subject to debate. Evidence at times favors providing cooked meals in school (Afridi 2011) while at other times take-home rations are found to have more impact (Nielsen et.al 2010). Still other evidence suggests that in-school meals and take-home rations have a similar positive impact on school attendance and thus are equally valuable (Alderman, Gilligan and Lehrer 2010). There is clearly an opportunity and a need for a systematic review to take Kristjansson a step further, asking which scheme is preferable under which circumstances, to help policymakers and programme managers make sense of what appears to be conflicting evidence.

Posted 2011-12-09

Comparison of Model-based and Design-based Impact Evaluations

A major argument used by proponents of Randomised Control Trials (RCTs) is that they produce unbiased estimates of impact. A new paper (Hansen, Klejntrup and Anderson 2011) examines four cases showing that estimates using observational data are as unbiased as RCT’s, i.e. they produce the same result as an RCT The authors’ infact argue that in a developing country context model-based approaches may be preferable to RCTs.

Posted 2011-11-22

Impact Evaluation of Trade Interventions: Challenges and Opportunities

In recent times the feasibility of conducting impact evaluations in the area of trade policy has increased, partly with the shift in focus from traditional trade assistance to targeted interventions, known as ‘aid-for-trade’. Since such policies are targeted they can be subject to counterfactual analysis (Cadot, Fernandez, Gourdon and Mattoo 2011). However evidence generated from such evaluations may be compromised on account of poor availability of relevant and unbiased data, inadequate sample size, and existence of confounding factors such as other inter-connected public policies, selection bias, and information spillovers (Martincus 2011, Gourdon et.al 2011). Moreover evaluations often forget to address the context within which a program operates, inspite of it being an important determinant (Khandelwal and Atkin 2011). One possible way out then is to carefully build an impact evaluation design into trade programs right from the start (Cadot, Fernandez, Gourdon and Mattoo 2011).

Posted 2011-10-25

Improving Management and Business Skills for Small-Medium Enterprises

Evaluations of interventions seeking to improve business skills amongst small-medium sized firms provide contrasting evidence about the extent to which they influence performance. While an experiment conducted in India (Mahajan and Bloom 2011) found that training Indian firms in modern management practices increased their productivity by 11% and led to profits of around $228,000. However, in another experiment in Peru (Karlan and Valdivia 2010) entrepreneur trainings did not have any impact on business profits or growth. With such contrasting results it is useful to dig deeper to see what causes programs to succeed or fail. For example, a study in Tanzania found that participation levels during training, extent of formal education and cognitive skills affected outcomes: an extra five business training sessions were associated with a 10% increase in business performance, and the training had no impact on entrepreneurs with higher levels of schooling but significant effects were found for those with higher cognitive skills (Bjorvatn and Tungodden 2010).

Posted 2011-08-16

Evaluating the impact of vocational training for the youth

Training programmes for the youth have been introduced in recent years, particularly in Latin American countries. But the evidence to date does not suggest these programmes are universally beneficial. An evaluation of the Programa Joven in Argentina found a statistically significant impact on employment only for adult females (Aedo and Núñez, 2004). Similarly, the Colombian Jóvenes en Acción increased wage and salaried earnings and the probability of having paid employment for women but not men (Attanasio et al., 2011). In the case of theJuventud y Empleo training programme in the Dominican Republic, the overall positive impact comes from a large positive effect for better-educated workers, with only an insignificant effect for the less educated (Card et al., 2011).

Posted 2011-08-09

Can we build peace through community-based reintegration?
Community-driven development in post-conflict regions is widely viewed as a mechanism for empowering the poor, supporting local goods provision, strengthening governance and building social cohesion. But impact evaluations show a far from uniform picture of the impact of these programmes on both economic and social wellbeing.

A community-driven development programme offering block grants of over $20 million to conflict-affected villages in Aceh, Indonesia led to welfare gains and improvements in the perceptions of well-being in the treatment villages (Barron et.al., 2009). And in Sierra Leone, the GoBiFo programme (“Move Forward”) led to positive short-run effects on local public goods provision and market activity (Casey et al., 2011). But there was little evidence to show that a community-driven development programme in neighbouring Liberia had a positive impact on material well being (Fearon et.al., 2008).

Evidence is similarly mixed for social impact. The programme in Aceh did not lead to a significant improvement in social cohesion and relations between citizens and government. In some cases, the programme even led to lower levels of acceptance of ex-combatants by conflict victims (Barron et.al., 2009). Similarly the GoBiFo programme had no sustained impacts on fund-raising, decision-making processes, or the involvement of marginalized groups in local affairs (Casey et al., 2011). But in Liberia, there is strong evidence to show that community-driven development was successful in increasing social cohesion, reinforcing democratic political attitudes and increasing confidence in local decision making procedures (Fearon et.al., 2008).

Posted 2011-06-01

Enhancing the relevance and quality of randomised controlled trials (RCTs)

Randomised Controlled Trials test the efficacy of an intervention by randomly assigning the intervention among members of the eligible population. This approach is similar to the way drugs are tested by pharmaceutical companies. Whether RCTs can be used in development research has generated a great deal of debate. For example, Barrett and Carter (2010) illustrate how RCTs can suffer important pitfalls like ethical dilemmas and uncontrollable treatments resulting in biased results.

Many of these criticisms are addressed in a recent 3ie working paper (White, 2011). Since randomization is most often done across the pipeline, a section of the eligible population is usually untreated and the evaluation is just exploiting this fact to assess impact. To address the challenge of limited external validity, it should be ensured that the experimental pilot is as close as possible to how the programme will be in the scaled up version. Moreover, the paper echoes an increasing recognition that the randomized trial has to be set in a broader framework which unpacks the causal chain; see, for example, Gertler et al., 2010 and White, 2009.

More explicitly, Karlan (2009) argues that RCTs are not the opposite of qualitative methodologies because a good RCT evaluation often involves a thorough assessment of how the programme functions, its initial design, theory of change, beneficiary participation and so on. And, in a similar vein, Vaessen (2010) argues for developing and testing ‘rigorous’ mixed method approaches within a framework of theory-driven evaluation.

Posted 2011-03-25

Evaluating the impact of water, sanitation and hygiene interventions

Comprehensive water, sanitation, and hygiene interventions for poor communities are aimed at reducing the incidence of disease and deaths from water-borne diseases. But evaluations show that providing clean water at the community level tends not to deliver significant health benefits. Point of use water treatment and improved sanitation do deliver such benefits, but there are problems in both take up and sustaining adoption from those initially adopting the improvement (Waddington et al., 2009).

For example, an intervention for creating sanitation infrastructure in Tamil Nadu, India resulted in increasing the feeling of safety for women using private toilets. However, the intervention did not result in reducing the prevalence of diarrhoea or improving hygiene practices and behaviour. Nearly 40 per cent of the households with a private toilet continued with the practice of open defecation, which suggests that such interventions need to also tackle cultural, logistical or other barriers (Khush et al., 2009).

Community-based approaches have been proposed to overcome these barriers. A community-based intervention involving the formation of Water Sanitation and Hygiene committees in rural Bangladesh helped in motivating people to use safe water, and improving sanitation and hygienic practices. The intervention, which was provided through cluster meetings, preventive home visits, popular theater and celebrating sanitation month, resulted in reducing the overall prevalence of water-borne diseases, especially among children under-five (Rana, 2009).

3ie is funding new studies on water supply and sanitation to deepen our understanding of these issues. Read more

Posted 2011-02-27

Evaluating the impact of farmer field schools

Farmer field schools are intensive, season-long programmes where farmers meet regularly to learn new agricultural techniques. But how effective are these training programmes in contributing to agricultural development and farmers’ welfare?

In Kenya, Tanzania and Uganda, younger farmers and women were more likely to participate in farmer field schools. While in Kenya there was a significant increase in crop production, in Tanzania there was a large increase in agriculture income (Davis et al., 2010).

Cotton farmers in China, India and Pakistan used less pesticide and chose those with lower toxicity after a training programme in integrated pest management. In China, participants performed better in terms of both cotton yield and financial gain but there was no economic impact in India. In all three countries, there was no diffusion effect from trained farmers to their neighbours (Pananurak,2010).

Rice farmers in Thailand also significantly reduced their use of pesticides after training but there were no significant economic benefits to farmers. A possible reason is that yield gains are usually small in technologically advanced rice production systems and increased productivity effects through better timing of chemical pesticides are small unless there are pest outbreaks (Praneetvatakul and Waibel, 2006).