August 18: Community Based Micro Health Insurance: A recipe for replication?

 Health risks are a serious threat to poor households as health shocks can significantly increase out- of-pocket expenditure. Community- based Health Insurance schemes in such cases are seen as a simple and affordable solution for poor communities to manage health risks. These are small scale, voluntary health insurance programs, organized and managed in a participatory manner and draw on resources of social solidarity and cohesion (Tabor, 2005).

The discussion at the seminar raised questions concerning the possible negative spillover effects on the control group and whether this impact assessment could produce findings that can contribute to replication in other contexts.

To download MIA's presentation, click here.

July 14: An assessment of the impact of water supply and sanitation interventions

Diarrhoeal diseases account for about 22% of child deaths around the world. It is the leading cause of death of children under the age of 5. Asia and Africa are the most affected regions with diarrhoea accounting for about 14 per cent of mortalities under the age of 5 in India.

Dr. Thomas Clasen, Lecturer at the London School of Hygiene and Tropical Medicine cited research at the Delhi seminar to show that diarrhoea was a result of not just contaminated water but also open defecation and poor sanitation. While treating water may improve water quality there wasn’t enough evidence about the accrued health gains. In such a situation “treating water at source may not be enough as water can get re-contaminated when stored at home” he said.

 Dr. Clasen presented a case for Household Water Treatments (HWTS) which included using filters, chlorine tablets, sodium hydrochloride, and boiling water. However, low take-up and compliance over the long term remains a significant challenge in the adoption of HWTS. According to Dr. Clasen, a potential solution would be to target poor households with children and make interventions affordable or give them away free. “That’s where the health gains could be achieved”, he said.

June 16: The miracle of microfinance ? Evidence from the Spandana Evaluation

Over 150 million people access microfinance services globally. But the jury is still out about whether microfinance improves the lives of the poor.

"While the bulk of the development community looks at microfinance as a panacea to the problem of poverty, this study brings more realistic expectations into the discourse,” said Justin Oliver, Executive Director of the Centre for Micro Finance at IFMR Chennai, during his presentation of the findings of the first randomized evaluation of Spandana microcredit intervention in the slums of Hyderabad, India. 

Spandana is the fifth largest Microfinance institution in the world. While no effect of access to microcredit on average monthly expenditure per capita was found 15 to 18 months after lending began, expenditure on durable goods increased in treatment areas. The number of businesses also increased by one third. However, there was no impact on health, education, or women’s decision making outcomes.

“The presence of microfinance in this case increased seriousness in terms of how people spent their money,” said Oliver.

During the discussion, Dr. Vegard Iversen from the Institute of Development Policy and Management in Manchester stressed that the absence of discernible microcredit impacts on average per capita consumption expenditure and basic well-being indicators like - child illness shocks and aggregate health expenditure within the household, education enrollment or expenses or female decision-making power within the household - should not come as a great surprise.
 
“Both the choice of location like Hyderabad and the very careful ‘screening’ of eligible microfinance customers and candidate settlement areas carried out by Spandana would result in target groups with comparatively high initial figures. For education, this is illustrated by the enrollment of 98 % among 7-11 year olds,” said Iversen.
 
“Compared to a rural setting with thinner credit markets, lower human development benchmarks and less mobility among potential customers, the present study would plausibly underestimate poverty and human development impacts while the impact on enterprise establishment would be expected to be lower,” he added.

To download Justin Oliver's presentation, click here
To read the Spandana study, click here


May 5: What do we know about the impact of Microinsurance? 

Over the last decade, the need for microinsurance as a risk management tool for the poor has been increasingly recognised. So, the sector has been growing, as has the number of evaluations. However, these studies mainly rely on descriptive statistics, thus overestimating the impact of the schemes. So far, little is known about the true impact of microinsurance.

Ralf Radermacher, Director of the Micro Insurance Academy in Delhi, presented the findings of a recent systematic review including 35 impact studies mainly in rural Africa and Asia and reporting the impact of 122 different microinsurance schemes, most of which focused on health insurance.

“Outcome measures need to be streamlined so we have comparable reporting and we understand why things work. Most studies look at ‘whether’ there is impact and not ‘why’ there is impact,” said Mr. Radermacher.

The review shows that microinsurance can increase access to healthcare and financial protection. However, the methodology used is often weak with potential biases.The stock taking analysis conducted by the Impact Working Group of Microinsurance Network shows that there has been a recent shift towards Randomized Controlled Trial evaluation designs. There has also been more diversity in the focus, looking for instance at agriculture and other sectors besides health insurance.

To download Mr. Radermacher's presentation, click here


April 7: Impact Evaluation in the Post-Disaster Setting: A luxury or an essential?

The humanitarian and development communities invest billions of dollars each year in post-disaster relief and recovery efforts. There is, therefore, an urgent need to build evidence on what interventions have worked so that the same mistakes are not made after when the next disaster strikes.

“While some might say that impact evaluation is a luxury in a post-disaster situation, not doing an impact evaluation is actually the luxury you cannot afford. In a post-disaster situation, we need to see what works and what doesn’t and how cost-effective it is,” said Dr Howard White, 3ie Executive Director. 

Conducting an impact evaluation presents some serious and unique challenges – such as the lack of baseline data or the ethical issues of excluding individuals from relief and recovery by creating a comparison group. However, Dr White pointed out that there is plenty of scope and time to conduct quality impact evaluations during the reconstruction phase.  There are ways to also reconstruct a baseline by using pre-disaster household surveys, needs assessment data, recall and Census-based poverty estimates. Other challenges raised include the limited time to plan and implement interventions and the triple selection bias from selection in exposure to the shock in the first place, selection from out migration and selection in program placement.

To download Dr White’s presentation, click here.


March 5: Evaluating A Large Health Education Program: The Sure Start Project in Uttar Pradesh

Impact evaluations of health programmes are key in terms of their policy relevance and impact. There, however, needs to be emphasis on mapping out the causal chain between inputs and outputs and its eventual impact. These were some of the questions raised at the Delhi Seminar series on "Evaluating Large Health Education Program: The Sure Start Project in Uttar Pradesh".

Presenting at the seminar, Dr. Arnab Acharya, Sr. Lecturer, London School of Hygiene and Tropical Medicine, elaborated on the challenges in designing, planning and conducting the evaluation of the ‘Sure Start Project’ - a large health education program in Uttar Pradesh. “The programme design is simple but the analysis is not. If we succeed in Uttar Pradesh, we will succeed elsewhere,” he said.

The Sure Start Project is designed to enhance pregnancy and neo-natal health in 7 districts of the state of Uttar Pradesh. It is aimed at supplementing the efforts for the National Rural Health Mission (NRHM) and if successful, will be scaled up nationally. A baseline survey was carried out on the pregnancy behaviour of 17,000 women for a pregnancy that occurred in 2007 and was completed at least a month before the survey. To conduct such an evaluation, the main challenges include: the difficulties in using the difference in difference approach in the case of rare occurrences such as death, and the fact that the project will not have the same subject being pregnant at the end of project, 2010-11.

Vikram Rajan, Health Systems Specialist, World Bank pointed out that this was a key project particularly since there is a paucity of rigorous impact evaluations in this sector. “The programme is important in terms of policy relevance and policy impact, particularly since Uttar Pradesh is the biggest contributor in India to Neonatal Mortality Rate”, he said.

Ralf Radermacher, Director, Micro Insurance Academy raised the issue of spillover effects and contamination resulting from the geographical distance between beneficiaries and non-beneficiaries. 

Among other points discussed at the seminar, were questions concerning the structure of incentives provided to ground level staff, like Accredited Social Health Activists (ASHA), who are involved in implementing health measures.

To download his presentation, click here.


December 15: When and how to apply non-randomized methods of impact evaluation

While randomization is clearly a useful tool for rigorous impact evaluation designs, it is sometimes not feasible from an ethical, political or logistical point of view. Therefore, non-randomized methods - such as reflexive evaluation, propensity score matching, difference-in-differences, instrumental variables, and regression discontinuity design -are often used to select appropriate counterfactuals in such situations. 

Dr. Ahmed from IFPRI presented at the Delhi seminar series the challenges of constructing counter factual using five non-randomized methods of impact assessment: reflexive evaluation, propensity score matching, difference-in-differences, instrumental variables, and regression discontinuity design. 

To download his presentation, click here.


November 11: Eight key steps to conduct a Synthetic Review on water and sanitation 

How do we assess the range of evidence on development programmes to provide guidance to policy makers and planners? Synthetic review enables us to assess the quality of evidence and summarise it to provide clear guidance to policy makers and practitioners. 

Hugh Waddington from 3ie presented at the Delhi seminar series the eight key steps used to conduct his review on ‘water sanitation and hygiene interventions to combat childhood diarrhoea in developing countries’.

1.Scoping to identify relevant topic. This involves looking at:

The kind of interventions: Water supply, water treatment, sanitation, and hygiene programs
Populations: Children in low- and middle-income countries
Outcomes: Diarrhoea disease morbidity, intervention adoption or compliance
Theory: Disease transmission model and behaviour change

2.Background: determine the policy relevance and knowledge gaps

In a year, 6 times more children are killed by diarrhoea than by conflict around the world. So, what is the most effective kind of intervention to reduce the prevalence of diarrhoea among children in developing countries? 

3.  Theory: unpack the causal chain and factors influencing behavioural change

Theory: Disease transmission pathways



4.Rigorous search to identify published and unpublished sources and application of strict inclusion criteria, set out in Study Protocol

5.Systematic data collection and coding of information relating to:

a. Intervention and comparison group
b. Study quality (assessment of internal validity)
c. Contextual factors (external validity)Outcomes (impact ‘effect size’)
d. Causal chain (behavioural change)

6. Quantitative synthesis using meta-analysis, including assessment of impact heterogeneity

7. Synthesis of quantitative and qualitative information relating to causal chain

8. Review updated as new evidence emerges: every 3 years

To download his presentation, click here.


October 21: How to institutionalize evaluation: India and Mexico learning from each other

“CONEVAL’s original sin is that we are based within the Ministry of Social Development and we will need to move out from the Ministry in order to become independent,” stressed the head of the National Council for Evaluation of Social Development Policies in Mexico, Gonzalo Hernandez-Licona.

This was an important lesson from the case of Mexico at the time where India is setting up a new Independent Evaluation Office, which was raised by Hernandez-Licona at the international conference on development evaluation hosted by the Planning Commission and reiterated at the 3ie Delhi seminar.

Mexico was the first country to introduce mandatory impact evaluation for all its social programs. This was in part a result of the lessons learnt from the first evaluation of the Government flagship program Progresa/ Oportunidades, which provides cash transfers conditional upon regular school attendance, health clinic visits and nutritional support to children. By rigorously demonstrating the program’s success in reducing children’s malnutrition rates and child labor, as well as increasing boys and girls’ enrolment in secondary school through independent evaluations, the program survived the change of government and was scaled up.

More recently, Mexican President Felipe Calderon was influenced by the results of an evaluation “Housing, Health and Happiness" showing that replacing dirt floors with cement significantly improves the health of young children in Mexico, and decided to further invest in this national program.
 
Read more and download: Gonzalo Hernandez-Licona’s presentation on “The Importance of Impact Evaluation in Mexico’s Monitoring & Evaluation System” 


September 30: Systematic Reviews 01 - How to promote evidence based policy making

How can policy makers use evidence from academic studies? Evidence-based medicine has led the way in systematically reviewing available evidence to give rigorous information on what works and what doesn’t. 

As part of 3ie’s Delhi seminar series, Dr. Ruhi Saith, Senior Researcher from the Institute of Public Health Group and consulting editor for the Cochrane Public Health Group, presented today a practical introduction onsystematic reviews. Her presentation explains their advantages compared to ordinary literature reviews, and how they can benefit policy makers and development practitioners and translate into policy changes. 

Systematic reviews start with a comprehensive review of all available studies, grades them by quality criteria, only including in the review those which meet certain standards, and, where applicable, provide a consolidated estimate of the effectiveness of the intervention by pooling the results of all studies. Such systematic reviews, supported by the Cochrane Collaboration, have become the cornerstone of evidence-based medicine.

Systematic reviews started in the area of medicine started in the area of medicine, but are playing an increasing role in social policy. In the United States the What Works Clearing House summarizes evidence on education policy and programs. Internationally the Campbell Collaboration promotes quality standards for systematic reviews in education, crime and justice and social welfare.

For instance, a recent review of 44 research studies on CCTV schemes by the Campbell Collaboration found that they do have a modest impact on crime overall but are at their most effective in cutting vehicle crime in car parks, especially when used alongside improved lighting and the introduction of security guards. The findings were extensively reported in the international media and was cited by British Home Office ministers in response to concerns in the House of Lords about the surveillance society. 

3ie is partnering with the Campbell Collaboration in the production of synthetic reviews of development policies and programs. All 3ie-supported reviews are carried out in accordance with the Campbell guidelines and procedures. The third call for proposals will be made in early 2010. Read more on how to apply and 3ie criteria. 

Campbell Collaboration promotes quality standards for systematic reviews in education, crime and justice and social welfare. 

View video clip on “How to institutionalize evaluation: India and Mexico learning from each other”: http://www.youtube.com/watch?v=OXFTLD0U46E 


July 27: The impact of Water Supply and Sanitation interventions on child health - Evidence from DHS surveys conducted by Dr. Ron Bose

Evidence from DHS surveys in Nepal shows a 40 per cent reduction of diarrhoea cases attributable to sanitation coverage among children under 5 years. Dr. Bose presented the findings of a new 3ie case study on Nepal.  


June 23: Theory Based Impact Evaluation - How to unpack the causal chain

Recent years have seen an increased demand for rigorous impact evaluation of development interventions. This new quest is not just about finding what works, but understanding why.

A theory-based approach to impact evaluation maps out the causal chain from inputs to outcomes and impact. It tests the underlying assumptions as to why an intervention has worked or not. If theory based impact evaluation is widely accepted in principle, its application remains weak in practice.

3ie hosted a discussion on ‘Theory-Based Impact Evaluation’ as part of its Delhi seminar series where Howard White, 3ie Executive Director, presented some case studies and proposed steps to bridge the gap from theory to practice.

The event was chaired by Dr. S.P. Pal, President of the Department Evaluation Society of India and Dr. Milindo Chakrabarty, as main discussant and Executive Director of the Development Evaluation Society of India.

“Often the assumptions behind any programme design are not correlated to the reality on the ground”, stressed Dr. Chakrabarty. 

“In the case of the National Rural Employment Guarantee Scheme, we found out during the evaluation that people requested job cards to document their citizenship and not necessarily because they needed work”, he added.’

Participants also stressed the difficulty to map out the causal chain in a cost effective way, and design a counter-factual particularly for interventions in sectors other than health and education.

To download Dr. White’s presentation, click here.
Read more on ‘Theory Based Impact Evaluation’ in 3ie working paper 3.
Read more and post your comments, join the follow-up discussion on 3ie social network: http://3ieimpact.ning.com/forum/topics/launch-of-impact-evaluation


May 21: Reproductive health of married and unmarried youth in India: where are we now and what have we learned?
Many adolescents in India continue to suffer from poor reproductive health and limited access to appropriate health services. India has one of the highest rates of child marriage in the world, a practice often resulting in early child bearing and thus serious reproductive health problems. 

With the growing number of adolescents in the country, youth reproductive heath and sexual health has now become a priority issue for the Government. Policy-makers and development practitioners are under pressure to find out more about the effectiveness and costs of reproductive health interventions amongst young people.  What works and what doesn’t and at what cost are some of the pressing questions in urgent need of elucidation from the evaluation community.

3ie launched its Delhi seminar series on Impact Evaluation with the senior Youth Reproductive Health Specialist Sunayana Walia from the International Center for Research on Women (ICRW) who presented the findings of a 10 year research program conducted with different community based and nongovernmental organizations across India. 

The discussion focused on the challenges and opportunities of conducting rigorous impact evaluation on women’s reproductive health interventions, and the possibility of exploring partnerships to improve the state of evaluation in India. Researchers and practitioners agreed that the key questions that needed to be addressed were related to the important issue of attribution: when looking at the change outcome, can one identify the impact and contribution of the intervention itself? 

In his response, the discussant and World Bank health economist, Dr. Jerry la Forgia, stressed that policy makers are often interested in packaged interventions, which requires examining what of combination of interventions work best and in what context to affect desired outcomes.

The issue of boys and men involvement in the program was also raised as one of the main challenges. No reproductive health interventions and policies are currently targeting fathers. The specific role and involvement of fathers should be further investigated. 
The event was chaired by the Deputy Director of the UK Department for International Development (DFID) Nick York, and included the Secretary of the Development Evaluation Society of India Dr. Milindo Chakrabarty and senior representatives from other Delhi based research and development agencies.

Full report available at: http://www.rockfound.org/library/2006-improvingrepohealth-india.pdf

Read more and post your comments, join the follow-up discussion on 3ie social network: http://3ieimpact.ning.com/forum/topics/launch-of-impact-evaluation