My recent blog, How big is big enough?, argued that you need sector expertise to judge if the effect of a programme is meaningful rather than just statistically significant. But the need for sector expertise goes far deeper than that.
I have recently been reading impact evaluations of water supply and sanitation studies. The studies by the non-sector researchers (mostly economists) collect data on the outcome of interest, usually child diarrhoea. But they do little more than that.
The exclusive focus on outcomes contrasts sharply with, for example, a 3ie-funded impact evaluation by Tom Clasen and others of the Total Sanitation Campaign (TSC) in the state of Odisha in India. The TSC combines social mobilisation with government subsidy for toilet construction. The study collected data on several indicators of latrine use such as the smell of faeces, stain from faeces or urine, the presence of soap, the presence of a broom or brush for cleaning, and the presence of slippers. The researchers also tested for faecal indicator bacteria in water sources and in household drinking water, as well as on children’s and mothers’ hands and on children’s toys. They tested for hand contamination of household members using hand rinse samples. And they set fly traps to measure the density of flies. This is an excellent example of the approach advocated by 3ie of theory-based impact evaluation, of measuring indicators along the causal chain. And so when there was no improvement in child diarrhoea despite a substantial increase in the latrine coverage the researchers could identify the likely reason: not all family members were using the latrine.
Another example of the use of the causal-chain approach comes from an impact evaluation of a handwashing and hygiene education intervention in Pakistan. In addition to child diarrhoea, data were collected on water treatment and handling, typical hand cleansing materials, direct observation of hand washing, and reported occasions for hand washing. The careful analysis of variables along the causal chain is again notable: who was handwashing and were they doing it regularly and properly. By and large they were, explaining the observed reduction in diarrhoea.
3ie is a strong advocate for embedding impact evaluations in a broader analysis of the causal chain. Many 3ie staff have blogged about it (See here, here and here). But it is not something a lot of impact evaluation teams apparently get. They generally fail to see the need to include sector experts in the study design teams, or even to consult them at that stage. They also often engage insufficiently with the relevant sector literature.
So, impact evaluations such as those mentioned above in India and Pakistan deserve to be more widely read. In addition to the causal chain analysis, both studies have striking findings. In India, a very substantial increase in latrine coverage and use didn’t achieve a reduction in diarrhoea. And in Pakistan, the intervention continued to have sustained improvements in hygiene practices three years after completion.
I talk a lot about the difficulty of achieving behaviour change, and how it can take decades rather than months. So, the Pakistan study intrigues me. Why it worked is the subject of another blog.