Do funding agencies distort impact evaluations? A session organised by BetterEvaluation on choosing and using evaluation methods, at the recent South Asian Conclave of Evaluators in Kathmandu, focused on this issue. Participants were quite candid about funding agencies dictating terms to researchers. “The terms of reference often define the log frame of evaluation (i.e. the approach to designing, executing and assessing projects) and grants are awarded on the basis of budgets that applicants submit. It’s a bidding process — all about executing the funding agency’s design at a minimal cost,” said a researcher.

But participants stated that 3ie’s approach to funding impact evaluations is refreshingly different. Instead of a priori structuring of impact evaluation design, 3ie focuses on active engagement with researchers and implementing agencies to develop an appropriate study design for the evaluation. This approach was demonstrated by an on-going 3ie-supported study to assess the impact of media campaigns on the incidence of early child marriage.

Child marriage continues to be practised in India, even though it is a punishable crime. The implementing agency, Breakthrough, a global human rights organization uses interventions like mass media, community mobilization and training of community members, developed during an earlier campaign on domestic violence. Breakthrough approached the government to partner them on the child marriage campaign, but they were asked to demonstrate the effectiveness of their strategies. The organisationthen approached 3ie to assist in the evaluation of the programme. Through an open bidding process, Catalyst Management Services from Bangalore were selected to design and undertake an impact evaluation, funded through 3ie’s Policy Window.

Extensive discussions were held between 3ie, the research team and the implementing agency, to arrive at a study design that would encompass both qualitative and quantitative aspects. “3ie worked with us not merely as funders but as partners on the study design,” said Urvashi Wattal, project coordinator, Catalyst Management Services. The collaboration led to changes in the project design. The study, initially planned in three districts, was expanded to include three more districts. A mid-line survey was also added.

Collecting data on a sensitive issue like child marriage is challenging. The research team will use the polling booth method, which they have used in other studies. The community members will be asked to enter an enclosed area resembling a polling booth and vote on certain issues related to child marriage. The voting will be devised through colours and other means, thus accommodating the low level of literacy.

As a staff member of 3ie I was pleased to attend a session – which 3ie was not at all involved in organising in any way – which highlighted as best practice the way we work in order to respond to the policy needs of Southern governments and NGOs. More and more implementing agencies are approaching 3ie to undertake similar studies supported by our Policy Window.