I recently attended the launch of the Independent Evaluation Office (IEO) in India – an event that marks the beginning of an era of evidence-based policymaking in India.
Are we being overly optimistic and maybe somewhat naïve in thinking this? Maybe.
The launch event, which happened in Delhi, included an eclectic mix of presenters and panelists consisting of key policymakers (including the chairperson of India’s planning commission), bureaucrats, India-based researchers and representatives from the Indian media. The discussions at the event brought to the fore several challenges that the IEO will face as it moves forward:
IEO’s identity: The IEO’s first challenge is to clarify its main role and objectives. The discussion at the launch highlighted divergent expectations about the IEO’s role. While the media expects the IEO to become a channel of accountability for the government, the bureaucrats in the audience emphasised the IEO’s role in increasing the coordination between different ministries and departments for better delivery of social programmes.
However, researchers and monitoring and evaluation experts debated the relative importance of monitoring and process evaluations compared to impact and outcome evaluations .
There are clearly many expectations of the IEO right now. So, what is the IEO’s role and what is its mandate?
A good way to answer this identity question is to have a theory of change. A theory of change for the IEO should explicitly mention the key outcomes it seeks to achieve, its thinking behind those outcomes and how it expects to get to them. A clear theory of change will help define the IEO’s identity and clarify its scope and objectives.
The Implementation challenges: What is the use of an evaluation if the the programme is not implemented properly? The director of the IEO, Dr. Ajay Chibber, refers to the problem of poor implementation as India’s Achilles heel. Should the IEO make implementation an important focus of its work? Yes, the IEO will explicitly have to tackle this problem and one way of doing this is by having strong monitoring and evaluation frameworks for different programmes and policies.
Yamini Aiyer, director of the Accountability Initiative at the Centre for Policy Research says that improving implementation would involve increasing the efficiency of the ‘murky middle’. The ‘murky middle’ consists of middle to lower level bureaucrats who are actually tasked with managing the money flow from the planners to the implementers. The issue here is not the amount of money but how the money reaches its intended beneficiaries.
Diverse contexts and programmes: “In India, programmes always work in certain regions and they always fail in some others. Is there something in the DNA of these regions that make implementation a disaster in these regions?,” asked a speaker at the event. The question resonated quite strongly with many in the audience. But the question itself missed the point because it was looking through the wrong end of the pipe. If we have learned only one lesson in poverty reduction programmes and international development, it is that context matters: programming must take into account local social and structural factors, such as culture, gender, poverty, education, as well as how local bureaucracies work (or don’t). This is true of several national policies, such as the National Rural Employment Guarantee Act, National Rural Health Mission, and the National Rural Livelihoods Mission. Evaluations of these programmes need to be considered at the state, or at least the regional level. It was clear at the event that Dr. Chibberunderstands this basic requirement for sound programming (and evaluation). The fact that he envisages regional offices for the IEO is a major step in this direction.
Using evaluation findings: “India has a small appetite for evidence,” said a researcher from India. The generation of evidence is an important challenge. But its use is an even bigger challenge. Political expediencies are a crucial factor in determining the longevity of programmes. In India, we tend to immortalise programmes by naming them after leaders and politicians. So, you have the Mahatma Gandhi National Rural Guarantee Scheme, Jawaharlal Nehru National Urban Renewal Mission, or the Rajiv Gandhi Scheme for empowerment of adolescent girls.
In a country where policies are determined by political compulsions, Dr. Chibber’s commitment to releasing the IEOs reports simultaneously to the government and to the media is a great idea. This will hopefully lead to better and more informed decision-making. The IEO can also use its convening power to help generate debates and policy dialogues around its findings. They will need to find champions at all levels of government and cultivate strong relationships with and find allies among researchers, the media and civil society.
The independence of the IEO: Even though the IEO is a government entity, the word independent in its name is very important. Being truly independent is going to be challenging.
CONEVAL the Mexican government’s independent evaluation office is an illustration of how this can be made possible. Set up about a decade ago, it has now grown to be a successful evaluation institution that is a valued source of evidence that does contribute to evidence-informed policymaking. Dr. Gonzalo Hernández Licona, executive secretary of CONEVAL was at the IEO launch and also spoke about the importance of maintaining independence and how they worked to maintain it in his agency. It is important for key government players to know that the IEO has equivalents elsewhere in the world that are working well and producing valuable information for governments.
As the world’s largest democracy takes its first steps in the direction of institutionalising evaluation, we at 3ie are hopeful and cheering for the IEO’s success. We think that the IEO can become a powerful and influential institution in its own right. We realise that policymaking is a complex process and evidence is only one part of the decision-making equation. But the IEO can ensure that the evidence is there and that it is sound and worth serious consideration in decision-making. We wish the IEO of India lots of success and hope that it becomes the next CONEVAL – a role model that other countries can emulate.