Implications of the COVID-19 pandemic on impact evaluation activities in East Africa

Overview

The ongoing COVID-19 pandemic has affected virtually every sphere of life across the world, including impact evaluations. To better understand these effects, we conducted a rapid informal survey of evaluation researchers in East Africa via email. A total of 71 respondents from Kenya, Uganda, Tanzania, Ethiopia and Rwanda participated in the survey between 9-24 April. This blog post presents our findings.

Our survey shows that evaluation research has been seriously disrupted in East Africa. The vast majority of evaluation researchers (87%) have stopped all in-person field work, although a slight majority (51%) were still conducting research via the internet. Most (64%) respondents reported that all meetings were now taking place online. Two-thirds of respondents reported new delays or other issues in their collaborations, and a quarter of respondents have already seen funding delayed. Most respondents expect fewer funding opportunities in the future. More details on our findings are below.

First things first: Who are we?

Perhaps you are not very familiar with what impact evaluations are all about. Well, impact evaluations are rigorous scientific studies often used by governments and development organisations with the aim of understanding if and how well a program worked. This short animation by the World Bank provides simple examples of how impact evaluations work in the real world.

As for us, we are the Network of Impact Evaluation Researchers in Africa (NIERA) ─ a network of African scholars seeking to advance decision-focused impact evaluations of development programs through capacity building, evidence generation and policy outreach. Our membership is drawn primarily from alumni of the East Africa Social Science Translation (EASST) Collaborative that is hosted by the Center for Effective Global Action, University of California at Berkeley.

Our membership is spread across five East African countries: Kenya, Uganda, Tanzania, Ethiopia and Rwanda. Our members have high-level academic degrees and experience in conducting rigorous evaluations in various thematic areas including agriculture, health, economics, education, finance and gender.

NIERA’s Secretariat is hosted at the United States International University in Africa (USIU-A) in Nairobi, Kenya. Our core funder is the Hewlett Foundation.

Survey method and participants

The survey was conducted online via Google Forms. A link to the survey was shared by email to all NIERA members and their relevant networks. The questionnaire included a total of 18 multiple choice questions and one open-ended question. Invited participants were required to first read the survey information and then indicate by a click of a button whether they accepted to participate. All questions in the survey were optional and participants were free to skip any that they did not feel comfortable with.

A total of 71 people participated in the survey, with 67% identifying as male and the rest as female. All five NIERA member countries were represented in the survey, though a slight plurality of participants (34%) were from Tanzania. Most participants had a master’s degree (64%) and more than ten years of work experience (62%).

Scale of the pandemic’s impact

Working conditions

A majority (70%) of respondents reported that their work had been partially disrupted by the pandemic, with 23 per cent indicating that their work had been totally disrupted. About 56 per cent were fully working from home. Among those whose work typically involved fieldwork (e.g. data collection surveys), a plurality (49%) indicated that no form of fieldwork was ongoing while 38% had fully shifted to virtual fieldwork (e.g. online surveys). The remaining respondents (13%) were conducting both in-person and online research work.
Most respondents (64%) also indicated that all work-related meetings were now being held virtually. A plurality (42%) reported that they were unable to continue delivering or receiving training or mentoring as a result of the pandemic. On a positive note, 51 per cent indicated that they had more time for scientific writing, including papers and proposals.

External collaborations

About 66 per cent of respondents indicated that their collaborations with external partners (local and international) had either been put on hold or had declined. A third of respondents reported that their communication with funding organisations had deteriorated, and 6 per cent had not even heard from any of their funding partners. A strong plurality of respondents (49%) indicated that their ability to engage with policymakers had been partially disrupted and another 34 per cent said that this ability was totally disrupted.

Funding prospects

Although just under 30 per cent of respondents reported that their funding sources had not been impacted by the pandemic, 25 per cent reported that their funding had been delayed. A further 13 per cent indicated that their funding had been diverted to the COVID-19 response. A clear majority (68%) indicated that they anticipated that their future funding opportunities would be reduced as a result of the pandemic though 11 per cent anticipated the reverse.

General concerns

The leading concern expressed among respondents was the risk of the pandemic to their personal health and safety. This primary concern was followed closely by worries about the impact of the pandemic on the economy as well as the inability of their governments to adequately control the pandemic.

Role of impact evaluations in ending the pandemic

An overwhelming majority (86%) of respondents indicated that impact evaluations are critical to addressing the pandemic. Some proposed areas of intervention include: 1) testing novel approaches to virtual learning/education; 2) estimating the socio-economic impact of the pandemic; 3) understanding the impact of government policy responses to the pandemic.

Going forward

It is evident that the COVID-19 pandemic has taken a negative toll on impact evaluation activities in East Africa. Nevertheless, those engaged in impact evaluation activities have adapted to the situation in various ways, like teleworking and focusing more on scientific writing. It is troubling, albeit not unexpected, that communication with external audiences such as funders and policymakers have taken a major hit. For impact evaluations to be effective in any context, there needs to be continuous engagement across the spectrum of stakeholders in the evaluation ecosystem. We are encouraged to see several initiatives aimed at supporting the wider field of international development through this difficult period. For example, several philanthropic organisations have pledged to continue to support their partners over this period. NIERA’s core funder – the Hewlett Foundation – has reassured us about its commitment to supporting our work.

The role that impact evaluations could play in addressing the pandemic cannot be overstated. We have already seen a wide range of public and private initiatives across Africa aimed at addressing the health, social and economic impacts of the pandemic. This is an opportune time to apply impact evaluation methods in determining if and how these interventions work. Such knowledge will be crucial either for scaling up these interventions, revising them, or halting them completely. Once the pandemic is over, impact evaluations can play a useful role in retrospectively assessing what has worked, in order to improve our preparedness for future pandemics. We are pleased to see that some East African governments and an increasing number of funding agencies such as the European & Developing Countries Clinical Trials Partnership (EDCTP) are offering emergency funding for COVID-19 research in Africa, although not necessarily with an explicit focus on impact evaluations.

Like every study, this one has some limitations. The sample size is by no means representative of the full range of impact evaluation experts in East Africa. Furthermore, the largely multiple-choice questionnaire limits our understanding of the nuances behind each response. Nevertheless, it is our hope that this rapid survey has provided useful information about the current state of impact evaluation work in East Africa. Of course, the situation remains quite fluid as we write this blog post. Overall, NIERA remains committed to advancing the use of impact evaluations in informing policy-making across the sub-region with the ultimate vision of improving lives and advancing inclusive, sustainable development in Africa.

This is a guest post by authors from NIERA.

 - Samuel Oti (Secretary-General, NIERA), Annet Adong, Getachew Mullu Kassa, Grace Mhalu, Ronald Mulebeke, Samuel Muhula (NIERA), Jennifer Nyakinya (Program Manager, NIERA), Anne M Maina (Program Assistant, NIERA), Amos Njuguna (Chairperson, NIERA)