3ie ~ International Initiative for Impact Evaluation

skip to content

Unraveling the opaque through impact evaluations

Markus Olapade

“If anybody tells you to conduct an impact evaluation, tell that person to go to hell!” 

This was the comment made by a renowned impact evaluator at a recent conference after Khalid Al Kudair presented his NGO Glowork's remarkable success in mobilising the female labour force in Saudi Arabia. The evaluator was trying to make the point that if a programme is obviously working, it is a waste of time and money to conduct an impact evaluation. I disagree.

In Saudi Arabia, many women want to work but they often face difficulties getting their applications through to the human resources department of an organisation. Glowork’s online portal solves this problem by bringing demand and supply together. Since many women prefer to work from home, Glowork created a work-from-home scheme, allowing women to set up an office in their own house with employer supervision of the home worker. Glowork’s system seems to have created a win-win situation: it gives families an additional source of income and allows the economy to benefit from a well-educated part of the population that had been excluded from the labour market.

The fact that women have been brought into the workforce through this system seems to show the success of the idea. Yet the comment of the impact evaluator kept ringing in my ear. I kept asking myself whether an impact evaluation could uncover something more. Marx summed up my doubts very well when he said, 'If the essence and appearance of things directly coincided, all science would be superfluous’. Assessing success by merely looking at the number of women brought into the workforce might be deceiving. What if there is something going on under the surface?

Let’s take the example of microcredit. Until very recently we were quite sure about its positive effect. The high number of people taking up credit suggested a success story. But there is now a debate about whether microcredit really works and if so, then for whom. Impact evaluation is able to provide answers by examining why some benefit while others don’t. The microfinance crisis in Andhra Pradesh, India, clearly showed that people do not always benefit from receiving loans. The risk of microcredit is that people can get pulled into a debt-trap. Borrowers may end up cross-financing their loans, taking up one loan to pay back another one. Both the unintended consequences and the positive impacts of microcredit are revealed only by rigorous impact evaluations. It’s clear that impressive stats on the high number of people taking up microcredit did not tell us the whole story. 

The example of microcredit shows that the numbers on take-up tell only part of the story. The highly skilled and experienced commentator at the conference should have known that he is making a conclusion about the impact of Glowork from monitoring data. Yes, the number of women participating in the labour force did increase. And yes, female employment increased as well. But what these changes have meant for the woman’s role in the household is still unclear. What do these numbers really convey about the empowerment of women?

As the founder of Glowork pointed out, female labour market participation changes the distribution of tasks in a household. Does the weight of a woman’s voice increase merely by giving her a job? We hope it does! But a woman’s new job could also have negative unintended effects on her bargaining power vis-à-vis her husband.

Monitoring cannot tell us if such changes are taking place. Impact evaluation on the other hand can do this by peeling away layers of opacity and shedding light on issues like empowerment and social transformation. A comprehensive theory of change, which should be the basis of any impact evaluation, unravels not only the direct causal pathway but also factors in possible negative externalities or spill-overs. Does a new job imply that the woman spends less time on household chores? What is the effect on the upbringing of children? How does it affect household dynamics? These second generation questions are much more interesting and useful for programme design than the first generation question of whether the chances of being employed increase as a result of Glowork.

So I disagree with the comment that impact evaluation is unnecessary for Glowork. Khalid Al Khudair, Glowork’s founder, showed great insight in deciphering the problems of women in the Saudi labour market. He was also highly creative in finding solutions. The results of an impact evaluation are likely to only further fuel his creativity in designing more effective programmes.

(This blog is authored by Markus Olapade as a reflection from his participation in the Arab Youth and Entrepreneurship conference in Doha. Markus is an Evaluation Specialist at 3ie)

Published on: 28 feb 2013