Understanding real-world complexities for greater uptake of evaluation findings

Most evaluations largely ignore the complexities of the context in which development programs are designed and implemented. This two-part blog series by 3ie Senior Fellow Michael Bamberger underscores the need for and challenges to designing ‘complexity-responsive’ evaluations. Bamberger discusses solutions and techniques that help evaluators take into consideration real-world factors that affect project implementation and outcomes.

Most development programs are designed and implemented in complex political, socio-cultural, economic and ecological contexts where outcomes are influenced by many factors over which program management has little or no control. These factors interact in different ways in different project locations. Consequently, a project, which has a clear design and implementation strategy, may produce significantly varying outcomes in different locations or at different points in time.

Despite the widespread acknowledgement by evaluators and stakeholders that project evaluations are ’complex’, most evaluations apply designs that assume the project is ’simple’ with a clearly defined relationship between the project inputs and a defined (and usually limited) set of outcomes. For example, most quantitative evaluations adopt a pre-test post-test comparison group design where outcomes are measured for the project and a comparison group at the start of the project (baseline) and at the end line. Project impacts are defined as the difference in the rate of change of the two groups over the life of the project. A variant of this design provides the underlying logic for randomized control trials, propensity score matching, double difference, instrumental variable estimation, regression discontinuity and pipeline designs.

What unites these quantitative evaluation designs is the lack of any systematic consideration of the multiple factors affecting a program’s implementation and outcomes. In fact, most widely-used quantitative evaluation designs eliminate complexity by matching the project and control (comparison) groups so as to eliminate the effects of external factors and estimate the effects of the project intervention ’when everything else is held constant’. The evaluations assess how the project would operate under controlled laboratory conditions, but largely fail to assess how projects perform in the face of real-world complexities.  

Space does not permit a discussion of how different qualitative evaluations treat complexity, nor a consideration of the different challenges in the treatment of complexity in project, sector program, country level and policy evaluations.  

Figure 1:  Complexity Map

Mapping complexity
One of the challenges in understanding complexity is that many discussions are very technical and theoretical. Breaking it into the following four interrelated components (figure 1) makes complexity more understandable to policymakers and program staff:

• Intervention dimensions. Projects and programs have subcomponents that can be rated in terms of level of complexity (see box 1 for examples).
• Interactions among stakeholders and other actors. Large programs may involve funding agencies, consultants, government agencies and line ministries, civil society, and community, business and special interest groups. Their interactions, differing priorities and approaches can turn even a ’simple’ project into a complex evaluation challenge.
• Internal and external systems. These include political, economic, administrative and legal systems, socio-cultural influences, public services and infrastructure, and the influence of social media. They operate within unique historical traditions that influence how programs work in a particular sector or country and often resist change (inertia).
• Process of causality and change. This can vary from relatively simple (with a clear linear relationship between program inputs and a limited set of outcomes) to relatively complex systems with multiple pathways linking program inputs to a wide range of outcomes.

A ‘complexity-responsive evaluation’ requires an analysis of the level of complexity of each of these four components, and how interactions among them affect the implementation and outcomes of the project. In collaboration with colleagues, I have developed a complexity checklist  which can be used to rate the level of complexity of each component and sub-component. This helps policymakers, managers and other non-specialists to understand the dimensions of complexity, and to rate projects on whether they are sufficiently complex to justify the use of a complexity-responsive evaluation.

Box 1

Implications of ignoring complexity
The lack of attention to complexity can have serious implications for the practical utility of evaluation findings. When multiple factors promoting and constraining the achievement of desired outcomes are not well understood, it becomes difficult to use the findings to design more effective projects and policies. Moreover, policymakers and managers may consider the findings of simple evaluations too unrealistic to be of practical utility¹.  As projects and policies produce a much wider range of outcomes than those addressed in conventional evaluations, project designers miss the opportunity to broaden the range of positive program outcomes. In addition to these, projects have different effects – both positive and negative – on different groups, but conventional evaluations often only capture the average effects for the total population, frequently overlooking the consequences for vulnerable or less vocal groups. Thus, the one-size-fits-all approach to projects fails to adapt projects to different contexts and to the needs of different groups.  

Why do so many evaluations ignore complexity?
Despite the widespread recognition that most development programs involve elements of complexity, only a very small proportion of evaluations in fact systematically address complexity. There are several reasons for this:

• The evaluation profession tends to be conservative with many evaluators continuing to use the methods – mainly designed to assess simple, linear models – they know and in which they have invested considerable time and resources to develop.
• Many clients continue to request the kinds of evaluation with which they are familiar and there has been relatively little demand for complexity-responsive evaluations.
• Evaluation procurement procedures are often very rigid and constrain the ability of evaluators to propose new methods or that new issues, such as complexity, be addressed.
• Finally, many complexity evaluation methods such as systems analysis, systems dynamics and social network analysis require access to large quantities of data, which, until recently, were not available to most evaluators. The increasing availability of big data and data analytics means that many kinds of large data sets are now becoming accessible.

Building on this discussion, the next piece in this blog series offers practical approaches for the evaluation of complex programs. It reviews the wide range of complexity-focused evaluation tools and the new sources of big data that are now becoming available – that make it possible for evaluators to begin to address complexity in a more systematic way.

¹A 2021 report by CECAN (Centre for the Evaluation of Complexity across the Nexus) found that many UK policymakers considered that most evaluations were of very limited practical utility because they were based on simple (linear) models that largely ignored the complex contexts in which most policies are designed and implemented.