Importance of mixed-methods approaches in development research and evaluation

As necessary as conventional quantitative methods might be, the importance of adopting a mixed-methods approach in order to understand and answer complex development questions cannot be overemphasized. In the first blog of this two-part series, 3ie Senior Research Fellow Michael Bamberger explains how an integrated mixed-methods approach goes much beyond incorporating just a few tools from both the quantitative and qualitative traditions and why that is more effective for addressing real-world problems.

Most development researchers and evaluators – working within either the quantitative or qualitative research traditions [section 1.1] – regularly draw up tools and techniques from both.  Mixed-methods writers refer to these traditions as paradigms [section 1.2]  Many quantitative impact evaluations incorporate case studies, focus groups, key informant interviews, or other qualitative techniques into their research to provide depth of understanding of organizational processes or differences of behavior and attitudes among different groups within the project population. Similarly, qualitative researchers may use quantitative sample-selection techniques or convert unstructured data from interviews or observations into a form where quantitative statistical analysis can strengthen the representativity and generalizability of the data.

While the benefits of using tools and techniques from both traditions are widely recognized and practiced, in most evaluations – due to reasons of researcher training combined with cost and time constraints – one of the two approaches is almost always dominant.  For example, a quantitative evaluation may only incorporate qualitative methods after the quantitative analysis has been completed to help explain some unanticipated findings. Similarly, a qualitative evaluation may use quantitative techniques to strengthen the generalizability of the findings and to address criticisms that the sample was selected purposively and that the findings are biased. For some researchers, while the tools and techniques of the other tradition are useful, they are considered less rigorous (for example, when qualitative data are considered “anecdotal”) and not an integral part of the evaluation design. Consequently, there is an important distinction between the incorporation of a few tools and techniques from the other tradition and a systematic mixed-methods approach that gives equal weight to both quantitative and qualitative methodologies in all stages of the evaluation.

Figure 1.1 The mixed-methods research/evaluation continuum

Benefits of an integrated mixed-methods approach

An integrated mixed-methods design can broaden the range of perspectives the evaluation can bring to the complex development questions being studied. It can also strengthen the rigor of both the quantitative and qualitative components of the evaluation by combining the breadth and generalizability provided by quantitative approaches with the depth of understanding of processes, attitudes, and behavior provided by qualitative approaches. Table 1.1 lists some of the ways in which mixed-methods can strengthen evaluation design and use.

Mixed methods can be used in all sectors

Mixed methods have been used in all development sectors, including poverty assessments, education, agriculture and irrigation, health and water supply, gender analysis and as part of a broad organizational evaluation strategy [section 1.3].  The wide applicability of mixed methods is shown in the following two examples:

• Mixed methods were used in Mozambique to assess the effectiveness and scalability of community-based, patient-centered delivery systems in reducing the high attrition rates in an anti-retroviral program (Rutenberg and Heard 2018). The evaluation combined:
  • An electronic database to input paper-based tools (patient-held appointment card, clinic-based patient file and a group monitoring form) to track monthly enrolment, patient follow-up, and outcomes
  • Qualitative study of stakeholders focusing on relevance, group dynamics, and impact
  • Informal patient interviews to track changing patient needs
  • Monitoring and qualitative data to track implementation fidelity, gaps, and factors contributing to success outcomes
  • A quasi-experimental design to track retention
  • Propensity-score matching to compare mortality rates

This mixed-methods approach offered some advantages. A number of sources that could not easily be quantified (informal interviews, observation, review of various paper forms) were used to track implementation fidelity. The changing needs of patients could be assessed and triangulation of different data sources could be used to increase reliability and to provide perspectives of different stakeholders for interpreting the findings.

• Mixed methods were used in Cartagena, Colombia to understand the importance of interhousehold transfers as an informal social safety net for low-income (particularly female-headed) households, and to assess the operational feasibility of incorporating these transfers into the estimation of household capacity to pay for low-income housing (Wandsworth, Jones and Kappaz 2000). The evaluation combined:
  • Participant observation (a researcher lived with a family in the study community for six weeks) to understand the cultural patterns driving these transfers, and to guide the development of the survey questionnaire
  • A randomly-selected household survey to quantify the different forms of transfer of money, food and other goods, and services, and to record the household attributes of the receiving and sending household
  • Econometric analysis to estimate the influence of household attributes and the sex of the household head on the volume and direction of transfers

The main advantage of mixed methods was to be able to observe the complex cultural mechanisms determining the patterns of transfers which cannot be understood or captured through conventional quantitative survey methods. At the same time, the econometric analysis was able to estimate the volume of transfers and to understand the factors determining the volume and direction of transfers.

Different approaches to the use of mixed methods

Mixed-methods approaches can be classified according to whether:

1. evaluation design is quantitative, qualitative, or giving equal weight to both paradigms.
   Figure 1 shows there is a continuum  
2. the quantitative and qualitative approaches are used sequentially (one after the other), or concurrently (both at the same time)
3. mixed-method approaches are used at a single stage of the evaluation (e.g., data collection), at several stages, or at all stages through an integrated mixed-methods design
4. mixed methods are used at a single level (for example, village-level schools) or at multiple levels (e.g., all levels of the state education system)
5. nested design is used with multi-level sampling (Tashakkori, Johnson and Teddlie 2021
   pp 173-176)

The importance of triangulation

A key feature of mixed methods is the use of triangulation to improve validity through the systematic comparison of two or more independent estimates of key variables. For example, estimates of household income could be obtained from an income and expenditure survey, direct observation of the quality of house construction, obtaining information on household consumer durables through unstructured interviews and direct observation, and social maps where families rate the relative wealth or poverty of other families in the community. These estimates are compared and if estimates are inconsistent there is a follow-up to understand reasons for the differences and to agree upon the best estimate.  

One important application of triangulation is during the data collection phase. During pilot testing of the survey questionnaire, a qualitative research specialist could revisit a sample of interviewed households to further explore some of the key questions covered in the survey. The quantitative and qualitative estimates are compared to determine which kinds of information are consistent with both methods, and which produce significantly different estimates. All members of the research team can then meet to discuss the reasons for the differences and to discuss whether any revisions are needed as to how the questions are formulated or the information collected.  An example of a factor explaining inconsistent information obtained on household income could be which member of the household provided the information. The wife may have more accurate information on expenditures, so if the woman answered she might give a higher figure for expenditures. However, in many households, the man will insist on answering.

Part II of the blog discusses how to design and implement a mixed-methods evaluation.