Integrating mixed-methods approaches with impact evaluations like process evaluations and implementation studies has been in practice for a while, and for very good reasons. In this blog, we share how embedding a process evaluation within an ongoing evaluation of an aquaculture intervention in Bangladesh is adding value to evaluation findings. From checking for patterns in implementation to highlighting changes in the theory of change to helping qualify baseline data, the process evaluation is helping us in at least five different ways.
As Dixon and Bamberger (2022) highlight in this paper, a process evaluation delves into project context and emerging outcomes of implementation activities to check whether implementation proceeded according to plan. The main goals of the assessment are to contribute to the explanation of any positive, adverse, significant, insignificant, or unanticipated outcomes of an intervention. Process evaluations can clarify how various organizational, administrative, economic, political, sociocultural, and other factors besides the intervention played a role in the implementation process.
While the quantitative causal impact of the Bangladesh project is yet to be measured, the process evaluation is already helping us explain the drivers of the interventions’ results. It is helping us test whether assumptions in the theory of change (ToC) are valid and which of the components of the interventions (figure 1) are working well and why. A critical role of the process evaluation is also to capture changes and disruptions in implementation, including those arising from the COVID-19 pandemic, and how these changes would affect the logic of the project and potentially, our evaluation of outcomes.
The process evaluation of the aquaculture project is a critical component of our mixed methods impact evaluation. The project aims to help smallholder farmers enhance aquaculture productivity, increase the value of marketed fish and enhance the quality, reach, efficiency, and sustainability of extension services around aquaculture. It also seeks to increase access to nutrient-rich foods, mainly fish, and to recognize women in the fishing value chain.
The project has two intervention arms. The first is a pure local service provider (LSP) or private sector model, where the implementing agency directly works with LSPs in selected areas, imparting training on good aquaculture practices. In the other model, the hybrid model, the agency imparts training to two large local NGOs. These NGOs are responsible for identifying and training their own pool of LSPs. In both models, LSPs are responsible for forming and working with farmer groups to promote aquaculture best practices, create credit linkages with financial institutions, and linkages to quality input sources like fish feed and fingerlings, among other components.
Checking for spillovers
If an LSP provides training to farmers in a control union, our evaluation would be less able to tease out the value added by the interventions. Hence, it was important to understand the extent of possible spillovers in our process evaluation. Using data from our phone-based survey of LSPs, we found that LSPs largely worked within the unions the implementing agency, or the NGOs assigned them to form farmer groups and deliver extension services. We found that the implementing agency played a key role in ensuring that spillovers were limited through proper mobilization of LSPs and consistent monitoring.
Tracing implementation patterns
From our telephonic surveys, we observed some interesting differences in the operation of the LSPs in private sector and hybrid model unions. In hybrid model unions, we found LSPs focused more on nutrition and gender training of fish farmers, while LSPs in the direct private sector model focused more on aquaculture productivity training. This was because, unlike LSPs in the latter unions, LSPs in hybrid model unions received support from community health and nutrition workers hired by the NGOs.
The process evaluation also helped us understand changes and additions to intervention subcomponents. After Phase 1 of the project in 2020, the implementing agency realized that LSPs from the direct private sector model areas could not effectively deliver gender transformative behavior change communication. As a result, they started experimenting with new approaches like street plays and interactive games within a few areas under both models to provide nutrition and gender messages. These messages were better received by farmers. However, we found that the implementing agency is not planning to scale up these experiments to more treatment areas because there is very little time left in the project to scale up. So, the process evaluation helped us understand the coverage of these sub-interventions and the context influencing how they might work towards the intended outcomes.
Clarifying issues of targeting
The pandemic prompted the implementing agency to provide access to live training sessions and various kinds of information on aquaculture practices and access to inputs and nutrition through a Facebook group. From focus group discussions with farmers, we found that most of them, especially women farmers, did not have access to smartphones. To address this issue, the project team tried to add members of farmer households who owned a smartphone to this Facebook group even though the LSPs did not directly target them. In some cases, these smartphone users were not engaged in aquaculture and were not interested in it. Thus, the process evaluation also helped us understand these targeting issues concerning digital training.
Measuring the quality of implementation
COVID-19 lockdowns that kicked in shortly after the start of the project hampered regular follow-up and training of farmers. Farmers mobilized early in the project expressed disappointment with the lengthy gap between training sessions and lack of follow-up. To maintain continuity, the implementing agency disseminated short messages on nutrition and aquaculture practices. But process evaluation showed that most farmers didn't read these short messages on their phones. Farmers mobilized later in the project after the lockdowns had been lifted had better recall and satisfaction around the training sessions provided by LSPs.
Adding depth to the baseline data
In the baseline survey for the evaluation, we found that the awareness of aquaculture best practices was high on average. However, the process evaluation showed that the levels of awareness and adoption of best practices varied by the size of the pond managed by the farmers. Small pond owners have less understanding of the details of many of these good practices compared to medium pond owners, especially ones that had recently started aquaculture. Most medium pond owners reported being aware of best practices even before the project because they were engaged in more commercial aquaculture and received information from various sources like government fisheries departments and input dealers. This gives us clues for future analysis, telling us why the intervention components focused on knowledge of best practices may have differentiated outcomes.
