“Hard a starboard!” These were the orders of First Officer William Murdoch aboard RMS Titanic on the night of 14 April 1912 immediately after lookouts signaled a rather large obstacle ahead. Though still half a kilometer away, the gigantic ship nonetheless struck the iceberg that was in its way. It takes time for a big ship to veer away from disaster. This is perhaps why the reform of the world’s multilateral development system to address global challenges was reportedly the most closely watched issue at the recently concluded annual meetings of the World Bank and International Monetary Fund in Marrakech.

Initial accounts from Marrakech show some significant accomplishments. In a statement leading up to the meetings, US Treasury Secretary Janet Yellen noted, a year since she kick-started the process, “the tremendous progress we’ve made.” There has been broad buy-in from the G-20 on the multilateral development bank (MDB) reform agenda, which has been called the “evolution road map”. The World Bank has shifted its vision from its twin goals to end extreme poverty and boost shared prosperity to “a world free of poverty on a livable planet”. The MDBs, including the Asian Development Bank (ADB), have agreed to work more closely to achieve the transformative changes needed if such visions are to avoid becoming delusional.

The course shifts of the MDBs will have profound implications for the work program of the evaluation community which assesses their actions. I will focus on three types of interventions that I think will loom larger and pose challenges for us as evaluators, based on some recent evaluations done at the ADB. These are: support to address global public goods, especially climate change, the private sector, and enhance the enabling environment for project success.

Evaluating support for global public goods

There will be a need to do more evaluations of activities that address global public goods. They include interventions meant to open international markets, to control health pandemics and, most cited of all, to address climate change, the existential ‘iceberg’ looming dead ahead.

The resources that need to be generated to address climate change needs in developing countries are substantial. With no plans for major capital increases, the MDBs are becoming more innovative in using guarantees and loosening lending limits to generate some of the needed resources. Just recently, news outlets reported on the ADB’s announcement that it will be able to increase its annual lending by 40% through such measures without jeopardizing its AAA status. The World Bank Group announced in Marrakech that it could lend $157 billion more over the next decade through similar measures. Even more will be needed – the UN system is estimating that to achieve the 2030 Agenda for Sustainable Development, there is a gap in funding of $4 trillion a year.

The support these large amounts will finance will mean a significant boost in the work program of evaluators in a relatively new area of work. When the ADB prepared its first major dedicated climate change evaluation just three years ago, it encountered at least three major challenges that future studies must address.

First, it was difficult to identify what to evaluate. MDBs self-tag which projects to designate under the overall climate change umbrella. Some projects – such as support to develop renewable energy alternatives – are clearly climate-related. But for many others, the justifications for the tagging are less clear. There may be errors of inclusion. For example, the ADB’s evaluation found that climate accounting was driven by financial objectives rather than projected climate outcomes. An analysis by the Center for Global Development earlier this year found that the World Bank “has developed a climate portfolio that lacks estimates of greenhouse gas (GHG) emissions reductions and has no standardized reporting on GHG estimates. Further, hundreds of projects tagged climate—many in poorer countries—appear to have little to do with climate change mitigation or adaptation.” The Bank promised a revamp.

There may also be errors of exclusion. After all, any development intervention which enhances economic activity will have potential knock-on effects which could have significant effects on climate change. Some of these effects could be unintended and count as development impacts, which do not affect overall ratings of project success.

Second, the outcomes of climate change interventions are hard to observe directly in the time frame of most evaluations. While the immediate effects of mitigation interventions on energy efficiency can be measured, the ultimate impacts on global emissions and warming will not be observed until (hopefully) far into the future. The success of adaptation interventions, especially those that provide protection for greater risk of disasters, is difficult to assess when the risks are not realized. Evaluations must rely more on theory-based reasoning or modeling to predict outcomes.

Third, for all interventions to address a problem of a global commons, attribution to an intervention by a single institution, already a challenge for most evaluations, will be even more difficult.

Evaluating support for the private sector

There is widespread recognition that development challenges cannot be addressed by the public sector alone. According to the 2023 report of independent experts to the G20 Presidency on Strengthening MDBs, “One of the greatest opportunities for transformation is in MDB engagement with the private sector.”

In response, MDBs have ambitious plans to grow their private sector operations. ADB’s Strategy 2030 calls for at least one-third of all ADB projects to be private sector by 2024. At present, only around one-quarter of operations are non-sovereign, accounting for around 10% of the total value of all ADB investments. According to their strategy documents, the International Finance Corporation (IFC) and the European Bank for Reconstruction and Development (EBRD) aspire to boost lending by 28% and 30%, respectively, by 2025.

The evaluation of private sector operations has a well-established tradition, including its own “Good Practice Standards” developed by the Evaluation Cooperation Group of all major MDBs. Among the challenges identified by a recent major ADB evaluation is the difficulty of justifying MDB support, not only for their financial returns, but also for their development impact. They must also show “additionality” – that they do not crowd out private sources of finance. At the ADB, roughly a third of self-ratings on additionality were downgraded in 2022 because of the lack of evidence to substantiate these claims.

Another issue is that if MDBs are going to increase and be truly innovative in their support for the private sector, they will need to assess risk prudently while still being timely. A preliminary finding of an ongoing evaluation finds that ADB could be clearer in defining its risk appetite and in aligning incentives of operations with those of the risk assessment teams.

Evaluations beyond investment projects

Project lending by MDBs in most countries is a drop in the bucket relative to needs and relative to what the countries themselves are already doing. According to a recent report, even if all of overseas development assistance were targeted on the 60 poorest countries, it would amount to only 1/25th of their collective economies. And “if ODA is small relative to the size of low-income countries’ (LICs) GDP, it is miniscule relative to the middle-income country (MIC) economies.”

For their financing to make more of a difference in such an environment, MDBs intend, first, to expand budget support operations to bolster policy reforms which affect all public spending. ADB is considering lifting its caps on budget support for both its regular as well as concessional and grant operations. Second, MDBs will be doing more to transfer knowledge to countries which are constrained, much less by a dearth of external financing, and more by ideas and capacity. Even before the idea of being ‘climate change banks’ took hold, many MDBs, including the ADB, styled themselves as ‘knowledge banks’.

Recent evaluations by ADB provide some insights into the challenges in evaluating these interventions. Evaluations of budget support operations need to assess the alignment of policy reforms with meaningful outcomes and their grounding in diagnostics, theory and policy realities. Traditional efficiency metrics have limited relevance. Evaluations of knowledge are hampered by the fact that the subject matter is of great value but is intangible and outcomes are difficult to measure. The impact of knowledge investments is even more difficult to track and measure.

In conclusion

The evolving agenda for MDBs will change evaluation. This note focused on at least three areas where evaluation must focus – climate change, the private sector and going beyond investment support. This work will provide important lessons for MDBs as they try to shift their directions. Given the urgency of the problems at hand, perhaps evolution is the wrong word to describe what they must do and by when. After all, after First Officer Murdoch gave the order and the Titanic started its leftward turn to avoid the iceberg, the collision was already imminent.

---------------

This is a guest blog post by an author at the Asian Development Bank, a 3ie member organization. The views outlined in this blog do not necessarily reflect the views of 3ie or its board.