Our latest blog in our series on regional trends in impact evaluation research takes a slightly different tack. Instead of surveying a region, we focus on a single country: China. China has emerged as a major hub for impact evaluation (IE) research in recent years. With China recently hosting the tenth Asia Evaluation Week, we thought it made sense to look at its evidence landscape in depth. Another blog will examine East Asia more broadly.
A dramatic rise in impact evaluations in China
A striking trend—one we noticed long before analyzing the data for this blog—is the rapid increase in China-focused impact evaluations in the past 10 years or so. Based on data from 3ie’s Development Evidence Portal (DEP), we have identified that:
- 3,735 impact evaluations have been conducted in China — the highest of any country, and more than twice India’s 1,526.
- Of the IEs conducted in China, about 450 are randomized controlled trials (RCTs).
- China has seen an extremely rapid increase in IE production over the last 10 years. Between 2000 and 2015, about 20 China-focused impact evaluations were published per year on average. From 2016 to 2022, the annual average was nearly 350. In that seven-year period, the number of IEs in the DEP as a whole saw a 90% increase, while the evidence from China grew by over 1,000%. Thus, China’s share of the global evidence base is growing dramatically.
- The striking rise in IE production stems primarily from an increase in the use of study designs that make use of existing data. In particular, studies with data on treatment and control groups using fixed effects estimation (including difference-in-differences techniques – DiD) to control for selection bias and confounding. While RCTs were the most commonly-used method in the early 2010s, their rate of production has held relatively steady at around 30-40 per year. Meanwhile, since 2016 the publication of DiD studies has skyrocketed. There were 27 China-focused DiD studies published in 2016. In 2022, there were 659. (See figure.)
- The DEP primarily contains English-language impact evaluations conducted in China, but even more are published in Chinese. In 2022 Xiuxia Li and colleagues conducted a review of social science RCTs by Chinese researchers over the period 2000 to 2020, which found that 112 of 316 studies were in Chinese.
What is being evaluated?
By far the most frequently evaluated intervention is environmental regulation, with 671 IEs. Evaluations of emissions trading schemes make up another 139 studies. This is a large literature base to make sense of, but our informal reviews of these studies suggest there are many policies being enacted at various administrative levels (which may not always be identified by the same names across various papers written in English). So while it’s difficult to keep track of exactly which environmental regulations are being evaluated, it appears to be a fairly heterogeneous set, rather than a few signature policies.
Likewise, environmental considerations loom large in terms of the outcomes most commonly measured. The top two outcomes are greenhouse gas emissions (n = 265 studies) and air pollution (n = 219).
Beyond environment-focused evaluations, studies on health interventions are also common, with 714 evaluations, or 19% of all evaluations from China.
It is important to note that these evaluations often cover overlapping interventions, with policies implemented at multiple administrative levels, making synthesis of findings complex.
The majority of the evidence base is nationally funded and generated
Perhaps not surprisingly, China stands out from most other low- and middle-income countries by the large share of research being funded by domestic sources and conducted by teams of national researchers. Countries vary by how typical it is for IEs focused on a country to have at least one author based at an institution in that country. China has one of the highest rates of national authorship (i.e., at least one author based at a Chinese institution) at 93% (only Iran has a higher rate, 97%); across all low- and middle-income countries, the overall rate of national authorship is 68%. The rate of studies with exclusively China-based authors is also among the highest at 69%.
93% of China-focused IEs include at least one China-based author, 69% are authored exclusively by China-based teams
The Chinese government is by far the largest funder of IE research in China. The top five funders of IEs in China are all Chinese government agencies, led by China’s National Science Foundations, which have funded 1,393 IEs. Together, these top five funders account for 50% of all IEs in China.
Opportunities and concerns
Taken together, these trends suggest that not only has China been active in enacting environmental policies, but there has been a concerted domestic effort – supported by the Chinese government and carried out by Chinese researchers – to evaluate the effects of such policies, using existing data. As 3ie has noted elsewhere, this approach may provide a useful template for the evaluation field at large, as a way to leverage existing data sets to generate insights on policy effectiveness (rather than relying on approaches that require expensive collection of new primary data).
At the same time, there are concerns about whether all studies meet quality standards. The Li et al review cited above notes that many of the RCTs it included did not meet international standards for what research methodology details should be reported for transparency. The DEP team began systematically checking for retracted articles in our database in 2024, and since then we’ve removed 10 retracted articles from DEP. All but one of these came from Chinese research teams.
China’s rise as a producer of impact evaluations is one of the most significant shifts in the global evidence landscape. The country has generated more evaluations than any other, with a strong focus on environmental regulation in particular. To maximise the value and usefulness of this significant research investment to global learning on what works it will be important to ensure that new studies are conducted in line with accepted standards for research ethics, transparency and reproducibility.
