Language Bias Among Chinese-Sponsored Randomized Clinical Trials in Systematic Reviews and Meta-analyses—Can Anything Be Done?

Clinical investigators working in non–English-speaking countries are more likely to report their studies in an international, English-language journal if the results are positive and tend to publish negative findings in their local languages,¹ which can cause exaggerated or even false-positive associations if investigators of systematic reviews and meta-analyses do not identify and include all relevant studies written in a non-English language.² Although language bias has been reported for randomized clinical trials (RCTs), it is incompletely clear whether there is any language bias specifically in Chinese-sponsored RCTs (CS-RCTs).

In JAMA Network Open, Jia et al³ compared CS-RCTs published in English with those published in Chinese. Using a retrospective cohort study, the authors verified that CS-RCTs with positive results are more likely to be published in English and indexed in English bibliographic databases than those with negative results. Based on this finding, the authors suggested an effect of language and indexing bias on the evaluation of RCT-based clinical evidence and recommended removing language restrictions and actively searching for Chinese bibliographic databases to reduce the influence of these biases on systematic reviews and meta-analyses.

Systematic reviews and meta-analyses of RCTs provide important evidence for informing decision-making in clinical practice and for identifying areas in which further research is needed. A principle of conducting systematic reviews is to comprehensively review all existing evidence, whereas most systematic reviews and meta-analyses only include RCTs published in English. As Jia et al³ suggested in their study, 60% of systematic reviews set a language restriction when searching English bibliographic databases, whereas only 3% searched at least 1 Chinese bibliographic database for clinical trials.

Because the number of CS-RCTs is increasing rapidly, it is important to include Chinese CS-RCTs when conducting systematic reviews and meta-analyses. By searching at ClinicalTrials.gov for the clinical trials initiated and sponsored by an organization or a person in mainland China, we found that the number of CS-RCTs increased every year, and the total number of CS-RCTs reached 13 175 by February 18, 2020, 7.5 times of that by 2011 (Figure). Of note, the number of CS-RCTs published in Chinese-language journals is also impressive. A previous study by Jia et al⁴ showed that the total number of published drug-related CS-RCTs has increased from 16 000 by 2006 to 44 000 by 2016, among which nearly 90% of CS-RCTs were published in Chinese-language journals. Because the negative results derived from CS-RCTs are more likely to be reported in Chinese- than in English-language journals, the overall weight of negative results can be underestimated if Chinese CS-RCTs are not included. Consequently, the value of a systematic review and meta-analysis to clinicians, policy makers, and other readers can be depreciated because relevant Chinese CS-RCTs, including those of good quality, are not comprehensively identified and included. Grading of Recommendations, Assessment, Development and Evaluations (GRADE) has also recommended rating down 1 level of evidence owing to such bias.⁵

Figure.  The Cumulative Number of Chinese-Sponsored Randomized Clinical Trials (CS-RCTs) Registered in ClinicalTrials.gov in the Past 10 Years

View LargeDownload

Interestingly, previously published studies have already indicated that researchers were more likely to publish RCTs in English-language journals if the results were statistically significant, and that language bias may be brought into systematic reviews and meta-analyses if only trials reported in English were included.⁶ As CS-RCTs become more common and increasingly important for reaching evidence-based conclusions, the study by Jia et al³ is significantly noteworthy and can be meaningful for the practice and presentation of future systematic reviews and meta-analyses. The potential effect of this study may further extend to changing or improving international searching engines.

On the other hand, 1 point of potential interest not included in the study by Jia et al³ might be a comparison of academic qualities between CS-RCTs published in Chinese vs English, and it is unclear whether including Chinese CS-RCTs will introduce other biases to subsequent systematic reviews and meta-analyses. The results of previous studies about differences in qualities among RCTs published in English and non-English languages vary and are sometimes controversial. For instance, a study by Egger et al⁶ has shown that differences in quality between studies in English and German did exist although probably were small. Nevertheless, another study by Moher et al⁷ found no significant difference between the results of the meta-analyses including vs excluding RCTs in non-English journals. In contrast, however, an early study by Wu et al⁸ in 2009 reported that only 6.8% of RCTs published in Chinese-language journals adhered to accepted methods for randomization, yet few studies have been performed to follow up on improvements in recent years. Therefore, further investigation is needed to clarify whether and how inclusion of CS-RCTs published in Chinese, and maybe also other non-English languages, affects the conclusions of a systematic review and meta-analysis.

To reduce the language bias associated with CS-RCTs and Chinese bibliographic databases, as suggested by Jia et al³ and the Cochrane Handbook for Systematic Reviews of Interventions, investigators of systematic reviews and meta-analyses in the future may need to actively search Chinese bibliographic databases and include relevant high-quality Chinese CS-RCTs. In such a context, abstracts of Chinese CS-RCTs need to provide more information, such as the design, key parameters, and findings of the studies indexed so that readers are able to more accurately evaluate the weight of the report. In parallel, inviting Chinese-speaking colleagues as collaborators from the very beginning of systematically searching Chinese bibliographic databases might also be helpful. Furthermore, searching registries of clinical trials, such as ClinicalTrials.gov and the Chinese Clinical Trial Registry, might help trace Chinese CS-RCTs that have not yet been published or cannot be indexed in major international bibliographic databases. Moreover, a major international bibliographic database, such as MEDLINE, needs to index more high-quality Chinese journals, given the fact that less than 6% of the 2500 Chinese journals indexed in Chinese bibliographic databases are currently indexed in MEDLINE. Taken together, Chinese CS-RCTs, as well as RCTs published in other non-English languages, have emerged as nontraditional sources on which evidence-based medicine relies and should be appropriately captured. Failure to maintain a global outlook may result in opportunistic dissemination of selected results.

Published: May 28, 2020. doi:10.1001/jamanetworkopen.2020.6370

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Mao C et al. JAMA Network Open.

Corresponding Author: Mengfeng Li, MD, PhD, Cancer Institute, Southern Medical University, Guangzhou, Guangdong 510515, China (limf@mail.sysu.edu.cn).

Conflict of Interest Disclosures: None reported.