Language Bias Among Chinese-Sponsored Randomized Clinical Trials in Systematic Reviews and Meta-analyses—Can Anything Be Done? | Global Health | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Invited Commentary
Medical Journals and Publishing
May 28, 2020

Language Bias Among Chinese-Sponsored Randomized Clinical Trials in Systematic Reviews and Meta-analyses—Can Anything Be Done?

Author Affiliations
  • 1Department of Epidemiology, School of Public Health, Southern Medical University, Guangzhou, China
  • 2Cancer Institute, Southern Medical University, Guangzhou, China
JAMA Netw Open. 2020;3(5):e206370. doi:10.1001/jamanetworkopen.2020.6370

Clinical investigators working in non–English-speaking countries are more likely to report their studies in an international, English-language journal if the results are positive and tend to publish negative findings in their local languages,1 which can cause exaggerated or even false-positive associations if investigators of systematic reviews and meta-analyses do not identify and include all relevant studies written in a non-English language.2 Although language bias has been reported for randomized clinical trials (RCTs), it is incompletely clear whether there is any language bias specifically in Chinese-sponsored RCTs (CS-RCTs).

In JAMA Network Open, Jia et al3 compared CS-RCTs published in English with those published in Chinese. Using a retrospective cohort study, the authors verified that CS-RCTs with positive results are more likely to be published in English and indexed in English bibliographic databases than those with negative results. Based on this finding, the authors suggested an effect of language and indexing bias on the evaluation of RCT-based clinical evidence and recommended removing language restrictions and actively searching for Chinese bibliographic databases to reduce the influence of these biases on systematic reviews and meta-analyses.

Systematic reviews and meta-analyses of RCTs provide important evidence for informing decision-making in clinical practice and for identifying areas in which further research is needed. A principle of conducting systematic reviews is to comprehensively review all existing evidence, whereas most systematic reviews and meta-analyses only include RCTs published in English. As Jia et al3 suggested in their study, 60% of systematic reviews set a language restriction when searching English bibliographic databases, whereas only 3% searched at least 1 Chinese bibliographic database for clinical trials.

Because the number of CS-RCTs is increasing rapidly, it is important to include Chinese CS-RCTs when conducting systematic reviews and meta-analyses. By searching at ClinicalTrials.gov for the clinical trials initiated and sponsored by an organization or a person in mainland China, we found that the number of CS-RCTs increased every year, and the total number of CS-RCTs reached 13 175 by February 18, 2020, 7.5 times of that by 2011 (Figure). Of note, the number of CS-RCTs published in Chinese-language journals is also impressive. A previous study by Jia et al4 showed that the total number of published drug-related CS-RCTs has increased from 16 000 by 2006 to 44 000 by 2016, among which nearly 90% of CS-RCTs were published in Chinese-language journals. Because the negative results derived from CS-RCTs are more likely to be reported in Chinese- than in English-language journals, the overall weight of negative results can be underestimated if Chinese CS-RCTs are not included. Consequently, the value of a systematic review and meta-analysis to clinicians, policy makers, and other readers can be depreciated because relevant Chinese CS-RCTs, including those of good quality, are not comprehensively identified and included. Grading of Recommendations, Assessment, Development and Evaluations (GRADE) has also recommended rating down 1 level of evidence owing to such bias.5

Figure.  The Cumulative Number of Chinese-Sponsored Randomized Clinical Trials (CS-RCTs) Registered in ClinicalTrials.gov in the Past 10 Years
The Cumulative Number of Chinese-Sponsored Randomized Clinical Trials (CS-RCTs) Registered in ClinicalTrials.gov in the Past 10 Years

Interestingly, previously published studies have already indicated that researchers were more likely to publish RCTs in English-language journals if the results were statistically significant, and that language bias may be brought into systematic reviews and meta-analyses if only trials reported in English were included.6 As CS-RCTs become more common and increasingly important for reaching evidence-based conclusions, the study by Jia et al3 is significantly noteworthy and can be meaningful for the practice and presentation of future systematic reviews and meta-analyses. The potential effect of this study may further extend to changing or improving international searching engines.

On the other hand, 1 point of potential interest not included in the study by Jia et al3 might be a comparison of academic qualities between CS-RCTs published in Chinese vs English, and it is unclear whether including Chinese CS-RCTs will introduce other biases to subsequent systematic reviews and meta-analyses. The results of previous studies about differences in qualities among RCTs published in English and non-English languages vary and are sometimes controversial. For instance, a study by Egger et al6 has shown that differences in quality between studies in English and German did exist although probably were small. Nevertheless, another study by Moher et al7 found no significant difference between the results of the meta-analyses including vs excluding RCTs in non-English journals. In contrast, however, an early study by Wu et al8 in 2009 reported that only 6.8% of RCTs published in Chinese-language journals adhered to accepted methods for randomization, yet few studies have been performed to follow up on improvements in recent years. Therefore, further investigation is needed to clarify whether and how inclusion of CS-RCTs published in Chinese, and maybe also other non-English languages, affects the conclusions of a systematic review and meta-analysis.

To reduce the language bias associated with CS-RCTs and Chinese bibliographic databases, as suggested by Jia et al3 and the Cochrane Handbook for Systematic Reviews of Interventions, investigators of systematic reviews and meta-analyses in the future may need to actively search Chinese bibliographic databases and include relevant high-quality Chinese CS-RCTs. In such a context, abstracts of Chinese CS-RCTs need to provide more information, such as the design, key parameters, and findings of the studies indexed so that readers are able to more accurately evaluate the weight of the report. In parallel, inviting Chinese-speaking colleagues as collaborators from the very beginning of systematically searching Chinese bibliographic databases might also be helpful. Furthermore, searching registries of clinical trials, such as ClinicalTrials.gov and the Chinese Clinical Trial Registry, might help trace Chinese CS-RCTs that have not yet been published or cannot be indexed in major international bibliographic databases. Moreover, a major international bibliographic database, such as MEDLINE, needs to index more high-quality Chinese journals, given the fact that less than 6% of the 2500 Chinese journals indexed in Chinese bibliographic databases are currently indexed in MEDLINE. Taken together, Chinese CS-RCTs, as well as RCTs published in other non-English languages, have emerged as nontraditional sources on which evidence-based medicine relies and should be appropriately captured. Failure to maintain a global outlook may result in opportunistic dissemination of selected results.

Back to top
Article Information

Published: May 28, 2020. doi:10.1001/jamanetworkopen.2020.6370

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Mao C et al. JAMA Network Open.

Corresponding Author: Mengfeng Li, MD, PhD, Cancer Institute, Southern Medical University, Guangzhou, Guangdong 510515, China (limf@mail.sysu.edu.cn).

Conflict of Interest Disclosures: None reported.

References
1.
Boutron  I, Page  M, Higgins  J,  et al. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions, version 6. Cochrane; updated July 2019.
2.
Liberati  A, Altman  DG, Tetzlaff  J,  et al.  The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.   PLoS Med. 2009;6(7):e1000100. doi:10.1371/journal.pmed.1000100 PubMedGoogle Scholar
3.
Jia  Y, Huang  D, Wen  J,  et al.  Assessment of language and indexing biases among Chinese-sponsored randomized clinical trials.   JAMA Netw Open. 2020;3(5):e205894. doi:10.1001/jamanetworkopen.2020.5894Google Scholar
4.
Jia  Y, Huang  D, Wen  J,  et al.  Language bias and location bias among Chinese-sponsored drug-related randomized controlled trials.  In:  Abstracts of the 26th Cochrane Colloquium, Santiago, Chile. 2020;1 suppl 1. doi:10.1002/14651858.CD201901Google Scholar
5.
Guyatt  GH, Oxman  AD, Montori  V,  et al.  GRADE guidelines, 5: rating the quality of evidence—publication bias.   J Clin Epidemiol. 2011;64(12):1277-1282. doi:10.1016/j.jclinepi.2011.01.011 PubMedGoogle ScholarCrossref
6.
Egger  M, Zellweger-Zähner  T, Schneider  M, Junker  C, Lengeler  C, Antes  G.  Language bias in randomised controlled trials published in English and German.   Lancet. 1997;350(9074):326-329. doi:10.1016/S0140-6736(97)02419-7 PubMedGoogle ScholarCrossref
7.
Moher  D, Pham  B, Lawson  ML, Klassen  TP.  The inclusion of reports of randomised trials published in languages other than English in systematic reviews.   Health Technol Assess. 2003;7(41):1-90. doi:10.3310/hta7410 PubMedGoogle ScholarCrossref
8.
Wu  T, Li  Y, Bian  Z, Liu  G, Moher  D.  Randomized trials published in some Chinese journals: how many are randomized?   Trials. 2009;10(1):46. doi:10.1186/1745-6215-10-46 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    ×