Assessment of Language and Indexing Biases Among Chinese-Sponsored Randomized Clinical Trials | Global Health | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure.  Study Flowchart Identifying Published Chinese-Sponsored Randomized Clinical Trials (CS-RCTs)
Study Flowchart Identifying Published Chinese-Sponsored Randomized Clinical Trials (CS-RCTs)

ChiCTR indicates Chinese Clinical Trial Registry; CT.gov, ClinicalTrials.gov; DCTRP, Drug Clinical Trial Registry Platform; and PI, principal investigator.

Table 1.  Coverage of Journal Articles by Bibliographic Databases
Coverage of Journal Articles by Bibliographic Databases
Table 2.  Positivity of CS-RCTs by Trial Registry and Bibliographic Database
Positivity of CS-RCTs by Trial Registry and Bibliographic Database
Table 3.  Factors Associated With Language Bias and Indexing Bias
Factors Associated With Language Bias and Indexing Bias
1.
Higgins  JPT, Green  S.  Cochrane Handbook for Systematic Reviews of Interventions. Vol 4. John Wiley & Sons; 2011.
2.
Egger  M, Zellweger-Zähner  T, Schneider  M, Junker  C, Lengeler  C, Antes  G.  Language bias in randomised controlled trials published in English and German.   Lancet. 1997;350(9074):326-329. doi:10.1016/S0140-6736(97)02419-7 PubMedGoogle ScholarCrossref
3.
Page  MJ, Shamseer  L, Altman  DG,  et al.  Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.   PLoS Med. 2016;13(5):e1002028. doi:10.1371/journal.pmed.1002028 PubMedGoogle Scholar
4.
Jüni  P, Holenstein  F, Sterne  J, Bartlett  C, Egger  M.  Direction and impact of language bias in meta-analyses of controlled trials: empirical study.   Int J Epidemiol. 2002;31(1):115-123. doi:10.1093/ije/31.1.115 PubMedGoogle ScholarCrossref
5.
National Science Board. Science and Engineering indicators 2018: the rise of China in science and engineering. Published 2018. Accessed April 21, 2020. https://www.nsf.gov/nsb/sei/one-pagers/China-2018.pdf
6.
Zhao  H, Zhang  J, Guo  L,  et al.  The status of registrations, ethical reviews and informed consent forms in RCTs of high impact factor Chinese medical journals.   Chinese J Evidence-Based Med. 2018;(7):16. doi:10.7507/1672-2531.201802027Google Scholar
7.
Elsevier. List of journal titles in Embase. Accessed July 14, 2019. https://www.elsevier.com/solutions/embase-biomedical-research/embase-coverage-and-content
8.
Cohen  JF, Korevaar  DA, Wang  J, Spijker  R, Bossuyt  PM.  Should we search Chinese biomedical databases when performing systematic reviews?   Syst Rev. 2015;4(1):23. doi:10.1186/s13643-015-0017-3 PubMedGoogle ScholarCrossref
9.
Vandenbroucke  JP, von Elm  E, Altman  DG,  et al; STROBE initiative.  Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration.   Ann Intern Med. 2007;147(8):W163-94. doi:10.7326/0003-4819-147-8-200710160-00010-w1 PubMedGoogle Scholar
10.
World Health Organization. International Clinical Trials Registry Platform (ICTRP): joint statement on public disclosure of results from clinical trials. Published 2018. Accessed April 21, 2020. https://www.who.int/ictrp/results/en/
11.
Center for Drug Evaluation of China Food and Drug. About the Drug Clinical Trial Registry Platform. Accessed April 21, 2020. http://www.chinadrugtrials.org.cn/eap/11.snipet.block
12.
United States Food and Drug Administration. Drugs@FDA: FDA approved drug products. Accessed April 21, 2020. https://www.accessdata.fda.gov/scripts/cder/daf/
13.
European Medicine Agency. Search medicines. Accessed April 21, 2020. https://www.ema.europa.eu/en/medicines
14.
Hartung  DM, Zarin  DA, Guise  J-M, McDonagh  M, Paynter  R, Helfand  M.  Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications.   Ann Intern Med. 2014;160(7):477-483. doi:10.7326/M13-0480 PubMedGoogle ScholarCrossref
15.
Ross  JS, Tse  T, Zarin  DA, Xu  H, Zhou  L, Krumholz  HM.  Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis.   BMJ. 2012;344:d7292. doi:10.1136/bmj.d7292 PubMedGoogle ScholarCrossref
16.
Rethlefsen  ML, Farrell  AM, Osterhaus Trzasko  LC, Brigham  TJ.  Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews.   J Clin Epidemiol. 2015;68(6):617-626. doi:10.1016/j.jclinepi.2014.11.025 PubMedGoogle ScholarCrossref
17.
Koffel  JB.  Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors.   PLoS One. 2015;10(5):e0125931. doi:10.1371/journal.pone.0125931 PubMedGoogle Scholar
18.
Xia  J, Wright  J, Adams  CE.  Five large Chinese biomedical bibliographic databases: accessibility and coverage.   Health Info Libr J. 2008;25(1):55-61. doi:10.1111/j.1471-1842.2007.00734.x PubMedGoogle ScholarCrossref
19.
Saldanha  IJ, Scherer  RW, Rodriguez-Barraquer  I, Jampel  HD, Dickersin  K.  Dependability of results in conference abstracts of randomized controlled trials in ophthalmology and author financial conflicts of interest as a factor associated with full publication.   Trials. 2016;17(1):213. doi:10.1186/s13063-016-1343-z PubMedGoogle ScholarCrossref
20.
Easterbrook  PJ, Berlin  JA, Gopalan  R, Matthews  DR.  Publication bias in clinical research.   Lancet. 1991;337(8746):867-872. doi:10.1016/0140-6736(91)90201-Y PubMedGoogle ScholarCrossref
21.
Decullier  E, Lhéritier  V, Chapuis  F.  Fate of biomedical research protocols and publication bias in France: retrospective cohort study.   BMJ. 2005;331(7507):19. doi:10.1136/bmj.38488.385995.8F PubMedGoogle ScholarCrossref
22.
De Angelis  C, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.   Circulation. 2005;111(10):1337-1338. doi:10.1161/01.CIR.0000158823.53941.BE PubMedGoogle ScholarCrossref
23.
Li  Y.  Chinese medical doctors negotiating the pressure of the publication requirement.   Ibérica Rev la Asoc Eur Lenguas para Fines Específicos. 2014;(28):107-127.Google Scholar
24.
Emerging Technology from the arXiv. The truth about China’s cash-for-publication policy. Published July 12, 2017. Accessed April 21, 2020. https://www.technologyreview.com/s/608266/the-truth-about-chinas-cash-for-publication-policy/
25.
Galandi  D, Schwarzer  G, Antes  G.  The demise of the randomised controlled trial: bibliometric study of the German-language health care literature, 1948 to 2004.   BMC Med Res Methodol. 2006;6(1):30. doi:10.1186/1471-2288-6-30 PubMedGoogle ScholarCrossref
26.
Morrison  A, Polisena  J, Husereau  D,  et al.  The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies.   Int J Technol Assess Health Care. 2012;28(2):138-144. doi:10.1017/S0266462312000086 PubMedGoogle ScholarCrossref
27.
Pham  B, Klassen  TP, Lawson  ML, Moher  D.  Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.   J Clin Epidemiol. 2005;58(8):769-776. doi:10.1016/j.jclinepi.2004.08.021 PubMedGoogle ScholarCrossref
28.
Morton  S, Berg  A, Levit  L, Eden  J.  Finding What Works in Health Care: Standards for Systematic Reviews. National Academies Press; 2011.
29.
Liu  Q, Tian  L-G, Xiao  S-H,  et al.  Harnessing the wealth of Chinese scientific literature: schistosomiasis research and control in China.   Emerg Themes Epidemiol. 2008;5(1):19. doi:10.1186/1742-7622-5-19 PubMedGoogle ScholarCrossref
30.
Mao  L, Zhang  P, Ding  J,  et al.  Quality assessment of randomized controlled trial report in the field of breast cancer published in Chinese.   Chinese J Gen Pract. 2016;19(05):615-620. doi:10.3969/j.issn.1007-9572.2016.05.029 Google Scholar
31.
Tian  J, Huang  Z, Ge  L,  et al.  Evaluation of quality of randomized controlled trials research papers among Chinese abstracts for breast cancer.   Chinese J Gen Pract. 2016;14(7):1182-1185. doi:10.16766/j.cnki.issn.1674-4152.2016.07.039 Google Scholar
32.
Wu  T, Li  Y, Bian  Z, Liu  G, Moher  D.  Randomized trials published in some Chinese journals: how many are randomized?   Trials. 2009;10(1):46. doi:10.1186/1745-6215-10-46 PubMedGoogle ScholarCrossref
33.
Vassar  M, Yerokhin  V, Sinnett  PM,  et al.  Database selection in systematic reviews: an insight through clinical neurology.   Health Info Libr J. 2017;34(2):156-164. doi:10.1111/hir.12176 PubMedGoogle ScholarCrossref
34.
Viergever  RF, Karam  G, Reis  A, Ghersi  D.  The quality of registration of clinical trials: still a problem.   PLoS One. 2014;9(1):e84727. doi:10.1371/journal.pone.0084727 PubMedGoogle Scholar
35.
Viergever  RF, Ghersi  D.  The quality of registration of clinical trials.   PLoS One. 2011;6(2):e14701. doi:10.1371/journal.pone.0014701 PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Medical Journals and Publishing
    May 28, 2020

    Assessment of Language and Indexing Biases Among Chinese-Sponsored Randomized Clinical Trials

    Author Affiliations
    • 1Bloomberg School of Public Health, The Johns Hopkins University, Baltimore, Maryland
    • 2Welch Medical Library, The Johns Hopkins University, Baltimore, Maryland
    • 3Institute of Information and Medical Library, Peking Union Medical College and Chinese Academy of Medical Sciences, Beijing, China
    • 4Department of Medicine, School of Medicine, The Johns Hopkins University, Baltimore, Maryland
    • 5Michigan Medicine, School of Public Health, University of Michigan, Ann Arbor
    JAMA Netw Open. 2020;3(5):e205894. doi:10.1001/jamanetworkopen.2020.5894
    Key Points español 中文 (chinese)

    Question  Do language and indexing biases exist among Chinese-sponsored randomized clinical trials on drug interventions?

    Findings  This cohort study included 891 eligible Chinese-sponsored randomized clinical trials identified from English- and Chinese-language clinical trial registries. Among 470 published trials as of August 2019, positive trial findings were more commonly published in English-language journals and indexed in English-language bibliographic databases than negative trial findings.

    Meaning  These findings suggest that language and indexing biases may lead to distorted, more positive results of drug interventions when synthesizing evidence.

    Abstract

    Importance  Language and indexing biases may exist among Chinese-sponsored randomized clinical trials (CS-RCTs). Such biases may threaten the validity of systematic reviews.

    Objective  To evaluate the existence of language and indexing biases among CS-RCTs on drug interventions.

    Design, Setting, and Participants  In this retrospective cohort study, eligible CS-RCTs were retrieved from trial registries, and bibliographic databases were searched to determine their publication status. Eligible CS-RCTs were for drug interventions conducted from January 1, 2008, to December 31, 2014. The search and analysis were conducted from March 1 to August 31, 2019. Primary trial registries were recognized by the World Health Organization and the Drug Clinical Trial Registry Platform sponsored by the China Food and Drug Administration.

    Exposures  Individual CS-RCTs with positive vs negative results (positive vs negative CS-RCTs).

    Main Outcomes and Measures  For assessing language bias, the main outcome was the language of the journal in which CS-RCTs were published (English vs Chinese). For indexing bias, the main outcome was the language of the bibliographic database where the CS-RCTs were indexed (English vs Chinese).

    Results  The search identified 891 eligible CS-RCTs. Four hundred seventy CS-RCTs were published by August 31, 2019, of which 368 (78.3%) were published in English. Among CS-RCTs registered in the Chinese Clinical Trial Registry (ChiCTR), positive CS-RCTs were 3.92 (95% CI, 2.20-7.00) times more likely to be published in English than negative CS-RCTs; among CS-RCTs in English-language registries, positive CS-RCTs were 3.22 (95% CI, 1.34-7.78) times more likely to be published in English than negative CS-RCTs. These findings suggest the existence of language bias. Among CS-RCTs registered in ChiCTR, positive CS-RCTs were 2.89 (95% CI, 1.55-5.40) times more likely to be indexed in English bibliographic databases than negative CS-RCTs; among CS-RCTs in English-language registries, positive CS-RCTs were 2.19 (95% CI, 0.82-5.82) times more likely to be indexed in English bibliographic databases than negative CS-RCTs. These findings support the existence of indexing bias.

    Conclusions and Relevance  This study suggests the existence of language and indexing biases among registered CS-RCTs on drug interventions. These biases may distort evidence synthesis toward more positive results of drug interventions.

    Introduction

    In non–English-speaking countries, researchers can choose to publish randomized clinical trials (RCTs) in English-language journals or journals in their native language. It is established that RCTs with positive results (hereinafter referred to as positive RCTs) are more likely to be published in English-language journals, a phenomenon termed language bias.1 This tendency may lead to disproportionally more positive RCTs in English literature and consequently more RCTs with negative findings (hereinafter referred to as negative RCTs) in non-English literature.2

    Ideally, this bias would not threaten the validity of systematic reviews because reviewers should comprehensively search for all existing evidence regardless of the language1; however, estimates indicate that almost 40% of systematic reviews were reportedly restricted to English-language articles indexed in English bibliographic databases.3 This estimate raises the concern that such reviews may miss negative RCTs that are only published in the non-English literature, leading to biased evidence.4

    Recently, scientific publications from Mainland China have been surging.5 Publications of RCTs sponsored by researchers in Mainland China (Chinese-sponsored RCTs [CS-RCTs]) are also split between English- and Chinese-language journals (English CS-RCTs and Chinese CS-RCTs).6 However, limited evidence is available regarding language bias among CS-RCTs. Whether we should include Chinese CS-RCTs indexed in English bibliographic databases to reduce the effect of speculated language bias is unknown.

    A further challenge, and one more difficult to address, is that most Chinese journals have not been indexed in English bibliographic databases owing to large quantity and varying quality.7 This challenge implies that systematic reviewers must not only remove language restrictions from searching English bibliographic databases but also actively search Chinese bibliographic databases to capture all Chinese CS-RCTs, a practice seldomly adopted by the systematic review community.8 Bias may exist if Chinese CS-RCTs with positive results are more likely to be indexed in English bibliographic databases than their negative counterparts, which are more commonly seen in Chinese bibliographic databases. We term this potential residual of language bias as indexing bias.

    Currently, the Cochrane Handbook for Systematic Reviews of Interventions only recommends searching Chinese bibliographic databases for systematic reviews on Chinese herbal medicine.1 Whether the recommendation should be extended to drug interventions is unknown. The objective of this study was to evaluate the existence of language and indexing biases among CS-RCTs on drug interventions to inform the potential update of the recommendation.

    Methods

    In this retrospective cohort study, we retrieved CS-RCTs from trial registries and searched bibliographic databases to determine their publication status. Two hypotheses were predefined: (1) positive CS-RCTs were more likely to be published in English than negative CS-RCTs (language bias), and (2) positive CS-RCTs were more likely to be indexed in English than Chinese bibliographic databases (indexing bias). Because this study was a literature review based on open-source data and did not include research participants, it was not subject to institutional review board approval. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.9

    Identifying CS-RCTs From Trial Registries

    We retrieved CS-RCTs from all 17 primary registries recognized by the World Health Organization10 and the Drug Clinical Trial Registry Platform (DCTRP) sponsored by the China Food and Drug Administration.11 A substance was considered a drug if recognized and regulated by the US Food and Drug Administration and/or the European Medicines Agency.12,13 We included all CS-RCTs that started January 1, 2008, and were completed by December 31, 2014, to allow a minimum of 4.5 years from trial completion to publication.14 We excluded phase 1 trials (including bioequivalence and pharmacokinetics studies) and CS-RCTs missing the study period or RCTs with an unclear study interval (eg, end date before the start date) or any unnamed experimental drug, principal investigator, or sponsor in the registries.

    Identification of Journal Articles From Bibliographic Databases

    The search and analysis were conducted from March 1 to August 31, 2019. We only included journal articles produced from eligible CS-RCTs. Conference abstracts, research letters, and dissertations were not included. Publications of protocols, subgroup analyses, secondary analyses, and meta-analyses were also excluded.

    Based on previous studies,14,15 we developed search strategies for individual CS-RCTs with informationists from the Welch Medical Library at The Johns Hopkins University (L.R.) and the Institute of Information and Medical Library at Peking Union Medical College (Q.C.) (eTable 1 in the Supplement).16,17 We expanded the search terms with synonyms and spelling variations to increase sensitivity. The search terms were tailored and organized for each bibliographic database based on the database’s specific syntax (eTable 2 in the Supplement). Seven bibliographic databases, including 3 English language (PubMed, Embase, and the Cochrane Central Register of Controlled Trials [CENTRAL]) and 4 Chinese language (China National Knowledge Infrastructure, SinoMed, VIP information, and Wanfang Data), were subsequentially searched.18

    We conducted a 4-step process to identify matches of eligible CS-RCTs. First, we searched bibliographic databases to retrieve citations; second, we screened the citations for eligible CS-RCTs; third, we downloaded PDFs of possibly eligible trials; and fourth, we matched the PDFs with the registry records of eligible CS-RCTs. The criteria for screening and matching are shown in eTable 3 in the Supplement.

    The journal articles were classified as confirmed matches and probable matches according to the similarity between journal articles and registry records. Confirmed matches indicated the journal articles were consistent with the registry records, whereas probable matches indicated the journal articles were similar to the registry records but differed on or lacked only 1 data item. The primary analysis was conducted among the confirmed and probable matches, whereas a sensitivity analysis was conducted among the confirmed matches only.

    Two authors (Y.J., D.H.) independently searched bibliographic databases and identified matching PDFs of eligible CS-RCTs. Discrepancies were discussed and resolved by a third author (J.W.). If multiple journal articles existed, we only considered the one with the largest sample size or the earliest one if identical sample size was reported in multiple articles.

    Statistical Analysis
    Exposure

    The exposure was the finding of individual CS-RCTs (positive vs negative) according to the CS-RCT’s primary outcome reported in the journal article.19 If multiple primary outcomes were reported in a CS-RCT, we selected the first one reported in the result section. If no primary outcome was defined, the selection of the CS-RCT’s primary outcome was based on the following hierarchical order: the first outcome used in the sample size calculation, the first outcome defined in the study objective, or the first outcome reported in the Results section. When the time point was not specified for the CS-RCT’s primary outcome that was measured at multiple time points, we considered the last point in our main analysis and the first point in sensitivity analyses.

    We defined a positive result as favoring the experimental group with statistical significance in superiority trials or showing no difference between treatment groups for equivalence or noninferiority trials. Results that were not statistically significant, significantly favored the control group, or failed to show equivalence or noninferiority were defined as negative.

    Outcome

    Two main outcomes were defined: the language of the publication (English vs Chinese) and the language of the bibliographic database where the publication was indexed (English vs Chinese). An article published in both Chinese and English was considered published in English; similarly, an article indexed in both English and Chinese bibliographic databases was considered indexed in an English bibliographic database. We assumed all English-language articles were indexed in English bibliographic databases, but Chinese-language articles were possibly indexed in English or Chinese bibliographic databases.

    Measurements of Associations

    Bias was estimated by relative risk (RR), including point estimates and 95% CIs. An RR larger than 1.00 indicated that positive CS-RCTs were more likely than negative CS-RCTs to be published in English or indexed in English bibliographic databases. The RRs were estimated using log binomial models with 5 covariates: sample size (<100 vs ≥100), funding source (industry vs nonindustry), study design (superiority vs noninferiority or equivalence), number of recruitment centers (single vs multiple), and registration type (prospective vs retrospective).20,21 Industrial funding was considered as long as 1 funder was from industry; prospective registration was considered when registration occurred before the first participant was recruited.22 We included an interaction term in the models to evaluate the heterogeneity of bias across registries. Statistical significance was defined as 2-sided P < .05 for the main effect and P < .10 for interaction, generated by Wald χ2 tests as in the log binomial models. SAS, version 9.4 (SAS Institute Inc) was used for data cleaning and analysis.

    Results

    The search through trial registries and bibliographic databases was conducted from March 1 to August 31, 2019. Among the 17 primary registries and DCTRP, 5 were found to include eligible CS-RCTs: the Chinese Clinical Trial Registry (ChiCTR), ClinicalTrials.gov, International Standard Randomized Controlled Trials Number (ISRCTN), the Australia New Zealand Clinical Trials Registry (ANZCTR), and DCTRP. In total, 5084 CS-RCTs were retrieved from these trial registries and screened for eligibility. Eventually, 891 eligible CS-RCTs were identified, 470 (52.8%) of which had been published in 229 English-language journals and 72 Chinese-language journals. The screening results are shown in the Figure.

    Characteristics of CS-RCTs

    Among the 470 journal articles corresponding to 470 CS-RCTs, 368 (78.3%) were published in English and 102 (21.7%) were in Chinese; 432 (91.9%) were confirmed matches to registry records and 38 (8.1%) were probable matches. Thirty of 38 probable matches (79.0%) were published in Chinese. The matching results are shown in eTable 3 in the Supplement.

    The distribution of CS-RCTs across bibliographic databases is shown in Table 1. The 3 English bibliographic databases only indexed a small proportion of Chinese articles, ranging from 22 of 102 (21.6%) by PubMed to 24 of 102 each (23.5%) by Embase and CENTRAL. The low coverage of the China National Knowledge Infrastructure was mainly due to unindexed or partially unindexed medical journals sponsored by the Chinese Medical Association.

    Most CS-RCTs (306 [65.1%]) were registered in ChiCTR, followed by ClinicalTrials.gov (143 [30.4%]), DCTRP (13 [2.8%]), ISRCTN (4 [0.9%]), and ANZCTR (1 [0.2%]). The CS-RCTs in DCTRP, ISRCTN, and ANZCTR were too few to be analyzed separately. We expected that both biases would be similar across ClinicalTrials.gov, ISRCTN, and ANZCTR, so we combined CS-RCTs from these 3 registries to form an English registry category. On the other hand, DCTRP, which was only available in Chinese and was not a primary registry, was excluded from the inferential analyses. Of the 470 CS-RCTs, 323 (68.7%) were positive, 2 (0.4%) were excluded from inferential analyses because of unknown positivity (no intergroup comparison), 286 (60.9%) recruited at least 100 participants, 322 (68.5%) were conducted at a single center, 377 (80.2%) were supported by nonindustry funding, 315 (67.0%) were retrospectively registered, and 442 (94.0%) were superiority trials.

    The distribution of covariates, including the sample size, funding source, study design, number of recruitment centers, and registration type, was similar between English and Chinese CS-RCTs (eTable 4 in the Supplement). The distribution of covariates among CS-RCTs indexed in English bibliographic databases was similar to that of CS-RCTs only indexed in Chinese bibliographic databases, although the number of positive CS-RCTs indexed in Chinese bibliographic databases was slightly larger than the number of negative CS-RCTs (eTable 5 in the Supplement).

    Language Bias

    Four hundred sixty-eight CS-RCTs were included for this analysis. As shown in Table 2, positive CS-RCTs were more likely to be published in English (180 of 211 [85.3%] vs 31 of 211 [14.7%]) and were more likely to be indexed in English bibliographic databases (186 of 211 [88.2%] vs 25 of 211 [11.8%]) compared with CS-RCTs published in Chinese. After adjusting for covariates, positive CS-RCTs were more commonly published in English than negative CS-RCTs. The RRs were 3.92 (95% CI, 2.20-7.00) and 3.22 (95% CI, 1.34-7.78) among CS-RCTs registered in ChiCTR or English-language registries, respectively (Table 3). The interaction between registry and positivity of CS-RCTs was not statistically significant (Wald χ2 = 0.13; P = .71), indicating no evidence of heterogeneity of language bias across registries. Other factors associated with increased likelihood of being published in English among CS-RCTs were a sample size of at least 100 (RR, 2.09; 95% CI, 1.19-3.67), single-center as opposed to multicenter trials (RR, 1.85; 95% CI, 1.01-3.41), and financial support from a nonindustrial source compared with an industrial source (RR, 1.99; 95% CI, 1.06-3.75).

    Indexing Bias

    Four hundred sixty-eight CS-RCTs were included for this analysis. As shown in Table 2, English-language CS-RCTs (91 of 104 [87.5%] vs 13 of 104 [12.5%]) or CS-RCTs indexed in English bibliographic databases (93 of 104 [89.4%] vs 11 of 104 [10.6%]) were more likely to be positive than Chinese-language CS-RCTs or CS-RCTs only indexed in Chinese bibliographic databases. After adjusting for covariates, positive CS-RCTs were more commonly indexed in English bibliographic databases than negative CS-RCTs. The RRs were 2.89 (95% CI, 1.55-5.40) and 2.19 (95% CI, 0.82-5.82) among CS-RCTs registered in ChiCTR or English-language registries, respectively (Table 3). The interaction between registry and language of bibliographic databases was not statistically significant (Wald χ2 = 0.22; P = .64), indicating no evidence of heterogeneity of indexing bias across registries. The only other factor associated with an increased likelihood of being indexed in English bibliographic databases among CS-RCTs was sample size of at least 100 (RR, 2.04; 95% CI, 1.11-3.72).

    Sensitivity Analyses

    Two sensitivity analyses were conducted. When only confirmed matches (n = 432) were analyzed, the RRs increased regarding language bias (4.14 [95% CI, 2.09-8.21] among ChiCTR and 3.58 [95% CI, 1.32-9.72] among English-language registries) and decreased regarding indexing bias (2.47 [95% CI, 1.15-5.34] among ChiCTR and 1.92 [95% CI, 0.62-6.04] among English-language registries). When the first assessment of the CS-RCTs’ primary outcomes was analyzed (rather than the last assessment at the end of follow-up), the RRs increased regarding both language bias (5.34 [95% CI, 3.00-9.68] among ChiCTR and 3.73 [95% CI, 1.53-9.07] among English-language registries) and indexing bias (4.76 [95% CI, 2.49-9.08] among ChiCTR and 2.67 [95% CI, 1.01-7.11] among English-language registries).

    Discussion

    Our study supports the existence of language bias and indexing bias among CS-RCTs included in trial registries. As hypothesized, positive CS-RCTs were more likely to be published in English or indexed in English bibliographic databases compared with negative CS-RCTs.

    Language Bias

    Reputation, job prospects, and academic progress may critically depend on publishing in English-language journals among Chinese researchers.23,24 Positive CS-RCTs are more likely to be submitted to English-language journals because they typically have a higher chance of being accepted; accordingly, English-language journals contain more positive CS-RCTs than their Chinese counterparts.

    Theoretically, language bias disappears if all clinical trials shift to be published in English. This ideal has been echoed by a trend toward publishing in English in some countries, such as Germany.25 With the mean number of RCTs per German-language journal decreasing from a maximum of 11.2 annually from 1970 to 1986 to only 1.7 annually from 2002 to 2004, language bias from German-speaking countries may no longer be a concern.

    We did not detect such a trend among CS-RCTs. According to an ongoing study, the number of CS-RCTs published in Chinese may be as many as 44 000 in 2016 as opposed to fewer than 1000 being published in English (Y.J., Jun Liang, J.W., et al; unpublished data; June 2020). The deep gap between the Chinese and English literature has allowed significant space for language bias to develop.

    Several studies attempted to evaluate the effect of language bias based on non–English-language trials included in systematic reviews.26,27 Because most systematic reviews were constrained to work within English bibliographic databases only, what those studies measured was a fraction of language bias—the difference between English-language and non–English-language trials indexed in English bibliographic databases. The effect of language bias cannot be comprehensively evaluated unless non–English-language trials, especially the ones not indexed in English bibliographic databases, are included and evaluated.

    Indexing Bias

    As the primary source for systematic reviewers, English bibliographic databases may index some non–English-language literature, but they vary in the amount and scope. The Cochrane Handbook for Systematic Reviews and the United States Institute of Medicine Guidelines for Systematic Reviews have recommended including non–English-language literature indexed in English bibliographic databases.1,28 Including non–English-language trials indexed in English bibliographic databases may not eliminate the effect of language bias but could reduce it to the scope of indexing bias.

    To date, English bibliographic databases do not represent the Chinese-language literature. This is problematic because an ongoing study shows more than 10 000 clinical trials have been published out of China in 2016 (Y.J., Jun Liang, J.W., et al; unpublished data; June 2020). However, Embase only indexes 80 Chinese-language journals.7 Although we did not simulate actual systematic reviews, it appears plausible that, owing to language and indexing bias, drug interventions might appear more positive than they are when existing evidence is synthesized, for example in systematic reviews.

    How to Eliminate the Effect of Language Bias Regarding CS-RCTs

    The effect of language bias regarding CS-RCTs might be eliminated if reviews comprehensively searched Chinese bibliographic databases, if major English bibliographic databases would index all Chinese-language literature, or if all CS-RCTs would be appropriately registered with results. There are, however, layers of complexity that warrant appreciation. There has been a discussion as to whether scientists should search Chinese bibliographic databases when conducting systematic reviews.8,29 Our study tipped the scales in this proposition’s direction: including Chinese-language literature may reduce bias and shrink the CIs of the estimates. Currently, the Cochrane Handbook for Systematic Reviews of Interventions only recommends searching Chinese bibliographic databases for topics related to complementary medicine or Chinese medicine,1 but our results suggest that it might be prudent to expand recommendations to studies on drug interventions as well.

    The reporting quality of Chinese CS-RCTs was low,6,30,31 which some may argue is a reason to not use Chinese CS-RCTs in systematic reviews. However, reporting quality may not completely represent the actual scientific quality.32 One study found no difference between one systematic review mainly using English-language trials and another mainly using Chinese-language trials on the same topic.8 It is the researchers’ decision to include or not include those trials (ie, trials published in Chinese and/or indexed in Chinese bibliographic databases) based on reporting quality, but it might be too simplistic to just ignore them.

    Limitations

    Our study has several limitations. First, we searched 7 prominent, but not all, bibliographic databases.33 Second, our search strategy relied on the information in trial registries, which may be inaccurate and/or incomplete.34,35 Third, less than 15% of Chinese-language articles reported registration,6 indicating CS-RCTs in trial registries may not be representative of all CS-RCTs in the period addressed. At this moment, it is unclear how much this study can be generalized to all CS-RCTs. In addition, we studied language bias and indexing bias from the level of the entire RCT community, but we did not assess whether such biases might have effects on the conclusion of individual systematic reviews. Such simulation or empirical studies might further elucidate the extent and direction of systematic error introduced by language and indexing bias.

    Conclusions

    Our study indicates the existence of language bias and indexing bias among CS-RCTs in trial registries. These biases might threaten the validity of evidence synthesis. When synthesizing evidence, drug interventions might appear more favorable than in reality owing to language and indexing bias. Removing language restrictions and actively searching Chinese bibliographic databases may reduce the effect of these 2 biases.

    Back to top
    Article Information

    Accepted for Publication: March 12, 2020.

    Published: May 28, 2020. doi:10.1001/jamanetworkopen.2020.5894

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Jia Y et al. JAMA Network Open.

    Corresponding Author: David D. Celentano, ScD, MHS, Bloomberg School of Public Health, The Johns Hopkins University, 615 N Wolfe St, Room W6041, Baltimore, MD 21205 (dcelent1@jh.edu).

    Author Contributions: Drs Ehrhardt and Celentano contributed equally to this study. Drs Ehrhardt and Celentano had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Jia, Wen, Wang, Rosman, Gagnier, Ehrhardt, Celentano.

    Acquisition, analysis, or interpretation of data: Jia, Huang, Wen, Wang, Chen, Robinson, Gagnier, Celentano.

    Drafting of the manuscript: Jia, Wen, Wang.

    Critical revision of the manuscript for important intellectual content: Jia, Huang, Wen, Wang, Rosman, Chen, Robinson, Gagnier, Ehrhardt, Celentano.

    Statistical analysis: Jia, Wen, Celentano.

    Administrative, technical, or material support: Huang, Wang, Chen, Celentano.

    Supervision: Robinson, Gagnier, Ehrhardt, Celentano.

    Conflict of Interest Disclosures: None reported.

    Funding/Support: This study was supported by the Doctoral Thesis Research Fund at the Department of Epidemiology, Bloomberg School of Public Health, The Johns Hopkins University (Mr Jia).

    Role of the Funder/Sponsor: The sponsor had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: Kevin M. Fain, DrPH, JD, National Library of Medicine of the National Institutes of Health, provided comments that greatly improved the manuscript. Riaz Qureshi, MSc, the Bloomberg School of Public Health of The Johns Hopkins University and Benjamin Rouse, MHS, ECRI Institute, edited and polished the manuscript. They received no compensation for their contributions.

    References
    1.
    Higgins  JPT, Green  S.  Cochrane Handbook for Systematic Reviews of Interventions. Vol 4. John Wiley & Sons; 2011.
    2.
    Egger  M, Zellweger-Zähner  T, Schneider  M, Junker  C, Lengeler  C, Antes  G.  Language bias in randomised controlled trials published in English and German.   Lancet. 1997;350(9074):326-329. doi:10.1016/S0140-6736(97)02419-7 PubMedGoogle ScholarCrossref
    3.
    Page  MJ, Shamseer  L, Altman  DG,  et al.  Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.   PLoS Med. 2016;13(5):e1002028. doi:10.1371/journal.pmed.1002028 PubMedGoogle Scholar
    4.
    Jüni  P, Holenstein  F, Sterne  J, Bartlett  C, Egger  M.  Direction and impact of language bias in meta-analyses of controlled trials: empirical study.   Int J Epidemiol. 2002;31(1):115-123. doi:10.1093/ije/31.1.115 PubMedGoogle ScholarCrossref
    5.
    National Science Board. Science and Engineering indicators 2018: the rise of China in science and engineering. Published 2018. Accessed April 21, 2020. https://www.nsf.gov/nsb/sei/one-pagers/China-2018.pdf
    6.
    Zhao  H, Zhang  J, Guo  L,  et al.  The status of registrations, ethical reviews and informed consent forms in RCTs of high impact factor Chinese medical journals.   Chinese J Evidence-Based Med. 2018;(7):16. doi:10.7507/1672-2531.201802027Google Scholar
    7.
    Elsevier. List of journal titles in Embase. Accessed July 14, 2019. https://www.elsevier.com/solutions/embase-biomedical-research/embase-coverage-and-content
    8.
    Cohen  JF, Korevaar  DA, Wang  J, Spijker  R, Bossuyt  PM.  Should we search Chinese biomedical databases when performing systematic reviews?   Syst Rev. 2015;4(1):23. doi:10.1186/s13643-015-0017-3 PubMedGoogle ScholarCrossref
    9.
    Vandenbroucke  JP, von Elm  E, Altman  DG,  et al; STROBE initiative.  Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration.   Ann Intern Med. 2007;147(8):W163-94. doi:10.7326/0003-4819-147-8-200710160-00010-w1 PubMedGoogle Scholar
    10.
    World Health Organization. International Clinical Trials Registry Platform (ICTRP): joint statement on public disclosure of results from clinical trials. Published 2018. Accessed April 21, 2020. https://www.who.int/ictrp/results/en/
    11.
    Center for Drug Evaluation of China Food and Drug. About the Drug Clinical Trial Registry Platform. Accessed April 21, 2020. http://www.chinadrugtrials.org.cn/eap/11.snipet.block
    12.
    United States Food and Drug Administration. Drugs@FDA: FDA approved drug products. Accessed April 21, 2020. https://www.accessdata.fda.gov/scripts/cder/daf/
    13.
    European Medicine Agency. Search medicines. Accessed April 21, 2020. https://www.ema.europa.eu/en/medicines
    14.
    Hartung  DM, Zarin  DA, Guise  J-M, McDonagh  M, Paynter  R, Helfand  M.  Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications.   Ann Intern Med. 2014;160(7):477-483. doi:10.7326/M13-0480 PubMedGoogle ScholarCrossref
    15.
    Ross  JS, Tse  T, Zarin  DA, Xu  H, Zhou  L, Krumholz  HM.  Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis.   BMJ. 2012;344:d7292. doi:10.1136/bmj.d7292 PubMedGoogle ScholarCrossref
    16.
    Rethlefsen  ML, Farrell  AM, Osterhaus Trzasko  LC, Brigham  TJ.  Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews.   J Clin Epidemiol. 2015;68(6):617-626. doi:10.1016/j.jclinepi.2014.11.025 PubMedGoogle ScholarCrossref
    17.
    Koffel  JB.  Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors.   PLoS One. 2015;10(5):e0125931. doi:10.1371/journal.pone.0125931 PubMedGoogle Scholar
    18.
    Xia  J, Wright  J, Adams  CE.  Five large Chinese biomedical bibliographic databases: accessibility and coverage.   Health Info Libr J. 2008;25(1):55-61. doi:10.1111/j.1471-1842.2007.00734.x PubMedGoogle ScholarCrossref
    19.
    Saldanha  IJ, Scherer  RW, Rodriguez-Barraquer  I, Jampel  HD, Dickersin  K.  Dependability of results in conference abstracts of randomized controlled trials in ophthalmology and author financial conflicts of interest as a factor associated with full publication.   Trials. 2016;17(1):213. doi:10.1186/s13063-016-1343-z PubMedGoogle ScholarCrossref
    20.
    Easterbrook  PJ, Berlin  JA, Gopalan  R, Matthews  DR.  Publication bias in clinical research.   Lancet. 1991;337(8746):867-872. doi:10.1016/0140-6736(91)90201-Y PubMedGoogle ScholarCrossref
    21.
    Decullier  E, Lhéritier  V, Chapuis  F.  Fate of biomedical research protocols and publication bias in France: retrospective cohort study.   BMJ. 2005;331(7507):19. doi:10.1136/bmj.38488.385995.8F PubMedGoogle ScholarCrossref
    22.
    De Angelis  C, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.   Circulation. 2005;111(10):1337-1338. doi:10.1161/01.CIR.0000158823.53941.BE PubMedGoogle ScholarCrossref
    23.
    Li  Y.  Chinese medical doctors negotiating the pressure of the publication requirement.   Ibérica Rev la Asoc Eur Lenguas para Fines Específicos. 2014;(28):107-127.Google Scholar
    24.
    Emerging Technology from the arXiv. The truth about China’s cash-for-publication policy. Published July 12, 2017. Accessed April 21, 2020. https://www.technologyreview.com/s/608266/the-truth-about-chinas-cash-for-publication-policy/
    25.
    Galandi  D, Schwarzer  G, Antes  G.  The demise of the randomised controlled trial: bibliometric study of the German-language health care literature, 1948 to 2004.   BMC Med Res Methodol. 2006;6(1):30. doi:10.1186/1471-2288-6-30 PubMedGoogle ScholarCrossref
    26.
    Morrison  A, Polisena  J, Husereau  D,  et al.  The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies.   Int J Technol Assess Health Care. 2012;28(2):138-144. doi:10.1017/S0266462312000086 PubMedGoogle ScholarCrossref
    27.
    Pham  B, Klassen  TP, Lawson  ML, Moher  D.  Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.   J Clin Epidemiol. 2005;58(8):769-776. doi:10.1016/j.jclinepi.2004.08.021 PubMedGoogle ScholarCrossref
    28.
    Morton  S, Berg  A, Levit  L, Eden  J.  Finding What Works in Health Care: Standards for Systematic Reviews. National Academies Press; 2011.
    29.
    Liu  Q, Tian  L-G, Xiao  S-H,  et al.  Harnessing the wealth of Chinese scientific literature: schistosomiasis research and control in China.   Emerg Themes Epidemiol. 2008;5(1):19. doi:10.1186/1742-7622-5-19 PubMedGoogle ScholarCrossref
    30.
    Mao  L, Zhang  P, Ding  J,  et al.  Quality assessment of randomized controlled trial report in the field of breast cancer published in Chinese.   Chinese J Gen Pract. 2016;19(05):615-620. doi:10.3969/j.issn.1007-9572.2016.05.029 Google Scholar
    31.
    Tian  J, Huang  Z, Ge  L,  et al.  Evaluation of quality of randomized controlled trials research papers among Chinese abstracts for breast cancer.   Chinese J Gen Pract. 2016;14(7):1182-1185. doi:10.16766/j.cnki.issn.1674-4152.2016.07.039 Google Scholar
    32.
    Wu  T, Li  Y, Bian  Z, Liu  G, Moher  D.  Randomized trials published in some Chinese journals: how many are randomized?   Trials. 2009;10(1):46. doi:10.1186/1745-6215-10-46 PubMedGoogle ScholarCrossref
    33.
    Vassar  M, Yerokhin  V, Sinnett  PM,  et al.  Database selection in systematic reviews: an insight through clinical neurology.   Health Info Libr J. 2017;34(2):156-164. doi:10.1111/hir.12176 PubMedGoogle ScholarCrossref
    34.
    Viergever  RF, Karam  G, Reis  A, Ghersi  D.  The quality of registration of clinical trials: still a problem.   PLoS One. 2014;9(1):e84727. doi:10.1371/journal.pone.0084727 PubMedGoogle Scholar
    35.
    Viergever  RF, Ghersi  D.  The quality of registration of clinical trials.   PLoS One. 2011;6(2):e14701. doi:10.1371/journal.pone.0014701 PubMedGoogle Scholar
    ×