Assessment of Language and Indexing Biases Among Chinese-Sponsored Randomized Clinical Trials

This cohort study evaluates the existence of language and indexing biases among Chinese-sponsored randomized clinical trials on drug interventions.


Introduction
In non-English-speaking countries, researchers can choose to publish randomized clinical trials (RCTs) in English-language journals or journals in their native language. It is established that RCTs with positive results (hereinafter referred to as positive RCTs) are more likely to be published in English-language journals, a phenomenon termed language bias. 1 This tendency may lead to disproportionally more positive RCTs in English literature and consequently more RCTs with negative findings (hereinafter referred to as negative RCTs) in non-English literature. 2 Ideally, this bias would not threaten the validity of systematic reviews because reviewers should comprehensively search for all existing evidence regardless of the language 1 ; however, estimates indicate that almost 40% of systematic reviews were reportedly restricted to English-language articles indexed in English bibliographic databases. 3 This estimate raises the concern that such reviews may miss negative RCTs that are only published in the non-English literature, leading to biased evidence. 4 Recently, scientific publications from Mainland China have been surging. 5 Publications of RCTs sponsored by researchers in Mainland China (Chinese-sponsored RCTs [CS-RCTs]) are also split between English-and Chinese-language journals (English CS-RCTs and Chinese CS-RCTs). 6 However, limited evidence is available regarding language bias among CS-RCTs. Whether we should include Chinese CS-RCTs indexed in English bibliographic databases to reduce the effect of speculated language bias is unknown.
A further challenge, and one more difficult to address, is that most Chinese journals have not been indexed in English bibliographic databases owing to large quantity and varying quality. 7 This challenge implies that systematic reviewers must not only remove language restrictions from searching English bibliographic databases but also actively search Chinese bibliographic databases to capture all Chinese CS-RCTs, a practice seldomly adopted by the systematic review community. 8 Bias may exist if Chinese CS-RCTs with positive results are more likely to be indexed in English bibliographic databases than their negative counterparts, which are more commonly seen in Chinese bibliographic databases. We term this potential residual of language bias as indexing bias.
Currently, the Cochrane Handbook for Systematic Reviews of Interventions only recommends searching Chinese bibliographic databases for systematic reviews on Chinese herbal medicine. 1 Whether the recommendation should be extended to drug interventions is unknown. The objective of this study was to evaluate the existence of language and indexing biases among CS-RCTs on drug interventions to inform the potential update of the recommendation.

Methods
In this retrospective cohort study, we retrieved CS-RCTs from trial registries and searched bibliographic databases to determine their publication status. Two hypotheses were predefined: (1) positive CS-RCTs were more likely to be published in English than negative CS-RCTs (language bias), and (2) positive CS-RCTs were more likely to be indexed in English than Chinese bibliographic databases (indexing bias). Because this study was a literature review based on open-source data and did not include research participants, it was not subject to institutional review board approval. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. 9

Identifying CS-RCTs From Trial Registries
We retrieved CS-RCTs from all 17 primary registries recognized by the World Health Organization 10 and the Drug Clinical Trial Registry Platform (DCTRP) sponsored by the China Food and Drug Administration. 11 A substance was considered a drug if recognized and regulated by the US Food and Drug Administration and/or the European Medicines Agency. 12,13 We included all CS-RCTs that started January 1, 2008, and were completed by December 31, 2014, to allow a minimum of 4.5 years from trial completion to publication. 14 We excluded phase 1 trials (including bioequivalence and pharmacokinetics studies) and CS-RCTs missing the study period or RCTs with an unclear study interval (eg, end date before the start date) or any unnamed experimental drug, principal investigator, or sponsor in the registries.

Identification of Journal Articles From Bibliographic Databases
The search and analysis were conducted from March 1 to August 31, 2019. We only included journal articles produced from eligible CS-RCTs. Conference abstracts, research letters, and dissertations were not included. Publications of protocols, subgroup analyses, secondary analyses, and metaanalyses were also excluded.
Based on previous studies, 14 information, and Wanfang Data), were subsequentially searched. 18 We conducted a 4-step process to identify matches of eligible CS-RCTs. First, we searched bibliographic databases to retrieve citations; second, we screened the citations for eligible CS-RCTs; third, we downloaded PDFs of possibly eligible trials; and fourth, we matched the PDFs with the registry records of eligible CS-RCTs. The criteria for screening and matching are shown in eTable 3 in the Supplement.
The journal articles were classified as confirmed matches and probable matches according to the similarity between journal articles and registry records. Confirmed matches indicated the journal articles were consistent with the registry records, whereas probable matches indicated the journal articles were similar to the registry records but differed on or lacked only 1 data item. The primary analysis was conducted among the confirmed and probable matches, whereas a sensitivity analysis was conducted among the confirmed matches only.

Exposure
The exposure was the finding of individual CS-RCTs (positive vs negative) according to the CS-RCT's primary outcome reported in the journal article. 19 If multiple primary outcomes were reported in a CS-RCT, we selected the first one reported in the result section. If no primary outcome was defined, the selection of the CS-RCT's primary outcome was based on the following hierarchical order: the first outcome used in the sample size calculation, the first outcome defined in the study objective, or the first outcome reported in the Results section. When the time point was not specified for the CS-RCT's primary outcome that was measured at multiple time points, we considered the last point in our main analysis and the first point in sensitivity analyses.
We defined a positive result as favoring the experimental group with statistical significance in superiority trials or showing no difference between treatment groups for equivalence or noninferiority trials. Results that were not statistically significant, significantly favored the control group, or failed to show equivalence or noninferiority were defined as negative.

Outcome
Two main outcomes were defined: the language of the publication (English vs Chinese) and the language of the bibliographic database where the publication was indexed (English vs Chinese). An article published in both Chinese and English was considered published in English; similarly, an article indexed in both English and Chinese bibliographic databases was considered indexed in an English bibliographic database. We assumed all English-language articles were indexed in English bibliographic databases, but Chinese-language articles were possibly indexed in English or Chinese bibliographic databases.

Measurements of Associations
Bias was estimated by relative risk (RR), including point estimates and 95% CIs. An RR larger than 1.00 indicated that positive CS-RCTs were more likely than negative CS-RCTs to be published in English or indexed in English bibliographic databases. The RRs were estimated using log binomial models with 5 covariates: sample size (<100 vs Ն100), funding source (industry vs nonindustry), study design (superiority vs noninferiority or equivalence), number of recruitment centers (single vs multiple), and registration type (prospective vs retrospective). 20,21 Industrial funding was considered as long as 1 funder was from industry; prospective registration was considered when registration occurred before the first participant was recruited. 22 We included an interaction term in the models to evaluate the heterogeneity of bias across registries. Statistical significance was defined as 2-sided P < .05 for the main effect and P < .10 for interaction, generated by Wald χ 2 tests as in the log binomial models. SAS, version 9.4 (SAS Institute Inc) was used for data cleaning and analysis.

Results
The search through trial registries and bibliographic databases was conducted from March 1 to

Language Bias
Four hundred sixty-eight CS-RCTs were included for this analysis. As shown in Table 2

Indexing Bias
Four hundred sixty-eight CS-RCTs were included for this analysis. As shown in Table 2 (Table 3).
The interaction between registry and language of bibliographic databases was not statistically significant (Wald χ 2 = 0.22; P = .64), indicating no evidence of heterogeneity of indexing bias across registries. The only other factor associated with an increased likelihood of being indexed in English bibliographic databases among CS-RCTs was sample size of at least 100 (RR, 2.04; 95% CI, 1.11-3.72).

Sensitivity Analyses
Two sensitivity analyses were conducted. When only confirmed matches (n = 432) were analyzed,

Discussion
Our study supports the existence of language bias and indexing bias among CS-RCTs included in trial registries. As hypothesized, positive CS-RCTs were more likely to be published in English or indexed in English bibliographic databases compared with negative CS-RCTs.

Language Bias
Reputation, job prospects, and academic progress may critically depend on publishing in Englishlanguage journals among Chinese researchers. 23,24 Positive CS-RCTs are more likely to be submitted to English-language journals because they typically have a higher chance of being accepted; accordingly, English-language journals contain more positive CS-RCTs than their Chinese counterparts.
Theoretically, language bias disappears if all clinical trials shift to be published in English. This ideal has been echoed by a trend toward publishing in English in some countries, such as Germany. 25 With the mean number of RCTs per German-language journal decreasing from a maximum of 11.2 annually from 1970 to 1986 to only 1.7 annually from 2002 to 2004, language bias from Germanspeaking countries may no longer be a concern.
We did not detect such a trend among CS-RCTs. According to an ongoing study, the number of  Several studies attempted to evaluate the effect of language bias based on non-Englishlanguage trials included in systematic reviews. 26,27 Because most systematic reviews were constrained to work within English bibliographic databases only, what those studies measured was a fraction of language bias-the difference between English-language and non-English-language trials indexed in English bibliographic databases. The effect of language bias cannot be comprehensively evaluated unless non-English-language trials, especially the ones not indexed in English bibliographic databases, are included and evaluated.

Indexing Bias
As the primary source for systematic reviewers, English bibliographic databases may index some non-English-language literature, but they vary in the amount and scope. it appears plausible that, owing to language and indexing bias, drug interventions might appear more positive than they are when existing evidence is synthesized, for example in systematic reviews.

How to Eliminate the Effect of Language Bias Regarding CS-RCTs
The effect of language bias regarding CS-RCTs might be eliminated if reviews comprehensively searched Chinese bibliographic databases, if major English bibliographic databases would index all Chinese-language literature, or if all CS-RCTs would be appropriately registered with results. There are, however, layers of complexity that warrant appreciation. There has been a discussion as to whether scientists should search Chinese bibliographic databases when conducting systematic reviews. 8,29 Our study tipped the scales in this proposition's direction: including Chinese-language literature may reduce bias and shrink the CIs of the estimates. Currently, the Cochrane Handbook for Systematic Reviews of Interventions only recommends searching Chinese bibliographic databases for topics related to complementary medicine or Chinese medicine, 1 but our results suggest that it might be prudent to expand recommendations to studies on drug interventions as well.
The reporting quality of Chinese CS-RCTs was low, 6,30,31 which some may argue is a reason to not use Chinese CS-RCTs in systematic reviews. However, reporting quality may not completely represent the actual scientific quality. 32 One study found no difference between one systematic review mainly using English-language trials and another mainly using Chinese-language trials on the same topic. 8 It is the researchers' decision to include or not include those trials (ie, trials published in Chinese and/or indexed in Chinese bibliographic databases) based on reporting quality, but it might be too simplistic to just ignore them.

Limitations
Our study has several limitations. First, we searched 7 prominent, but not all, bibliographic databases. 33 Second, our search strategy relied on the information in trial registries, which may be inaccurate and/or incomplete. 34,35 Third, less than 15% of Chinese-language articles reported registration, 6 indicating CS-RCTs in trial registries may not be representative of all CS-RCTs in the period addressed. At this moment, it is unclear how much this study can be generalized to all CS-RCTs. In addition, we studied language bias and indexing bias from the level of the entire RCT community, but we did not assess whether such biases might have effects on the conclusion of individual systematic reviews. Such simulation or empirical studies might further elucidate the extent and direction of systematic error introduced by language and indexing bias.