[Skip to Content]
[Skip to Content Landing]
Figure 1.
Flowchart of Article Selection
Flowchart of Article Selection

RCTs indicates randomized clinical trials; SDs, standard deviations.

Figure 2.
Subgroup Analyses by Various Study Characteristics
Subgroup Analyses by Various Study Characteristics

ITT indicates intent to treat; mITT, modified intent to treat; and ROR, ratio of odds ratios.

aDifferent outcomes could be observed within the same trial.

Table 1.  
Characteristics of Included Randomized Clinical Trials With and Without Primary Outcome Change With Available Effect Sizea
Characteristics of Included Randomized Clinical Trials With and Without Primary Outcome Change With Available Effect Sizea
Table 2.  
Main and Sensitivity Analyses With ROR Between Randomized Clinical Trials With and Without Primary Outcome Changea
Main and Sensitivity Analyses With ROR Between Randomized Clinical Trials With and Without Primary Outcome Changea
1.
Kane  RL, Wang  J, Garrard  J.  Reporting in randomized clinical trials improved after adoption of the CONSORT Statement.  J Clin Epidemiol. 2007;60(3):241-249. doi:10.1016/j.jclinepi.2006.06.016PubMedGoogle ScholarCrossref
2.
De Angelis  C, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.  Ann Intern Med. 2004;141(6):477-478. doi:10.7326/0003-4819-141-6-200409210-00109PubMedGoogle ScholarCrossref
3.
Zarin  DA, Ide  NC, Tse  T, Harlan  WR, West  JC, Lindberg  DA.  Issues in the registration of clinical trials.  JAMA. 2007;297(19):2112-2120. doi:10.1001/jama.297.19.2112PubMedGoogle ScholarCrossref
4.
Kahan  BC, Jairath  V.  Outcome pre-specification requires sufficient detail to guard against outcome switching in clinical trials: a case study.  Trials. 2018;19(1):265. doi:10.1186/s13063-018-2654-zPubMedGoogle ScholarCrossref
5.
Mathieu  S, Boutron  I, Moher  D, Altman  DG, Ravaud  P.  Comparison of registered and published primary outcomes in randomized controlled trials.  JAMA. 2009;302(9):977-984. doi:10.1001/jama.2009.1242PubMedGoogle ScholarCrossref
6.
Chan  AW, Hróbjartsson  A, Haahr  MT, Gøtzsche  PC, Altman  DG.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.  JAMA. 2004;291(20):2457-2465. doi:10.1001/jama.291.20.2457PubMedGoogle ScholarCrossref
7.
Jones  CW, Keil  LG, Holland  WC, Caughey  MC, Platts-Mills  TF.  Comparison of registered and published outcomes in randomized controlled trials: a systematic review.  BMC Med. 2015;13:282. doi:10.1186/s12916-015-0520-3PubMedGoogle ScholarCrossref
8.
Ioannidis  JP, Caplan  AL, Dal-Ré  R.  Outcome reporting bias in clinical trials: why monitoring matters.  BMJ. 2017;356:j408. doi:10.1136/bmj.j408PubMedGoogle ScholarCrossref
9.
Ramagopalan  S, Skingsley  AP, Handunnetthi  L,  et al.  Prevalence of primary outcome changes in clinical trials registered on ClinicalTrials.gov: a cross-sectional study.  F1000Res. 2014;3:77. doi:10.12688/f1000research.3784.1PubMedGoogle ScholarCrossref
10.
PROSPERO. Influence of primary outcome change on treatment effect estimates in clinical trials: cross-sectional study of registered clinical trials. CRD42017058054. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=58054. Accessed June 7, 2019.
11.
Abraha  I, Cherubini  A, Cozzolino  F,  et al.  Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study.  BMJ. 2015;350:h2445. doi:10.1136/bmj.h2445PubMedGoogle ScholarCrossref
12.
Bafeta  A, Dechartres  A, Trinquart  L, Yavchitz  A, Boutron  I, Ravaud  P.  Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study.  BMJ. 2012;344:e813. doi:10.1136/bmj.e813PubMedGoogle ScholarCrossref
13.
Als-Nielsen  B, Chen  W, Gluud  C, Kjaergard  LL.  Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events?  JAMA. 2003;290(7):921-928. doi:10.1001/jama.290.7.921PubMedGoogle ScholarCrossref
14.
Kivimäki  M, Jokela  M, Nyberg  ST,  et al; IPD-Work Consortium.  Long working hours and risk of coronary heart disease and stroke: a systematic review and meta-analysis of published and unpublished data for 603,838 individuals.  Lancet. 2015;386(10005):1739-1746. doi:10.1016/S0140-6736(15)60295-1PubMedGoogle ScholarCrossref
15.
Fang  X, Han  D, Cheng  Q,  et al.  Association of levels of physical activity with risk of Parkinson disease: a systematic review and meta-analysis.  JAMA Netw Open. 2018;1(5):e182421. doi:10.1001/jamanetworkopen.2018.2421PubMedGoogle ScholarCrossref
16.
Chinn  S.  A simple method for converting an odds ratio to effect size for use in meta-analysis.  Stat Med. 2000;19(22):3127-3131. doi:10.1002/1097-0258(20001130)19:22<3127::AID-SIM784>3.0.CO;2-MPubMedGoogle ScholarCrossref
17.
Killeen  S, Sourallous  P, Hunter  IA, Hartley  JE, Grady  HL.  Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals.  Ann Surg. 2014;259(1):193-196. doi:10.1097/SLA.0b013e318299d00bPubMedGoogle ScholarCrossref
18.
Li  XQ, Yang  GL, Tao  KM, Zhang  HQ, Zhou  QH, Ling  CQ.  Comparison of registered and published primary outcomes in randomized controlled trials of gastroenterology and hepatology.  Scand J Gastroenterol. 2013;48(12):1474-1483. doi:10.3109/00365521.2013.845909PubMedGoogle ScholarCrossref
19.
Rongen  JJ, Hannink  G.  Comparison of registered and published primary outcomes in randomized controlled trials of orthopaedic surgical interventions.  J Bone Joint Surg Am. 2016;98(5):403-409. doi:10.2106/JBJS.15.00400PubMedGoogle ScholarCrossref
20.
Anand  V, Scales  DC, Parshuram  CS, Kavanagh  BP.  Registration and design alterations of clinical trials in critical care: a cross-sectional observational study.  Intensive Care Med. 2014;40(5):700-722. doi:10.1007/s00134-014-3250-7PubMedGoogle ScholarCrossref
21.
Smith  HN, Bhandari  M, Mahomed  NN, Jan  M, Gandhi  R.  Comparison of arthroplasty trial publications after registration in ClinicalTrials.gov.  J Arthroplasty. 2012;27(7):1283-1288. doi:10.1016/j.arth.2011.11.005PubMedGoogle ScholarCrossref
22.
Rosenthal  R, Dwan  K.  Comparison of randomized controlled trial registry entries and content of reports in surgery journals.  Ann Surg. 2013;257(6):1007-1015. doi:10.1097/SLA.0b013e318283cf7fPubMedGoogle ScholarCrossref
23.
Fleming  PS, Koletsi  D, Dwan  K, Pandis  N.  Outcome discrepancies and selective reporting: impacting the leading journals?  PLoS One. 2015;10(5):e0127495. doi:10.1371/journal.pone.0127495PubMedGoogle ScholarCrossref
24.
Milette  K, Roseman  M, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: a systematic review.  J Psychosom Res. 2011;70(3):205-217. doi:10.1016/j.jpsychores.2010.09.015PubMedGoogle ScholarCrossref
25.
Jones  CW, Platts-Mills  TF.  Quality of registration for clinical trials published in emergency medicine journals.  Ann Emerg Med. 2012;60(4):458-64.e1.PubMedGoogle ScholarCrossref
26.
Ross  JS, Mulvey  GK, Hines  EM, Nissen  SE, Krumholz  HM.  Trial publication after registration in ClinicalTrials.gov: a cross-sectional analysis.  PLoS Med. 2009;6(9):e1000144. doi:10.1371/journal.pmed.1000144PubMedGoogle ScholarCrossref
27.
Jones  CW, Misemer  BS, Platts-Mills  TF,  et al.  Primary outcome switching among drug trials with and without principal investigator financial ties to industry: a cross-sectional study.  BMJ Open. 2018;8(2):e019831. doi:10.1136/bmjopen-2017-019831PubMedGoogle ScholarCrossref
28.
Schulz  KF, Altman  DG, Moher  D; CONSORT Group.  CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.  BMJ. 2010;340:c332. doi:10.1136/bmj.c332PubMedGoogle ScholarCrossref
29.
Gandhi  R, Jan  M, Smith  HN, Mahomed  NN, Bhandari  M.  Comparison of published orthopaedic trauma trials following registration in Clinicaltrials.gov.  BMC Musculoskelet Disord. 2011;12:278.PubMedGoogle ScholarCrossref
30.
Nankervis  H, Baibergenova  A, Williams  HC, Thomas  KS.  Prospective registration and outcome-reporting bias in randomized controlled trials of eczema treatments: a systematic review.  J Invest Dermatol. 2012;132(12):2727-2734. doi:10.1038/jid.2012.231PubMedGoogle ScholarCrossref
31.
Ewart  R, Lausen  H, Millian  N.  Undisclosed changes in outcomes in randomized controlled trials: an observational study.  Ann Fam Med. 2009;7(6):542-546. doi:10.1370/afm.1017PubMedGoogle ScholarCrossref
32.
You  B, Gan  HK, Pond  G, Chen  EX.  Consistency in the analysis and reporting of primary end points in oncology randomized controlled trials from registration to publication: a systematic review.  J Clin Oncol. 2012;30(2):210-216. doi:10.1200/JCO.2011.37.0890PubMedGoogle ScholarCrossref
33.
Vera-Badillo  FE, Shapiro  R, Ocana  A, Amir  E, Tannock  IF.  Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer.  Ann Oncol. 2013;24(5):1238-1244. doi:10.1093/annonc/mds636PubMedGoogle ScholarCrossref
34.
Saquib  N, Saquib  J, Ioannidis  JP.  Practices and impact of primary outcome adjustment in randomized controlled trials: meta-epidemiologic study.  BMJ. 2013;347:f4313. doi:10.1136/bmj.f4313PubMedGoogle ScholarCrossref
35.
Wildt  S, Krag  A, Gluud  L.  Characteristics of randomised trials on diseases in the digestive system registered in ClinicalTrials.gov: a retrospective analysis.  BMJ Open. 2011;1(2):e000309. doi:10.1136/bmjopen-2011-000309PubMedGoogle ScholarCrossref
36.
Liu  JP, Han  M, Li  XX,  et al.  Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study.  BMJ Open. 2013;3(7):e002968. doi:10.1136/bmjopen-2013-002968PubMedGoogle ScholarCrossref
37.
Nüesch  E, Trelle  S, Reichenbach  S,  et al.  The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study.  BMJ. 2009;339:b3244. doi:10.1136/bmj.b3244PubMedGoogle ScholarCrossref
38.
Nüesch  E, Trelle  S, Reichenbach  S,  et al.  Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study.  BMJ. 2010;341:c3515. doi:10.1136/bmj.c3515PubMedGoogle ScholarCrossref
39.
Dechartres  A, Trinquart  L, Boutron  I, Ravaud  P.  Influence of trial sample size on treatment effect estimates: meta-epidemiological study.  BMJ. 2013;346:f2304. doi:10.1136/bmj.f2304PubMedGoogle ScholarCrossref
40.
Pildal  J, Hróbjartsson  A, Jørgensen  KJ, Hilden  J, Altman  DG, Gøtzsche  PC.  Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials.  Int J Epidemiol. 2007;36(4):847-857. doi:10.1093/ije/dym087PubMedGoogle ScholarCrossref
41.
Sterne  JA, Jüni  P, Schulz  KF, Altman  DG, Bartlett  C, Egger  M.  Statistical methods for assessing the influence of study characteristics on treatment effects in “meta-epidemiological” research.  Stat Med. 2002;21(11):1513-1524. doi:10.1002/sim.1184PubMedGoogle ScholarCrossref
42.
Huwiler-Müntener  K, Jüni  P, Junker  C, Egger  M.  Quality of reporting of randomized trials as a measure of methodologic quality.  JAMA. 2002;287(21):2801-2804. doi:10.1001/jama.287.21.2801PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 2,157
    Original Investigation
    Medical Journals and Publishing
    July 19, 2019

    Comparison of Clinical Trial Changes in Primary Outcome and Reported Intervention Effect Size Between Trial Registration and Publication

    Author Affiliations
    • 1Department of Epidemiology and Health Statistics, School of Public Health, Xi’an Jiaotong University Health Science Centre, Xi’an, China
    • 2Tropical Clinical Trials Unit, Department of Clinical Sciences, Liverpool School of Tropical Medicine, Liverpool, United Kingdom
    • 3Department of Health Education, Jiangsu Province Hospital on Integration of Chinese and Western Medicine, Nanjing, China
    • 4Medical Research and Biometrics Centre, Fuwai Hospital, National Centre for Cardiovascular Disease, Peking Union Medical College and Chinese Academy of Medical Sciences, Mentougou District, Beijing, China
    • 5Arthritis Research UK Primary Care Centre, Research Institute for Primary Care & Health Sciences, Keele University, Keele, United Kingdom
    • 6Imperial Clinical Trials Unit, School of Public Health, Imperial College London, London, United Kingdom
    JAMA Netw Open. 2019;2(7):e197242. doi:10.1001/jamanetworkopen.2019.7242
    Key Points español 中文 (chinese)

    Question  Does primary outcome change between trial registration and publication alter a randomized clinical trial’s reported intervention effect size?

    Findings  In this cross-sectional study that included 389 trials, 130 of them had at least 1 primary outcome change between registration and publication. This significantly overestimated the reported intervention effect size by 16% compared with those without primary outcome change.

    Meaning  Inconsistencies between registered and published primary outcomes in clinical trials are common, with a larger reported intervention effect size among those with primary outcome change than those without.

    Abstract

    Importance  Primary outcome change could threaten the validity of a clinical trial; however, evidence about the consequences on the reported intervention effect size is unclear.

    Objectives  To examine the status of randomized clinical trials whose primary outcome changed between trial registration and publication and to quantify the association of this change with the reported intervention effect size.

    Design, Setting, and Participants  In this cross-sectional study on the primary report of randomized clinical trials with clear prospectively registered primary outcomes, PubMed and Embase were searched for articles published between January 1, 2011, and December 31, 2015. The search was conducted in January 2016, identifying randomized clinical trials and the combination of keywords and text words related to registry.

    Main Outcomes and Measures  Based on the developed approach, trials were classified as having primary outcome change when there was a major discrepancy between the registered and published primary outcomes. Intervention effect was estimated or recalculated using the odds ratio (OR) for each comparison. Each component OR is structured so that an OR is less than 1 if the intervention group has a more favorable result than the control group. The ratio of ORs (ROR), which is the summary OR for trials with primary outcome change divided by those without, and its 95% CI were calculated, with a value less than 1 indicating a larger reported intervention effect size in trials with primary outcome change than those without.

    Results  Among 29 749 searched articles (28 810 MEDLINE and 939 Embase), 1488 articles were randomly selected for review. Of 389 trials with clear primary outcomes prospectively described in the registry (416 outcomes reported), 33.4% (130 of 389) of trials had at least 1 primary outcome change. Most (66 of 130) of the changes were either not reporting or omitting the primary outcome. In total, 338 trials (365 outcomes and 487 comparisons) were available for quantitative analysis on the reported intervention effect size bias assessment. Compared with those without primary outcome change, trials with primary outcome change showed a 16% (pooled ROR, 0.84; 95% CI, 0.73-0.96) larger reported intervention effect size. The result persisted after adjustment for potential confounders (ROR, 0.81; 95% CI, 0.71-0.93) and other sensitivity and subgroup analyses.

    Conclusions and Relevance  Results of this study suggest that inconsistencies between registered and published primary outcomes of clinical trials are common, and trials with primary outcome change are likely to have a larger intervention effect than those without.

    Introduction

    Randomized clinical trials (RCTs) have a crucial role in assessing the efficacy and safety of a treatment and in advancing medical knowledge.1 Clinical trials of investigational medicinal products have been legally required to be registered before participant enrollment since January 5, 2004; furthermore, to improve transparency of results, the International Committee of Medical Journal Editors member journals have required since January 7, 2005, that for publication clinical trials of any intervention should be preregistered.2 Registering a trial is mostly free, and options include ClinicalTrials.gov, EU Clinical Trials Register, and International Standard Randomized Controlled Trial Number (ISRCTN) register. Although the reported information differs between registries, a clearly defined and prespecified primary outcome is an important element.3,4 Discrepancies between registered and published outcomes may imply selective outcome reporting based on significant P values.5,6 This practice could threaten the validity of clinical trials by producing conclusions that may mislead physicians and policy makers.6-8

    Although it is well recognized that registries are important tools to reduce the risk of selective reporting of outcomes, Jones et al7 found that consistency between planned and published outcomes varied substantially among 27 eligible studies, with a median consistency proportion of 31% (interquartile range, 17%-45%). Similarly, another study9 analyzing all interventional trials registered on ClinicalTrials.gov from 1999 to 2012 showed that 32% of trials had their primary outcomes altered between the listed study start and completion dates. However, previous studies in this area have focused on specific design characteristics (eg, pain or continuous outcomes) and have not attempted to quantify to what extent a primary outcome change will alter the invention size that is being estimated.

    The objectives of this study were 2-fold. The first objective was to estimate the proportion of RCTs that had a primary outcome change, without restriction by journals, diseases, or registry entries. A second objective was to quantify the consequences associated with primary outcome change on the reported intervention effect size.

    Methods

    This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies. The study was not submitted for institutional review board approval because all data are publicly available.

    Search Strategy

    In this cross-sectional study, we studied the primary report of registered RCTs published between January 1, 2011, and December 31, 2015. A search was conducted in January 2016 using the suggested filters from the Cochrane Collaboration (MEDLINE via PubMed) and the BMJ Evidence Centre (Embase via Ovid) for identifying RCTs and the combination of keywords and text words related to registry. No restrictions by journal, disease, or outcome type (ie, continuous or dichotomous outcome) were applied except that reports had to be in English.

    We then randomly selected 5% of the retrieved records from each year. Details of the search strategy and sampling method are listed in eTables 1, 2, and 3 in the Supplement, and our study protocol was registered on PROSPERO.10

    Study Selection and Eligibility

    Articles were screened for relevance by title and abstract and then by the full text to identify primary trial reports. During this process, one of us (T.C.) excluded duplicate publications, protocol studies and analysis plans, system reviews or meta-analyses, and feasibility, pilot, or phase 1 studies, as well as ancillary studies (eg, subgroup analyses, exploratory analyses, secondary outcome analyses, preliminary results, interim analyses, post hoc analyses, pooled analyses, cost-effectiveness analyses, and mechanism research). Next, for each published trial report, we identified the registration number via published articles or clinical trial registries (ClinicalTrials.gov, ISRCTN register, or country-specific registries). We only included trials that were registered before study completion and gave a clear description of the primary outcome in the registry. Two of us (T.C. and R.Q.) checked the full-text articles in the next 2 processes.

    Data Extraction and Risk of Bias (Quality) Assessment

    We assigned a unique identification number to each trial included in this study. Data were extracted using a standardized extraction form. For published articles, we extracted the publication information (eg, author name and year of publication) and study characteristics, including study design (eg, noninferiority, superiority, or equivalence), sample size for each group, randomization method (eg, cluster or individual), and type of outcome (eg, time to event, binary, or continuous). For registered information, we extracted the following information: start and end dates of participant enrollment, registration date, the last amendment date, originally registered primary outcomes, and amended outcomes (if applicable).

    The risk of bias (RoB) for each trial was assessed by the RoB tool as recommended by the Cochrane Collaboration. Overall RoB was assessed as low risk (low for all Cochrane Collaboration components), high risk (high for ≥1 Cochrane Collaboration component), or unclear risk (unclear for ≥1 Cochrane Collaboration component). Other RoB items included intent-to-treat (ITT) status, trial centers, and source of funding. We defined and classified these items according to published references as follows: deviation from ITT principle (ie, ITT, modified ITT, or no ITT/unknown),11 trial centers (ie, multiple centers or single center),12 and source of funding (ie, public funding, cofinanced, for-profit funding, not funded, or not reported) (with not-for-profit funding and not funded considered high risk).13

    Outcome of Interest

    A major discrepancy was defined if the registered and published primary outcomes were different or were assessed at a different time point. This definition is according to a modified classification by Chan et al.6 Specifically, the following were considered major discrepancies: (1) a prespecified primary outcome in the trial registration protocol was subsequently reported as a secondary outcome in the final published article; (2) The published primary outcome was described as a secondary outcome in the registry; (3) The prespecified primary outcomes in the trial registration were either omitted or not reported or labeled from the published article; (4) a new primary outcome was introduced in the published article (eg, an outcome that does not appear at all in the registry but is introduced as primary in the article, or one of the components changed among a composite outcome); and (5) the timing of assessment of the primary outcome in the registered protocol and published article differed.

    Inconsistencies were independently identified by 2 of us (T.C. and C.L.), and disagreements were resolved by discussion until consensus (κ coefficient, 0.92; 95% CI, 0.88-0.96). If changes to the registered primary outcome were made, they were further reviewed by another one of us (V.C.), and disagreements were resolved by consensus.

    Statistical Analysis

    Categorical variables were described by frequencies and percentages and quantitative variables by medians and ranges. We used the κ coefficient to determine the degree of agreement between reviewers.

    To quantify overall the consequences associated with primary outcome change on the reported intervention effect size across different types of outcomes, we assumed relative risk, hazard ratio, and odds ratio (OR) to be the same measure. This strategy has been used in published meta-analyses of observational studies.14,15 In our study, we considered ORs to be a common estimate, but heterogeneity by different types of outcomes was explored in subgroup analyses. For continuous outcomes, we converted them to ORs according to the method by Hasselblad and Hedges by multiplying the standardized mean differences and their SEs by 1.81 to calculate the log ORs and the corresponding SEs.16

    For each comparison, we estimated the OR between the intervention group and the control group. Where necessary, we inverted the effect size so that each comparison was indicated by an OR less than 1 if the intervention group had a more favorable result than the control group.

    Because an enrolled trial may contribute 2 or more comparisons due to multiple groups and/or outcomes, we used a linear mixed model with the log ORs of each comparison as the dependent variable, primary outcome change as a fixed effect, and study identification number as a random effect after weighting the inverse variation of the log OR of each comparison. Differences are presented by estimating the ratio of ORs (ROR) after anti–log transformation. The ROR is the summary OR for trials with primary outcome change divided by those without, with a value less than 1 indicating a larger reported intervention effect size in trials with primary outcome change than those without. To test the robustness of our study, we conducted 4 sensitivity analyses. First was a mixed model with adjustments for trial characteristics (ie, deviation from ITT principle, study design, trial centers, type of comparator, randomization method, type of outcome, source of funding, and RoB). Second was a mixed model using primary outcome change based on reviewer 1 assessment only or reviewer 2 assessment only. Third, we repeated the mixed model but excluded trials in turn with (1) different levels of RoB (low risk, high risk, or unclear risk), (2) different type of outcomes (time to event, binary, or continuous), and (3) multiple outcomes and/or multiple groups. Fourth was a mixed model with the inverse variation of the log OR as additional covariates rather than weights.

    We also carried out subgroup analyses according to prespecified characteristics. These included deviation from ITT principle, study design, trial centers, randomization method, type of outcome, source of funding, and overall RoB.

    All data analyses were performed using statistical software (SAS, version 9.4; SAS Institute Inc). P values were 2 tailed, and P < .05 was considered statistically significant.

    Results

    Among 29 749 searched articles (28 810 MEDLINE and 939 Embase), 1488 articles were randomly selected for review. After excluding 864 reports by reviewing the titles and abstracts, we identified 624 potentially eligible publications. Of them, 65 were further excluded after reading through the full text. After comparing the remaining 559 publications with online registry, we found that 4 trials (0.7%) were not registered, 92 trials (16.5%) trials were registered after completion of the study, and 74 trials (13.2%) were registered with no description or unclear description of the primary outcome. Figure 1 shows the screening process for both published RCTs and registry records.

    Prevalence of Primary Outcome Change and Its Type

    Of 389 trials with clear primary outcomes prospectively described in the registry (416 outcomes reported), 33.4% (130 of 389) of trials had at least 1 primary outcome change. Among those studies with primary outcome change, we found that the most common discrepancy was either omission or not reporting or labeling a registered primary outcome (66 of 130). This was followed by publication of a new outcome (40 of 130), which included 9 composite outcomes with component changes; different timing of assessment in the article and the registry (17 of 130); a registered primary outcome reported as a secondary outcome in the article (6 of 130); and the published primary outcome registered as a secondary outcome (1 of 130). The detailed classification for those trials with primary outcome change and the type of change are listed in eTable 4 in the Supplement.

    Association of Primary Outcome Change With Intervention Effect

    To quantify the consequences of the change in primary outcome on the reported intervention effect size, we calculated the OR for each trial. Of the 389 trials, 22 did not report a primary outcome in the publication, and 29 did not have a reproducible way to calculate an OR. As a result, we included 338 trials (365 outcomes and 487 comparisons) for quantitative analysis on the reported intervention effect size bias assessment. The characteristics of trials with and without primary outcome change are listed in Table 1.

    On average, the reported intervention effect size in trials with primary outcome change was found to be larger by 16% (pooled ROR, 0.84; 95% CI, 0.73-0.96) compared with those without change. This result persisted after adjustment for potential confounders (ROR, 0.81; 95% CI, 0.71-0.93) and using the classification of the primary outcome change from reviewer 1 (unadjusted ROR, 0.83; 95% CI, 0.73-0.95) and reviewer 2 (unadjusted ROR, 0.85; 95% CI, 0.75-0.97). Similarly, we did not find material changes in other sensitivity analyses, with RORs ranging from 0.73 after regarding the study weight as additional covariates rather than the weights in the mixed model to 0.96 after excluding studies with multiple groups (Table 2).

    Subgroup Analyses

    For multicenter trials, the ROR between changed and unchanged primary outcomes was 0.83 (95% CI, 0.72-0.96). For studies assessing continuous outcomes, the corresponding result was 0.74 (95% CI, 0.57-0.94). For trials using a superiority study design, the ROR between changed and unchanged primary outcomes was 0.82 (95% CI, 0.71-0.94). Overestimation of the reported intervention effect size among trials with primary outcome change could be observed among other subgroups, although they did not all reach statistical significance. In addition, there was no evidence of an interaction between different trial characteristics (eg, study design, multiple centers or single center, and type of outcome) and the estimated intervention effects (Figure 2).

    Discussion

    Our cross-sectional study was a survey of contemporary trials that included a broad range of medical conditions and interventions. We found that 33.4% of the sample had at least 1 primary outcome inconsistency between registration and publication. Among studies for which we could calculate an intervention effect, we demonstrated that trials with primary outcome change reported larger intervention effect sizes. This finding remained even after adjustment for RoB items and other potential bias (eg, deviation from ITT principle, multiple centers, and source of funding) and a series of sensitivity analyses (eg, exclusion of studies with binary outcomes). Because we were unable to include trials that either did not declare a primary outcome in the registry or did not register their protocol, this study may underestimate the true consequences of the practice of primary outcome change.

    Several studies have assessed the discrepancy rates between registered and published clinical trial outcomes among specific clinical areas (eg, pain),17-22 journals (eg, general medical journals and high-impact journals),5,23-25 or registry entries (eg, ClinicalTrials.gov).5,9,21,26 A systematic review of studies up to 2014 that compared registered with reported primary outcomes demonstrated a median 31% rate of discrepancies.7 We found a similar rate in our study (33.4%), although it was lower than the 60% in recent study27 among 192 trials. These findings highlight this prevalent issue after publication of the Consolidated Standards of Reporting Trials (CONSORT) 2010 Statement.28 Notably, our analysis found that the 2 most common discrepancies were omission of registered primary outcomes and inclusion of new unregistered outcomes, which have been repeatedly reported in other studies.5,21,22,24,29-31 Unlike other studies that compared published outcomes with those in the registry at the time of the search18,22,25,32,33 or used a specific function (eg, “History of Changes” in the ClinicalTrials.gov archive site),9 our study attempted to identify all outcome changes after the initial registration and excluded those registered after the end of the trial. This is because comparison with the original registered outcome is more relevant in understanding the true consequences of outcome switching on the validity of a clinical trial. However, we still observed high rates of trial registration after study completion (92 of 559) and no primary outcome or unclear primary outcome in the registry (74 of 559) in this survey of contemporary trials after the CONSORT 2010 Statement. Such poor quality of trial registration has been highlighted in previous studies.24,27,32,34-36 Although due to our study design we were unable to address the reasons for primary outcome change, some possibilities need to be assessed in future studies, such as pressure to publish positive results with public funding or high rates of nonpublication among industry-sponsored trials with primary outcome change.

    Our study is the first to date to quantify the consequences of primary outcome change on the reported intervention effect size in individual RCTs. Some specific characteristics of a trial, such as deviation from ITT principle,11,37 small sample size,38,39 concealment of allocation,40,41 and single center,12 have been assessed in several meta-epidemiological studies. In general, various components of inadequate trial methods are associated with imprecision in the estimated intervention effects, but the magnitude and direction of the bias may vary depending on the medical conditions examined, the definition of inadequate methods, and analytic methods.11,12,37-41 In our study, we found that trials with discrepancies between registry and publication show more beneficial treatments than those without. Our results were robust to a series of supplementary adjusted analyses to adjust for potential confounding factors that may contribute to statistical precision in RCTs, as well as to sensitivity analyses to account for the classification of primary outcome change.

    Our study is based on a large sample of individual RCTs rather than a meta-epidemiological approach. The study was performed across a range of medical disciplines, registries, and types of outcomes. Therefore, the trials included in our study are likely representative of a cross-section of the general population, and we believe that our results are generalizable to multiple settings. To explore the association of primary outcome change with the reported intervention effect size, we used several analytic approaches, which gave consistent results.

    Limitations

    Some caveats should be recognized in our study. First, similar to other meta-epidemiological studies, our study mainly used published information and compared it with records in registries or protocols if necessary. Consequently, our results largely depended on the quality of reporting, which is often unsatisfactory.42 Second, almost 40% (221 of 559) of trials could not be assessed due to unavailable or insufficient information on primary outcomes from both articles and registries (n = 170) and unavailability of treatment effect size estimates (n = 51). Together with the reported publication bias (eg, trials sponsored by industry are less likely to be published), this probably led to an underestimation of the proportion of trials with primary outcome change and their association with the reported intervention effect size. Third, although the potential for selective reporting of primary outcomes was discernible in published trials, we first extracted the adjusted treatment differences and their SEs and then the unadjusted ones if there was no clear statement to specify which was the primary analysis result. This is because few trials released their original protocol and statistical analysis plan.

    Conclusions

    Results of this study suggest that primary outcome change in RCTs is common and likely overestimates intervention effects. Trial sponsors and investigators should register the primary outcomes, justify changes (if they occur), and report the results accordingly. This will allow the reader to critically appraise and interpret the trial results without bias. Reviewers and editors should routinely use prospectively registered data to avoid changes in primary outcomes during peer review, a practice that has been adopted by leading journals (eg, JAMA and BMJ). Readers and clinicians must be cautious about interpreting trial results and should be aware that trials with primary outcome change could lead to an overestimation of intervention effects.

    Back to top
    Article Information

    Accepted for Publication: May 28, 2019.

    Published: July 19, 2019. doi:10.1001/jamanetworkopen.2019.7242

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Chen T et al. JAMA Network Open.

    Corresponding Authors: Chao Li, PhD, Department of Epidemiology and Health Statistics, School of Public Health, Xi’an Jiaotong University Health Science Centre, Xi’an 710061, China (lcxjtu@xjtu.edu.cn); Tao Chen, PhD, Tropical Clinical Trials Unit, Department of Clinical Sciences, Liverpool School of Tropical Medicine, Liverpool L3 5QA, United Kingdom (tao.chen@lstmed.ac.uk).

    Author Contributions: Drs Chen and Li had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Chen, Li, Qin, Y. Wang, Yu, D. Wang, Cornelius.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Chen, Li, Qin.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Chen, Li, Y. Wang, Yu, D. Wang.

    Administrative, technical, or material support: Chen, Qin, Yu.

    Supervision: D. Wang, Cornelius.

    Conflict of Interest Disclosures: None reported.

    References
    1.
    Kane  RL, Wang  J, Garrard  J.  Reporting in randomized clinical trials improved after adoption of the CONSORT Statement.  J Clin Epidemiol. 2007;60(3):241-249. doi:10.1016/j.jclinepi.2006.06.016PubMedGoogle ScholarCrossref
    2.
    De Angelis  C, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.  Ann Intern Med. 2004;141(6):477-478. doi:10.7326/0003-4819-141-6-200409210-00109PubMedGoogle ScholarCrossref
    3.
    Zarin  DA, Ide  NC, Tse  T, Harlan  WR, West  JC, Lindberg  DA.  Issues in the registration of clinical trials.  JAMA. 2007;297(19):2112-2120. doi:10.1001/jama.297.19.2112PubMedGoogle ScholarCrossref
    4.
    Kahan  BC, Jairath  V.  Outcome pre-specification requires sufficient detail to guard against outcome switching in clinical trials: a case study.  Trials. 2018;19(1):265. doi:10.1186/s13063-018-2654-zPubMedGoogle ScholarCrossref
    5.
    Mathieu  S, Boutron  I, Moher  D, Altman  DG, Ravaud  P.  Comparison of registered and published primary outcomes in randomized controlled trials.  JAMA. 2009;302(9):977-984. doi:10.1001/jama.2009.1242PubMedGoogle ScholarCrossref
    6.
    Chan  AW, Hróbjartsson  A, Haahr  MT, Gøtzsche  PC, Altman  DG.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.  JAMA. 2004;291(20):2457-2465. doi:10.1001/jama.291.20.2457PubMedGoogle ScholarCrossref
    7.
    Jones  CW, Keil  LG, Holland  WC, Caughey  MC, Platts-Mills  TF.  Comparison of registered and published outcomes in randomized controlled trials: a systematic review.  BMC Med. 2015;13:282. doi:10.1186/s12916-015-0520-3PubMedGoogle ScholarCrossref
    8.
    Ioannidis  JP, Caplan  AL, Dal-Ré  R.  Outcome reporting bias in clinical trials: why monitoring matters.  BMJ. 2017;356:j408. doi:10.1136/bmj.j408PubMedGoogle ScholarCrossref
    9.
    Ramagopalan  S, Skingsley  AP, Handunnetthi  L,  et al.  Prevalence of primary outcome changes in clinical trials registered on ClinicalTrials.gov: a cross-sectional study.  F1000Res. 2014;3:77. doi:10.12688/f1000research.3784.1PubMedGoogle ScholarCrossref
    10.
    PROSPERO. Influence of primary outcome change on treatment effect estimates in clinical trials: cross-sectional study of registered clinical trials. CRD42017058054. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=58054. Accessed June 7, 2019.
    11.
    Abraha  I, Cherubini  A, Cozzolino  F,  et al.  Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study.  BMJ. 2015;350:h2445. doi:10.1136/bmj.h2445PubMedGoogle ScholarCrossref
    12.
    Bafeta  A, Dechartres  A, Trinquart  L, Yavchitz  A, Boutron  I, Ravaud  P.  Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study.  BMJ. 2012;344:e813. doi:10.1136/bmj.e813PubMedGoogle ScholarCrossref
    13.
    Als-Nielsen  B, Chen  W, Gluud  C, Kjaergard  LL.  Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events?  JAMA. 2003;290(7):921-928. doi:10.1001/jama.290.7.921PubMedGoogle ScholarCrossref
    14.
    Kivimäki  M, Jokela  M, Nyberg  ST,  et al; IPD-Work Consortium.  Long working hours and risk of coronary heart disease and stroke: a systematic review and meta-analysis of published and unpublished data for 603,838 individuals.  Lancet. 2015;386(10005):1739-1746. doi:10.1016/S0140-6736(15)60295-1PubMedGoogle ScholarCrossref
    15.
    Fang  X, Han  D, Cheng  Q,  et al.  Association of levels of physical activity with risk of Parkinson disease: a systematic review and meta-analysis.  JAMA Netw Open. 2018;1(5):e182421. doi:10.1001/jamanetworkopen.2018.2421PubMedGoogle ScholarCrossref
    16.
    Chinn  S.  A simple method for converting an odds ratio to effect size for use in meta-analysis.  Stat Med. 2000;19(22):3127-3131. doi:10.1002/1097-0258(20001130)19:22<3127::AID-SIM784>3.0.CO;2-MPubMedGoogle ScholarCrossref
    17.
    Killeen  S, Sourallous  P, Hunter  IA, Hartley  JE, Grady  HL.  Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals.  Ann Surg. 2014;259(1):193-196. doi:10.1097/SLA.0b013e318299d00bPubMedGoogle ScholarCrossref
    18.
    Li  XQ, Yang  GL, Tao  KM, Zhang  HQ, Zhou  QH, Ling  CQ.  Comparison of registered and published primary outcomes in randomized controlled trials of gastroenterology and hepatology.  Scand J Gastroenterol. 2013;48(12):1474-1483. doi:10.3109/00365521.2013.845909PubMedGoogle ScholarCrossref
    19.
    Rongen  JJ, Hannink  G.  Comparison of registered and published primary outcomes in randomized controlled trials of orthopaedic surgical interventions.  J Bone Joint Surg Am. 2016;98(5):403-409. doi:10.2106/JBJS.15.00400PubMedGoogle ScholarCrossref
    20.
    Anand  V, Scales  DC, Parshuram  CS, Kavanagh  BP.  Registration and design alterations of clinical trials in critical care: a cross-sectional observational study.  Intensive Care Med. 2014;40(5):700-722. doi:10.1007/s00134-014-3250-7PubMedGoogle ScholarCrossref
    21.
    Smith  HN, Bhandari  M, Mahomed  NN, Jan  M, Gandhi  R.  Comparison of arthroplasty trial publications after registration in ClinicalTrials.gov.  J Arthroplasty. 2012;27(7):1283-1288. doi:10.1016/j.arth.2011.11.005PubMedGoogle ScholarCrossref
    22.
    Rosenthal  R, Dwan  K.  Comparison of randomized controlled trial registry entries and content of reports in surgery journals.  Ann Surg. 2013;257(6):1007-1015. doi:10.1097/SLA.0b013e318283cf7fPubMedGoogle ScholarCrossref
    23.
    Fleming  PS, Koletsi  D, Dwan  K, Pandis  N.  Outcome discrepancies and selective reporting: impacting the leading journals?  PLoS One. 2015;10(5):e0127495. doi:10.1371/journal.pone.0127495PubMedGoogle ScholarCrossref
    24.
    Milette  K, Roseman  M, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: a systematic review.  J Psychosom Res. 2011;70(3):205-217. doi:10.1016/j.jpsychores.2010.09.015PubMedGoogle ScholarCrossref
    25.
    Jones  CW, Platts-Mills  TF.  Quality of registration for clinical trials published in emergency medicine journals.  Ann Emerg Med. 2012;60(4):458-64.e1.PubMedGoogle ScholarCrossref
    26.
    Ross  JS, Mulvey  GK, Hines  EM, Nissen  SE, Krumholz  HM.  Trial publication after registration in ClinicalTrials.gov: a cross-sectional analysis.  PLoS Med. 2009;6(9):e1000144. doi:10.1371/journal.pmed.1000144PubMedGoogle ScholarCrossref
    27.
    Jones  CW, Misemer  BS, Platts-Mills  TF,  et al.  Primary outcome switching among drug trials with and without principal investigator financial ties to industry: a cross-sectional study.  BMJ Open. 2018;8(2):e019831. doi:10.1136/bmjopen-2017-019831PubMedGoogle ScholarCrossref
    28.
    Schulz  KF, Altman  DG, Moher  D; CONSORT Group.  CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.  BMJ. 2010;340:c332. doi:10.1136/bmj.c332PubMedGoogle ScholarCrossref
    29.
    Gandhi  R, Jan  M, Smith  HN, Mahomed  NN, Bhandari  M.  Comparison of published orthopaedic trauma trials following registration in Clinicaltrials.gov.  BMC Musculoskelet Disord. 2011;12:278.PubMedGoogle ScholarCrossref
    30.
    Nankervis  H, Baibergenova  A, Williams  HC, Thomas  KS.  Prospective registration and outcome-reporting bias in randomized controlled trials of eczema treatments: a systematic review.  J Invest Dermatol. 2012;132(12):2727-2734. doi:10.1038/jid.2012.231PubMedGoogle ScholarCrossref
    31.
    Ewart  R, Lausen  H, Millian  N.  Undisclosed changes in outcomes in randomized controlled trials: an observational study.  Ann Fam Med. 2009;7(6):542-546. doi:10.1370/afm.1017PubMedGoogle ScholarCrossref
    32.
    You  B, Gan  HK, Pond  G, Chen  EX.  Consistency in the analysis and reporting of primary end points in oncology randomized controlled trials from registration to publication: a systematic review.  J Clin Oncol. 2012;30(2):210-216. doi:10.1200/JCO.2011.37.0890PubMedGoogle ScholarCrossref
    33.
    Vera-Badillo  FE, Shapiro  R, Ocana  A, Amir  E, Tannock  IF.  Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer.  Ann Oncol. 2013;24(5):1238-1244. doi:10.1093/annonc/mds636PubMedGoogle ScholarCrossref
    34.
    Saquib  N, Saquib  J, Ioannidis  JP.  Practices and impact of primary outcome adjustment in randomized controlled trials: meta-epidemiologic study.  BMJ. 2013;347:f4313. doi:10.1136/bmj.f4313PubMedGoogle ScholarCrossref
    35.
    Wildt  S, Krag  A, Gluud  L.  Characteristics of randomised trials on diseases in the digestive system registered in ClinicalTrials.gov: a retrospective analysis.  BMJ Open. 2011;1(2):e000309. doi:10.1136/bmjopen-2011-000309PubMedGoogle ScholarCrossref
    36.
    Liu  JP, Han  M, Li  XX,  et al.  Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study.  BMJ Open. 2013;3(7):e002968. doi:10.1136/bmjopen-2013-002968PubMedGoogle ScholarCrossref
    37.
    Nüesch  E, Trelle  S, Reichenbach  S,  et al.  The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study.  BMJ. 2009;339:b3244. doi:10.1136/bmj.b3244PubMedGoogle ScholarCrossref
    38.
    Nüesch  E, Trelle  S, Reichenbach  S,  et al.  Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study.  BMJ. 2010;341:c3515. doi:10.1136/bmj.c3515PubMedGoogle ScholarCrossref
    39.
    Dechartres  A, Trinquart  L, Boutron  I, Ravaud  P.  Influence of trial sample size on treatment effect estimates: meta-epidemiological study.  BMJ. 2013;346:f2304. doi:10.1136/bmj.f2304PubMedGoogle ScholarCrossref
    40.
    Pildal  J, Hróbjartsson  A, Jørgensen  KJ, Hilden  J, Altman  DG, Gøtzsche  PC.  Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials.  Int J Epidemiol. 2007;36(4):847-857. doi:10.1093/ije/dym087PubMedGoogle ScholarCrossref
    41.
    Sterne  JA, Jüni  P, Schulz  KF, Altman  DG, Bartlett  C, Egger  M.  Statistical methods for assessing the influence of study characteristics on treatment effects in “meta-epidemiological” research.  Stat Med. 2002;21(11):1513-1524. doi:10.1002/sim.1184PubMedGoogle ScholarCrossref
    42.
    Huwiler-Müntener  K, Jüni  P, Junker  C, Egger  M.  Quality of reporting of randomized trials as a measure of methodologic quality.  JAMA. 2002;287(21):2801-2804. doi:10.1001/jama.287.21.2801PubMedGoogle ScholarCrossref
    ×