Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes | Health Care Quality | JAMA | JAMA Network
[Skip to Navigation]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
DeAngelis CD, Drazen JM, Frizelle FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.  JAMA. 2004;292(11):1363-136415355936PubMedGoogle ScholarCrossref
Altman DG, Schulz KF, Moher D,  et al; CONSORT GROUP (Consolidated Standards of Reporting Trials).  The revised CONSORT statement for reporting randomized trials: explanation and elaboration.  Ann Intern Med. 2001;134(8):663-69411304107PubMedGoogle ScholarCrossref
Fletcher RH, Black B. “Spin” in scientific writing: scientific mischief and legal jeopardy.  Med Law. 2007;26(3):511-52517970249PubMedGoogle Scholar
Junger D. The rhetoric of research: embrace scientific rhetoric for its power.  BMJ. 1995;311(6996):617677870PubMedGoogle ScholarCrossref
Bailar JC. How to distort the scientific record without actually lying: truth, and arts of science.  Eur J Oncol. 2006;11(4):217-224Google Scholar
Marco CA, Larkin GL. Research ethics: ethical issues of data reporting and the quest for authenticity.  Acad Emerg Med. 2000;7(6):691-69410905651PubMedGoogle ScholarCrossref
Hróbjartsson A, Gøtzsche PC. Powerful spin in the conclusion of Wampold et al.'s re-analysis of placebo versus no-treatment trials despite similar results as in original review.  J Clin Psychol. 2007;63(4):373-37717279532PubMedGoogle ScholarCrossref
Jefferson T, Di Pietrantonj C, Debalini MG, Rivetti A, Demicheli V. Relation of study quality, concordance, take home message, funding, and impact in studies of influenza vaccines: systematic review.  BMJ. 2009;338:b35419213766PubMedGoogle ScholarCrossref
Hewitt CE, Mitchell N, Torgerson DJ. Listen to the data when results are not significant.  BMJ. 2008;336(7634):23-2518174597PubMedGoogle ScholarCrossref
Yank V, Rennie D, Bero LA. Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study.  BMJ. 2007;335(7631):1202-120518024482PubMedGoogle ScholarCrossref
Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed.  BMJ. 2010;340:c72320332510PubMedGoogle ScholarCrossref
Robinson KA, Dickersin K. Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed.  Int J Epidemiol. 2002;31(1):150-15311914311PubMedGoogle ScholarCrossref
Horton R. The rhetoric of research.  BMJ. 1995;310(6985):985-9877728037PubMedGoogle ScholarCrossref
Boutron I, Guittet L, Estellat C, Moher D, Hróbjartsson A, Ravaud P. Reporting methods of blinding in randomized trials assessing nonpharmacological treatments.  PLoS Med. 2007;4(2):e6117311468PubMedGoogle ScholarCrossref
Blader JC. Can keeping clinical trial participants blind to their study treatment adversely affect subsequent care? [published online ahead of print March 3, 2005].  Contemp Clin Trials. 2005;26(3):290-29915911463PubMedGoogle ScholarCrossref
Al-Marzouki S, Roberts I, Marshall T, Evans S. The effect of scientific misconduct on the results of clinical trials: a Delphi survey.  Contemp Clin Trials. 2005;26(3):331-33715911467PubMedGoogle ScholarCrossref
Gøtzsche PC. Believability of relative risks and odds ratios in abstracts: cross sectional study.  BMJ. 2006;333(7561):231-23416854948PubMedGoogle ScholarCrossref
Jørgensen KJ, Johansen HK, Gøtzsche PC. Flaws in design, analysis and interpretation of Pfizer's antifungal trials of voriconazole and uncritical subsequent quotations.  Trials. 2006;7:316542031PubMedGoogle ScholarCrossref
Hoekstra R, Finch S, Kiers HA, Johnson A. Probability as certainty: dichotomous thinking and the misuse of p values.  Psychon Bull Rev. 2006;13(6):1033-103717484431PubMedGoogle ScholarCrossref
Zinsmeister AR, Connor JT. Ten common statistical errors and how to avoid them.  Am J Gastroenterol. 2008;103(2):262-26618289193PubMedGoogle ScholarCrossref
Pocock SJ, Ware JH. Translating statistical findings into plain English.  Lancet. 2009;373(9679):1926-192819375158PubMedGoogle ScholarCrossref
Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine—reporting of subgroup analyses in clinical trials.  N Engl J Med. 2007;357(21):2189-219418032770PubMedGoogle ScholarCrossref
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials.  JAMA. 2009;302(9):977-98419724045PubMedGoogle ScholarCrossref
Rattinger G, Bero L. Factors associated with results and conclusions of trials of thiazolidinediones.  PLoS One. 2009;4(6):e582619503811PubMedGoogle ScholarCrossref
Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events?  JAMA. 2003;290(7):921-92812928469PubMedGoogle ScholarCrossref
Jørgensen AW, Hilden J, Gøtzsche PC. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review.  BMJ. 2006;333(7572):78217028106PubMedGoogle ScholarCrossref
Ioannidis JP. Limitations are not properly acknowledged in the scientific literature.  J Clin Epidemiol. 2007;60(4):324-32917346604PubMedGoogle ScholarCrossref
Matthews JN, Altman DG. Interaction 2: compare effect sizes not P values.  BMJ. 1996;313(7060):8088842080PubMedGoogle ScholarCrossref
Moyer CA. Between-groups study designs demand between-groups analyses: a response to Hernandez-Reif, Shor-Posner, Baez, Soto, Mendoza, Castillo, Quintero, Perez, and Zhang.  Evid Based Complement Alternat Med. 2009;6(1):49-5018955272PubMedGoogle ScholarCrossref
Hopewell S, Clarke M, Moher D,  et al;  CONSORT Group.  CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration.  PLoS Med. 2008;5(1):e2018215107PubMedGoogle ScholarCrossref
Horton R. The hidden research paper.  JAMA. 2002;287(21):2775-277812038909PubMedGoogle ScholarCrossref
Kaptchuk TJ. Effect of interpretive bias on research evidence.  BMJ. 2003;326(7404):1453-145512829562PubMedGoogle ScholarCrossref
Dwan K, Altman DG, Arnaiz JA,  et al.  Systematic review of the empirical evidence of study publication bias and outcome reporting bias.  PLoS One. 2008;3(8):e308118769481PubMedGoogle ScholarCrossref
Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation.  PLoS Med. 2008;5(11):e21719067477PubMedGoogle ScholarCrossref
Bero L, Oostvogel F, Bacchetti P, Lee K. Factors associated with findings of published trials of drug-drug comparisons: why some statins appear more efficacious than others.  PLoS Med. 2007;4(6):e18417550302PubMedGoogle ScholarCrossref
Woloshin S, Schwartz LM, Casella SL, Kennedy AT, Larson RJ. Press releases by academic medical centers: not so academic?  Ann Intern Med. 2009;150(9):613-61819414840PubMedGoogle ScholarCrossref
Bombardier C, Laine L, Reicin A,  et al; VIGOR Study Group.  Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis.  N Engl J Med. 2000;343(21):1520-152811087881PubMedGoogle ScholarCrossref
Krumholz HM, Ross JS, Presler AH, Egilman DS. What have we learnt from Vioxx?  BMJ. 2007;334(7585):120-12317235089PubMedGoogle ScholarCrossref
McGettigan P, Sly K, O’Connell D, Hill S, Henry D. The effects of information framing on the practices of physicians.  J Gen Intern Med. 1999;14(10):633-64210571710PubMedGoogle ScholarCrossref
Bucher HC, Weinbacher M, Gyr K. Influence of method of reporting study results on decision of physicians to prescribe drugs to lower cholesterol concentration.  BMJ. 1994;309(6957):761-7647950558PubMedGoogle ScholarCrossref
Chalmers I, Matthews R. What are the implications of optimism bias in clinical research?  Lancet. 2006;367(9509):449-45016473106PubMedGoogle ScholarCrossref
Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy.  N Engl J Med. 2008;358(3):252-26018199864PubMedGoogle ScholarCrossref
Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors.  BMJ. 2005;330(7494):75315681569PubMedGoogle ScholarCrossref
Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.  JAMA. 2004;291(20):2457-246515161896PubMedGoogle ScholarCrossref
Al-Marzouki S, Roberts I, Evans S, Marshall T. Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet.  Lancet. 2008;372(9634):20118640445PubMedGoogle ScholarCrossref
Song F, Parekh S, Hooper L,  et al.  Dissemination and publication of research findings: an updated review of related biases.  Health Technol Assess. 2010;14(8):iii, ix-xi, 1-19320181324PubMedGoogle Scholar
Original Contribution
May 26, 2010

Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes

Author Affiliations

Author Affiliations: Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (Drs Boutron and Altman and Ms Dutton); INSERM, U738, Paris, France (Drs Boutron and Ravaud); Assistance Publique des Hôpitaux de Paris, Hôpital Hôtel Dieu, Centre d’Épidémiologie Clinique, Paris (Drs Boutron and Ravaud); and Université Paris Descartes, Faculté de Médecine, Paris (Drs Boutron and Ravaud).

JAMA. 2010;303(20):2058-2064. doi:10.1001/jama.2010.651

Context Previous studies indicate that the interpretation of trial results can be distorted by authors of published reports.

Objective To identify the nature and frequency of distorted presentation or “spin” (ie, specific reporting strategies, whatever their motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results) in published reports of randomized controlled trials (RCTs) with statistically nonsignificant results for primary outcomes.

Data Sources March 2007 search of MEDLINE via PubMed using the Cochrane Highly Sensitive Search Strategy to identify reports of RCTs published in December 2006.

Study Selection Articles were included if they were parallel-group RCTs with a clearly identified primary outcome showing statistically nonsignificant results (ie, P ≥ .05).

Data Extraction Two readers appraised each selected article using a pretested, standardized data abstraction form developed in a pilot test.

Results From the 616 published reports of RCTs examined, 72 were eligible and appraised. The title was reported with spin in 13 articles (18.0%; 95% confidence interval [CI], 10.0%-28.9%). Spin was identified in the Results and Conclusions sections of the abstracts of 27 (37.5%; 95% CI, 26.4%-49.7%) and 42 (58.3%; 95% CI, 46.1%-69.8%) reports, respectively, with the conclusions of 17 (23.6%; 95% CI, 14.4%-35.1%) focusing only on treatment effectiveness. Spin was identified in the main-text Results, Discussion, and Conclusions sections of 21 (29.2%; 95% CI, 19.0%-41.1%), 31 (43.1%; 95% CI, 31.4%-55.3%), and 36 (50.0%; 95% CI, 38.0%-62.0%) reports, respectively. More than 40% of the reports had spin in at least 2 of these sections in the main text.

Conclusion In this representative sample of RCTs published in 2006 with statistically nonsignificant primary outcomes, the reporting and interpretation of findings was frequently inconsistent with the results.