Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes | Health Care Quality | JAMA | JAMA Network
[Skip to Navigation]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 35.175.212.130. Please contact the publisher to request reinstatement.
1.
DeAngelis CD, Drazen JM, Frizelle FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.  JAMA. 2004;292(11):1363-136415355936PubMedGoogle ScholarCrossref
2.
Altman DG, Schulz KF, Moher D,  et al; CONSORT GROUP (Consolidated Standards of Reporting Trials).  The revised CONSORT statement for reporting randomized trials: explanation and elaboration.  Ann Intern Med. 2001;134(8):663-69411304107PubMedGoogle ScholarCrossref
3.
Fletcher RH, Black B. “Spin” in scientific writing: scientific mischief and legal jeopardy.  Med Law. 2007;26(3):511-52517970249PubMedGoogle Scholar
4.
Junger D. The rhetoric of research: embrace scientific rhetoric for its power.  BMJ. 1995;311(6996):617677870PubMedGoogle ScholarCrossref
5.
Bailar JC. How to distort the scientific record without actually lying: truth, and arts of science.  Eur J Oncol. 2006;11(4):217-224Google Scholar
6.
Marco CA, Larkin GL. Research ethics: ethical issues of data reporting and the quest for authenticity.  Acad Emerg Med. 2000;7(6):691-69410905651PubMedGoogle ScholarCrossref
7.
Hróbjartsson A, Gøtzsche PC. Powerful spin in the conclusion of Wampold et al.'s re-analysis of placebo versus no-treatment trials despite similar results as in original review.  J Clin Psychol. 2007;63(4):373-37717279532PubMedGoogle ScholarCrossref
8.
Jefferson T, Di Pietrantonj C, Debalini MG, Rivetti A, Demicheli V. Relation of study quality, concordance, take home message, funding, and impact in studies of influenza vaccines: systematic review.  BMJ. 2009;338:b35419213766PubMedGoogle ScholarCrossref
9.
Hewitt CE, Mitchell N, Torgerson DJ. Listen to the data when results are not significant.  BMJ. 2008;336(7634):23-2518174597PubMedGoogle ScholarCrossref
10.
Yank V, Rennie D, Bero LA. Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study.  BMJ. 2007;335(7631):1202-120518024482PubMedGoogle ScholarCrossref
11.
Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed.  BMJ. 2010;340:c72320332510PubMedGoogle ScholarCrossref
12.
Robinson KA, Dickersin K. Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed.  Int J Epidemiol. 2002;31(1):150-15311914311PubMedGoogle ScholarCrossref
13.
Horton R. The rhetoric of research.  BMJ. 1995;310(6985):985-9877728037PubMedGoogle ScholarCrossref
14.
Boutron I, Guittet L, Estellat C, Moher D, Hróbjartsson A, Ravaud P. Reporting methods of blinding in randomized trials assessing nonpharmacological treatments.  PLoS Med. 2007;4(2):e6117311468PubMedGoogle ScholarCrossref
15.
Blader JC. Can keeping clinical trial participants blind to their study treatment adversely affect subsequent care? [published online ahead of print March 3, 2005].  Contemp Clin Trials. 2005;26(3):290-29915911463PubMedGoogle ScholarCrossref
16.
Al-Marzouki S, Roberts I, Marshall T, Evans S. The effect of scientific misconduct on the results of clinical trials: a Delphi survey.  Contemp Clin Trials. 2005;26(3):331-33715911467PubMedGoogle ScholarCrossref
17.
Gøtzsche PC. Believability of relative risks and odds ratios in abstracts: cross sectional study.  BMJ. 2006;333(7561):231-23416854948PubMedGoogle ScholarCrossref
18.
Jørgensen KJ, Johansen HK, Gøtzsche PC. Flaws in design, analysis and interpretation of Pfizer's antifungal trials of voriconazole and uncritical subsequent quotations.  Trials. 2006;7:316542031PubMedGoogle ScholarCrossref
19.
Hoekstra R, Finch S, Kiers HA, Johnson A. Probability as certainty: dichotomous thinking and the misuse of p values.  Psychon Bull Rev. 2006;13(6):1033-103717484431PubMedGoogle ScholarCrossref
20.
Zinsmeister AR, Connor JT. Ten common statistical errors and how to avoid them.  Am J Gastroenterol. 2008;103(2):262-26618289193PubMedGoogle ScholarCrossref
21.
Pocock SJ, Ware JH. Translating statistical findings into plain English.  Lancet. 2009;373(9679):1926-192819375158PubMedGoogle ScholarCrossref
22.
Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine—reporting of subgroup analyses in clinical trials.  N Engl J Med. 2007;357(21):2189-219418032770PubMedGoogle ScholarCrossref
23.
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials.  JAMA. 2009;302(9):977-98419724045PubMedGoogle ScholarCrossref
24.
Rattinger G, Bero L. Factors associated with results and conclusions of trials of thiazolidinediones.  PLoS One. 2009;4(6):e582619503811PubMedGoogle ScholarCrossref
25.
Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events?  JAMA. 2003;290(7):921-92812928469PubMedGoogle ScholarCrossref
26.
Jørgensen AW, Hilden J, Gøtzsche PC. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review.  BMJ. 2006;333(7572):78217028106PubMedGoogle ScholarCrossref
27.
Ioannidis JP. Limitations are not properly acknowledged in the scientific literature.  J Clin Epidemiol. 2007;60(4):324-32917346604PubMedGoogle ScholarCrossref
28.
Matthews JN, Altman DG. Interaction 2: compare effect sizes not P values.  BMJ. 1996;313(7060):8088842080PubMedGoogle ScholarCrossref
29.
Moyer CA. Between-groups study designs demand between-groups analyses: a response to Hernandez-Reif, Shor-Posner, Baez, Soto, Mendoza, Castillo, Quintero, Perez, and Zhang.  Evid Based Complement Alternat Med. 2009;6(1):49-5018955272PubMedGoogle ScholarCrossref
30.
Hopewell S, Clarke M, Moher D,  et al;  CONSORT Group.  CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration.  PLoS Med. 2008;5(1):e2018215107PubMedGoogle ScholarCrossref
31.
Horton R. The hidden research paper.  JAMA. 2002;287(21):2775-277812038909PubMedGoogle ScholarCrossref
32.
Kaptchuk TJ. Effect of interpretive bias on research evidence.  BMJ. 2003;326(7404):1453-145512829562PubMedGoogle ScholarCrossref
33.
Dwan K, Altman DG, Arnaiz JA,  et al.  Systematic review of the empirical evidence of study publication bias and outcome reporting bias.  PLoS One. 2008;3(8):e308118769481PubMedGoogle ScholarCrossref
34.
Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation.  PLoS Med. 2008;5(11):e21719067477PubMedGoogle ScholarCrossref
35.
Bero L, Oostvogel F, Bacchetti P, Lee K. Factors associated with findings of published trials of drug-drug comparisons: why some statins appear more efficacious than others.  PLoS Med. 2007;4(6):e18417550302PubMedGoogle ScholarCrossref
36.
Woloshin S, Schwartz LM, Casella SL, Kennedy AT, Larson RJ. Press releases by academic medical centers: not so academic?  Ann Intern Med. 2009;150(9):613-61819414840PubMedGoogle ScholarCrossref
37.
Bombardier C, Laine L, Reicin A,  et al; VIGOR Study Group.  Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis.  N Engl J Med. 2000;343(21):1520-152811087881PubMedGoogle ScholarCrossref
38.
Krumholz HM, Ross JS, Presler AH, Egilman DS. What have we learnt from Vioxx?  BMJ. 2007;334(7585):120-12317235089PubMedGoogle ScholarCrossref
39.
McGettigan P, Sly K, O’Connell D, Hill S, Henry D. The effects of information framing on the practices of physicians.  J Gen Intern Med. 1999;14(10):633-64210571710PubMedGoogle ScholarCrossref
40.
Bucher HC, Weinbacher M, Gyr K. Influence of method of reporting study results on decision of physicians to prescribe drugs to lower cholesterol concentration.  BMJ. 1994;309(6957):761-7647950558PubMedGoogle ScholarCrossref
41.
Chalmers I, Matthews R. What are the implications of optimism bias in clinical research?  Lancet. 2006;367(9509):449-45016473106PubMedGoogle ScholarCrossref
42.
Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy.  N Engl J Med. 2008;358(3):252-26018199864PubMedGoogle ScholarCrossref
43.
Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors.  BMJ. 2005;330(7494):75315681569PubMedGoogle ScholarCrossref
44.
Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.  JAMA. 2004;291(20):2457-246515161896PubMedGoogle ScholarCrossref
45.
Al-Marzouki S, Roberts I, Evans S, Marshall T. Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet.  Lancet. 2008;372(9634):20118640445PubMedGoogle ScholarCrossref
46.
Song F, Parekh S, Hooper L,  et al.  Dissemination and publication of research findings: an updated review of related biases.  Health Technol Assess. 2010;14(8):iii, ix-xi, 1-19320181324PubMedGoogle Scholar
Original Contribution
May 26, 2010

Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes

Author Affiliations

Author Affiliations: Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (Drs Boutron and Altman and Ms Dutton); INSERM, U738, Paris, France (Drs Boutron and Ravaud); Assistance Publique des Hôpitaux de Paris, Hôpital Hôtel Dieu, Centre d’Épidémiologie Clinique, Paris (Drs Boutron and Ravaud); and Université Paris Descartes, Faculté de Médecine, Paris (Drs Boutron and Ravaud).

JAMA. 2010;303(20):2058-2064. doi:10.1001/jama.2010.651
Abstract

Context Previous studies indicate that the interpretation of trial results can be distorted by authors of published reports.

Objective To identify the nature and frequency of distorted presentation or “spin” (ie, specific reporting strategies, whatever their motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results) in published reports of randomized controlled trials (RCTs) with statistically nonsignificant results for primary outcomes.

Data Sources March 2007 search of MEDLINE via PubMed using the Cochrane Highly Sensitive Search Strategy to identify reports of RCTs published in December 2006.

Study Selection Articles were included if they were parallel-group RCTs with a clearly identified primary outcome showing statistically nonsignificant results (ie, P ≥ .05).

Data Extraction Two readers appraised each selected article using a pretested, standardized data abstraction form developed in a pilot test.

Results From the 616 published reports of RCTs examined, 72 were eligible and appraised. The title was reported with spin in 13 articles (18.0%; 95% confidence interval [CI], 10.0%-28.9%). Spin was identified in the Results and Conclusions sections of the abstracts of 27 (37.5%; 95% CI, 26.4%-49.7%) and 42 (58.3%; 95% CI, 46.1%-69.8%) reports, respectively, with the conclusions of 17 (23.6%; 95% CI, 14.4%-35.1%) focusing only on treatment effectiveness. Spin was identified in the main-text Results, Discussion, and Conclusions sections of 21 (29.2%; 95% CI, 19.0%-41.1%), 31 (43.1%; 95% CI, 31.4%-55.3%), and 36 (50.0%; 95% CI, 38.0%-62.0%) reports, respectively. More than 40% of the reports had spin in at least 2 of these sections in the main text.

Conclusion In this representative sample of RCTs published in 2006 with statistically nonsignificant primary outcomes, the reporting and interpretation of findings was frequently inconsistent with the results.

×