The Scientific Impact of Positive and Negative Phase 3 Cancer Clinical Trials | Medical Journals and Publishing | JAMA Oncology | JAMA Network
[Skip to Navigation]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
Djulbegovic  B, Kumar  A, Soares  HP,  et al.  Treatment success in cancer: new cancer treatment successes identified in phase 3 randomized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006.  Arch Intern Med. 2008;168(6):632-642.PubMedGoogle ScholarCrossref
Hopewell  S, Loudon  K, Clarke  MJ, Oxman  AD, Dickersin  K.  Publication bias in clinical trials due to statistical significance or direction of trial results.  Cochrane Database Syst Rev. 2009;(1):MR000006.PubMedGoogle Scholar
Ramsey  S, Scoggins  J.  Commentary: practicing on the tip of an information iceberg? evidence of underpublication of registered clinical trials in oncology.  Oncologist. 2008;13(9):925-929.PubMedGoogle ScholarCrossref
Soares  HP, Kumar  A, Daniels  S,  et al.  Evaluation of new treatments in radiation oncology: are they better than standard treatments?  JAMA. 2005;293(8):970-978.PubMedGoogle ScholarCrossref
Korn  EL, Freidlin  B, Mooney  M, Abrams  JS.  Accrual experience of National Cancer Institute Cooperative Group phase III trials activated from 2000 to 2007.  J Clin Oncol. 2010;28(35):5197-5201.PubMedGoogle ScholarCrossref
Garfield  E.  Citation analysis as a tool in journal evaluation.  Science. 1972;178(4060):471-479.PubMedGoogle ScholarCrossref
Narin  F.  Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity. Washington, DC: National Science Foundation; 1976.
Liang  KY, Zeger  SL.  Longitudinal data analysis using generalized linear models.  Biometrika. 1986;73(1):13-22.Google ScholarCrossref
Zeger  SL, Liang  KY.  Longitudinal data analysis for discrete and continuous outcomes.  Biometrics. 1986;42(1):121-130.PubMedGoogle ScholarCrossref
Pan  W.  Akaike’s information criterion in generalized estimating equations.  Biometrics. 2001;57(1):120-125.PubMedGoogle ScholarCrossref
Simes  RJ.  Publication bias: the case for an international registry of clinical trials.  J Clin Oncol. 1986;4(10):1529-1541.PubMedGoogle Scholar
National Institutes of Health launches "" [news release]. Bethesda, MD: National Institutes of Health; February 29, 2000. Accessed March 4, 2016.
Djulbegovic  B.  The paradox of equipoise: the principle that drives and limits therapeutic discoveries in clinical research.  Cancer Control. 2009;16(4):342-347.PubMedGoogle Scholar
Hutchins  LF, Unger  JM, Crowley  JJ, Coltman  CA  Jr, Albain  KS.  Underrepresentation of patients 65 years of age or older in cancer-treatment trials.  N Engl J Med. 1999;341(27):2061-2067.PubMedGoogle ScholarCrossref
Unger  JM, Barlow  WE, Martin  DP,  et al.  Comparison of survival outcomes among cancer patients treated in and out of clinical trials.  J Natl Cancer Inst. 2014;106(3):dju002.PubMedGoogle ScholarCrossref
Hershman  DL, Unger  JM.  ‘Minority report’: how best to analyze clinical trial data to address disparities.  Breast Cancer Res Treat. 2009;118(3):519-521.Google ScholarCrossref
Institute of Medicine (IOM).  Sharing Clinical Trial Data: Maximizing Benefits, Minimizing Risk. Washington, DC: National Academies Press; 2015.
Kent  EE, Mitchell  SA, Castro  KM,  et al.  Cancer care delivery research: building the evidence base to support practice change in community oncology.  J Clin Oncol. 2015;33(24):2705-2711.PubMedGoogle ScholarCrossref
Users Guide. Version 4.0. Accessed December 12, 2015.
The YODA Project. Welcome to the YODA Project. Accessed December 12, 2015.
Khabsa  M, Giles  CL.  The number of scholarly documents on the public web.  PLoS One. 2014;9(5):e93949.PubMedGoogle ScholarCrossref
Harzing  A.  A longitudinal study of Google Scholar coverage between 2012 and 2013.  Scientometrics. 2014;98(1):565-575.Google ScholarCrossref
Bakkalbasi  N, Bauer  K, Glover  J, Wang  L.  Three options for citation tracking: Google Scholar, Scopus and Web of Science.  Biomed Digit Libr. 2006;3:7.PubMedGoogle ScholarCrossref
Meho  LI, Yank  K.  Impact of data sources on citation counts and rankings of LIS faculty: Web of Science vs Scopus and Google Scholar.  J Am Soc Inf Sci Technol. 2007;58(13):2105-2125.Google ScholarCrossref
Fortin  JM, Currie  DJ.  Big science vs little science: how scientific impact scales with funding.  PLoS One. 2013;8(6):e65263.PubMedGoogle ScholarCrossref
King  DA.  The scientific impact of nations.  Nature. 2004;430(6997):311-316.PubMedGoogle ScholarCrossref
Bollen  J, Van de Sompel  H, Hagberg  A, Chute  R.  A principal component analysis of 39 scientific impact measures.  PLoS One. 2009;4(6):e6022.PubMedGoogle ScholarCrossref
Original Investigation
July 2016

The Scientific Impact of Positive and Negative Phase 3 Cancer Clinical Trials

Author Affiliations
  • 1SWOG Statistical Center, Fred Hutchinson Cancer Research Center, Seattle, Washington
  • 2Fred Hutchinson Cancer Research Center, Seattle, Washington
  • 3SWOG Group Chair’s Office/Knight Cancer Institute, Oregon Health and Science University, Portland
  • 4Columbia University, New York, New York
JAMA Oncol. 2016;2(7):875-881. doi:10.1001/jamaoncol.2015.6487

Importance  Positive phase 3 cancer clinical trials are widely hailed, while trials with negative results are often interpreted as scientific failures. We hypothesized that these interpretations would be reflected in the scientific literature.

Objective  To compare the scientific impact of positive vs negative phase 3 cancer clinical treatment trials.

Design, Setting, and Participants  We examined the phase 3 trial history of SWOG, a national cancer clinical trials consortium, over a 30-year period (1985-2014). Scientific impact was assessed according to multiple publication and citation outcomes. Citation data were obtained using Google Scholar. Citation counts were compared using generalized estimating equations for Poisson regression. Any trial that was formally evaluated for the randomized treatment comparison was included for analysis of publication and citation outcomes. Trials were categorized as positive if they achieved a statistically significant result in favor of the new experimental treatment for the protocol-specified primary end point. Trials were categorized as negative if they achieved a statistically significant result in favor of standard therapy or a null result with no statistically significant benefit for either the experimental or standard therapy.

Main Outcomes and Measures  Impact factors for the journals publishing the primary trial results, and the number of citations for the primary trial articles and all secondary articles associated with the trials.

Results  Ninety-four studies enrolling n = 46 424 patients were analyzed. Twenty-eight percent of trials were positive (26 of 94). The primary publications from positive trials were published in journals with higher mean (SD) 2-year impact factors (28 [19] vs 18 [13]; P = .007) and were cited twice as often as negative trials (mean per year, 43 vs 21; relative risk, 2.0; 95% CI, 1.1-3.9; P = .03). However, the number of citations from all primary and secondary articles did not significantly differ between positive and negative trials (mean per year, 55 vs 45; relative risk, 1.2; 95% CI, 0.7-2.3; P = .53).

Conclusions and Relevance  The scientific impact of the primary articles from positive phase 3 randomized cancer clinical trials was twice as great as for negative trials. But when all of the articles associated with the trials were considered, the scientific impact between positive and negative trials was similar. Positive trials indicate clinical advances, but negative trials also have a sizeable scientific impact by generating important scientific observations and new hypotheses and by showing what new treatments should not be used.