[Skip to Content]
[Skip to Content Landing]
Figure 1.
Search Algorithm to Identify Practice-Influential Phase 3 Cancer Clinical Treatment Trials
Search Algorithm to Identify Practice-Influential Phase 3 Cancer Clinical Treatment Trials

FDA indicates US Food and Drug Administration; NCCN, National Comprehensive Cancer Network.

Figure 2.
Trial Selection Flow Diagram
Trial Selection Flow Diagram
Figure 3.
Disposition of Practice-Influential Trials
Disposition of Practice-Influential Trials

Practice influence is determined by the trial’s influence on National Comprehensive Cancer Network (NCCN) guidelines, US Food and Drug Administration (FDA) new drug approvals, or both. Other trials were deemed not practice influential.

Table 1.  
Characteristics of Phase 3 Cancer Treatment Clinical Trials at SWOG
Characteristics of Phase 3 Cancer Treatment Clinical Trials at SWOG
Table 2.  
Cost Estimates for New Drug Approval in the Pharmaceutical Industrya
Cost Estimates for New Drug Approval in the Pharmaceutical Industrya
1.
Institute of Medicine. National Cancer Clinical Trials System for the 21st Century: Reinvigorating the NCI Cooperative Group Program. Washington, DC: National Academies Press; 2010.
2.
National Cancer Institute. Cancer Moonshot Blue Ribbon Panel. https://www.cancer.gov/research/key-initiatives/moonshot-cancer-initiative/blue-ribbon-panel. Updated April 26, 2017. Accessed July 3, 2019.
3.
SWOG Cancer Research Network. History and impact. https://www.swog.org/about/history-impact. Accessed July 3, 2019.
4.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.  Lancet. 2007;370(9596):1453-1457. doi:10.1016/S0140-6736(07)61602-XPubMedGoogle ScholarCrossref
5.
Wagner  J, Marquart  J, Ruby  J,  et al.  Frequency and level of evidence used in recommendations by the National Comprehensive Cancer Network guidelines beyond approvals of the US Food and Drug Administration: retrospective observational study.  BMJ. 2018;360:k668. doi:10.1136/bmj.k668PubMedGoogle ScholarCrossref
6.
Cabana  MD, Rand  CS, Powe  NR,  et al.  Why don’t physicians follow clinical practice guidelines? a framework for improvement.  JAMA. 1999;282(15):1458-1465. doi:10.1001/jama.282.15.1458PubMedGoogle ScholarCrossref
7.
Francke  AL, Smit  MC, de Veer  AJE, Mistiaen  P.  Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review.  BMC Med Inform Decis Mak. 2008;8:38. doi:10.1186/1472-6947-8-38PubMedGoogle ScholarCrossref
8.
Lesaffre  E.  Superiority, equivalence, and non-inferiority trials.  Bull NYU Hosp Jt Dis. 2008;66(2):150-154.PubMedGoogle Scholar
9.
National Comprehensive Cancer Network. About the NCCN clinical practice guidelines in oncology. https://www.nccn.org/professionals/. Accessed April 3, 2019.
10.
National Comprehensive Cancer Network. Development and update of the NCCN guidelines. https://www.nccn.org/professionals/development.aspx. Accessed April 3, 2019.
11.
National Cancer Institute. A to Z list of cancer drugs. https://www.cancer.gov/about-cancer/treatment/drugs. Accessed April 3, 2019.
12.
Sun  J, Wei  Q, Zhou  Y, Wang  J, Liu  Q, Xu  H.  A systematic analysis of FDA-approved anticancer drugs.  BMC Syst Biol. 2017;11(suppl 5):87. doi:10.1186/s12918-017-0464-7PubMedGoogle ScholarCrossref
13.
IBM, Inc. IBM Micromedex web applications access. http://www.micromedexsolutions.com/home/dispatch. Updated 2019. Accessed October 1, 2018.
14.
US Food and Drug Administration. FDA approved drug products. https://www.accessdata.fda.gov/scripts/cder/daf/. Accessed October 1, 2018.
15.
National Institutes of Health. Research Portfolio Online Reporting Tools (RePORT). https://projectreporter.nih.gov/reporter.cfm. Updated July 20, 2019. Accessed October 16, 2018.
16.
US Department of Labor Statistics. Consumer Price Index US city average (1982-1984 = 100). https://www.bls.gov/regions/new-england/data/consumerpriceindex_us_table.htm. Updated 2019. Accessed December 17, 2018.
17.
R: A Language and Environment for Statistical Computing. Vienna, Austria: R Core Team; 2017. https://www.R-project.org/. Accessed October 4, 2017.
18.
Lamm  DL, Blumenstein  BA, Crawford  ED,  et al.  A randomized trial of intravesical doxorubicin and immunotherapy with bacille Calmette-Guérin for transitional-cell carcinoma of the bladder.  N Engl J Med. 1991;325(17):1205-1209. doi:10.1056/NEJM199110243251703PubMedGoogle ScholarCrossref
19.
Petersdorf  SH, Kopecky  KJ, Slovak  M,  et al.  A phase 3 study of gemtuzumab ozogamicin during induction and postconsolidation therapy in younger patients with acute myeloid leukemia.  Blood. 2013;121(24):4854-4860. doi:10.1182/blood-2013-01-466706PubMedGoogle ScholarCrossref
20.
U.S. Food and Drug Administration. FDA news release: FDA approves Mylotarg for treatment of acute myeloid leukemia. https://www.fda.gov/newsevents/newsroom/pressannouncements/ucm574507.htm. Released September 1, 2017. Accessed February 21, 2019.
21.
Benghiat  A.  The influence of clinical trials in the treatment of cancer: results of a survey.  Br J Radiol. 1985;58(690):572-573. doi:10.1259/0007-1285-58-690-572PubMedGoogle ScholarCrossref
22.
Stephens  R, Gibson  D.  The impact of clinical trials on the treatment of lung cancer.  Clin Oncol (R Coll Radiol). 1993;5(4):211-219. doi:10.1016/S0936-6555(05)80231-6PubMedGoogle ScholarCrossref
23.
Lewison  G, Sullivan  R.  The impact of cancer research: how publications influence UK cancer clinical guidelines.  Br J Cancer. 2008;98(12):1944-1950. doi:10.1038/sj.bjc.6604405PubMedGoogle ScholarCrossref
24.
Pallari  E, Fox  AW, Lewison  G.  Differential research impact in cancer practice guidelines’ evidence base: lessons from ESMO, NICE and SIGN.  ESMO Open. 2018;3(1):e000258. doi:10.1136/esmoopen-2017-000258PubMedGoogle ScholarCrossref
25.
Poonacha  TK, Go  RS.  Level of scientific evidence underlying recommendations arising from the National Comprehensive Cancer Network clinical practice guidelines.  J Clin Oncol. 2011;29(2):186-191. doi:10.1200/JCO.2010.31.6414PubMedGoogle ScholarCrossref
26.
Al-Sarraf  M, LeBlanc  M, Giri  PG,  et al.  Chemoradiotherapy versus radiotherapy in patients with advanced nasopharyngeal cancer: phase III randomized intergroup study 0099.  J Clin Oncol. 1998;16(4):1310-1317. doi:10.1200/JCO.1998.16.4.1310PubMedGoogle ScholarCrossref
27.
Adelstein  DJ, Li  Y, Adams  GL,  et al.  An intergroup phase III comparison of standard radiation therapy and two schedules of concurrent chemoradiotherapy in patients with unresectable squamous cell head and neck cancer.  J Clin Oncol. 2003;21(1):92-98. doi:10.1200/JCO.2003.01.008PubMedGoogle ScholarCrossref
28.
Cooper  JS, Pajak  TF, Forastiere  AA,  et al; Radiation Therapy Oncology Group 9501/Intergroup.  Postoperative concurrent radiotherapy and chemotherapy for high-risk squamous-cell carcinoma of the head and neck.  N Engl J Med. 2004;350(19):1937-1944. doi:10.1056/NEJMoa032646PubMedGoogle ScholarCrossref
29.
Herskovic  A, Martz  K, al-Sarraf  M,  et al.  Combined chemotherapy and radiotherapy compared with radiotherapy alone in patients with cancer of the esophagus.  N Engl J Med. 1992;326(24):1593-1598. doi:10.1056/NEJM199206113262403PubMedGoogle ScholarCrossref
30.
Macdonald  JS, Smalley  SR, Benedetti  J,  et al.  Chemoradiotherapy after surgery compared with surgery alone for adenocarcinoma of the stomach or gastroesophageal junction.  N Engl J Med. 2001;345(10):725-730. doi:10.1056/NEJMoa010187PubMedGoogle ScholarCrossref
31.
Crawford  ED, Eisenberger  MA, McLeod  DG,  et al.  A controlled trial of leuprolide with and without flutamide in prostatic carcinoma.  N Engl J Med. 1989;321(7):419-424. doi:10.1056/NEJM198908173210702PubMedGoogle ScholarCrossref
32.
Messing  EM, Manola  J, Sarosdy  M, Wilding  G, Crawford  ED, Trump  D.  Immediate hormonal therapy compared with observation after radical prostatectomy and pelvic lymphadenectomy in men with node-positive prostate cancer.  N Engl J Med. 1999;341(24):1781-1788. doi:10.1056/NEJM199912093412401PubMedGoogle ScholarCrossref
33.
Sweeney  CJ, Chen  YH, Carducci  M,  et al.  Chemohormonal therapy in metastatic hormone-sensitive prostate cancer.  N Engl J Med. 2015;373(8):737-746. doi:10.1056/NEJMoa1503747PubMedGoogle ScholarCrossref
34.
Warde  P, Mason  M, Ding  K,  et al; NCIC CTG PR.3/MRC UK PR07 Investigators.  Combined androgen deprivation therapy and radiation therapy for locally advanced prostate cancer: a randomised, phase 3 trial.  Lancet. 2011;378(9809):2104-2111. doi:10.1016/S0140-6736(11)61095-7PubMedGoogle ScholarCrossref
35.
Lamm  DL, Blumenstein  BA, David Crawford  E,  et al.  Randomized intergroup comparison of bacillus Calmette-Guerin immunotherapy and mitomycin C chemotherapy prophylaxis in superficial transitional cell carcinoma of the bladder: a Southwest Oncology Group Study.  Urol Oncol. 1995;1(3):119-126. doi:10.1016/1078-1439(95)00041-FPubMedGoogle ScholarCrossref
36.
Lamm  DL, Blumenstein  BA, Crissman  JD,  et al.  Maintenance bacillus Calmette-Guerin immunotherapy for recurrent TA, T1 and carcinoma in situ transitional cell carcinoma of the bladder: a randomized Southwest Oncology Group Study.  J Urol. 2000;163(4):1124-1129. doi:10.1016/S0022-5347(05)67707-5PubMedGoogle ScholarCrossref
37.
Moore  HC, Green  SJ, Gralow  JR,  et al; Southwest Oncology Group/Intergroup Study 9623.  Intensive dose-dense compared with high-dose adjuvant chemotherapy for high-risk operable breast cancer: Southwest Oncology Group/Intergroup Study 9623.  J Clin Oncol. 2007;25(13):1677-1682. doi:10.1200/JCO.2006.08.9383PubMedGoogle ScholarCrossref
38.
Peters  WP, Rosner  GL, Vredenburgh  JJ,  et al.  Prospective, randomized comparison of high-dose chemotherapy with stem-cell support versus intermediate-dose chemotherapy after surgery and adjuvant chemotherapy in women with high-risk primary breast cancer: a report of CALGB 9082, SWOG 9114, and NCIC MA-13.  J Clin Oncol. 2005;23(10):2191-2200. doi:10.1200/JCO.2005.10.202PubMedGoogle ScholarCrossref
39.
Stadtmauer  EA, O’Neill  A, Goldstein  LJ,  et al; Philadelphia Bone Marrow Transplant Group.  Conventional-dose chemotherapy compared with high-dose chemotherapy plus autologous hematopoietic stem-cell transplantation for metastatic breast cancer.  N Engl J Med. 2000;342(15):1069-1076. doi:10.1056/NEJM200004133421501PubMedGoogle ScholarCrossref
40.
Tallman  MS, Gray  R, Robert  NJ,  et al.  Conventional adjuvant chemotherapy with or without high-dose chemotherapy and autologous stem-cell transplantation in high-risk breast cancer.  N Engl J Med. 2003;349(1):17-26. doi:10.1056/NEJMoa030684PubMedGoogle ScholarCrossref
41.
Wong  CH, Siah  KW, Lo  AW.  Estimation of clinical trial success rates and related parameters.  Biostatistics. 2019;20(2):273-286. doi:10.1093/biostatistics/kxx069PubMedGoogle ScholarCrossref
42.
DiMasi  JA, Feldman  L, Seckler  A, Wilson  A.  Trends in risks associated with new drug development: success rates for investigational drugs.  Clin Pharmacol Ther. 2010;87(3):272-277. doi:10.1038/clpt.2009.295PubMedGoogle ScholarCrossref
43.
Kola  I, Landis  J.  Can the pharmaceutical industry reduce attrition rates?  Nat Rev Drug Discov. 2004;3(8):711-715. doi:10.1038/nrd1470PubMedGoogle ScholarCrossref
44.
Hay  M, Thomas  DW, Craighead  JL, Economides  C, Rosenthal  J.  Clinical development success rates for investigational drugs.  Nat Biotechnol. 2014;32(1):40-51. doi:10.1038/nbt.2786PubMedGoogle ScholarCrossref
45.
DiMasi  JA, Grabowski  HG, Hansen  RW.  Innovation in the pharmaceutical industry: new estimates of R&D costs.  J Health Econ. 2016;47:20-33. doi:10.1016/j.jhealeco.2016.01.012PubMedGoogle ScholarCrossref
46.
Adams  CP, Brantner  VV.  Estimating the cost of new drug development: is it really 802 million dollars?  Health Aff (Millwood). 2006;25(2):420-428. doi:10.1377/hlthaff.25.2.420PubMedGoogle ScholarCrossref
47.
Adams  CP, Brantner  VV.  Spending on new drug development.  Health Econ. 2010;19(2):130-141. doi:10.1002/hec.1454PubMedGoogle ScholarCrossref
48.
DiMasi  JA, Hansen  RW, Grabowski  HG.  The price of innovation: new estimates of drug development costs.  J Health Econ. 2003;22(2):151-185. doi:10.1016/S0167-6296(02)00126-1PubMedGoogle ScholarCrossref
49.
Gilbert  J, Preston  H, Singh  A. Rebuilding big pharma's business model in vivo. Business Med Rep 21. https://invivo.pharmaintelligence.informa.com/IV002027/Rebuilding-Big-Pharmas-Business-Model. Published November 1, 2003. Accessed July 3, 2019.
50.
Mestre-Ferrandiz  J, Sussex  J, Towse  A. The R&D cost of a new medicine. Monographs. https://www.ohe.org/publications/rd-cost-new-medicine. Published December 2012. Accessed July 3, 2019.
51.
O’Hagan  P, Farkas  C. Bringing pharma R&D back to health. Bain Brief. https://www.bain.com/insights/bringing-pharma-r-and-d-back-to-health/. Published March 30, 2009. Accessed July 3, 2019.
52.
Paul  SM, Mytelka  DS, Dunwiddie  CT,  et al.  How to improve R&D productivity: the pharmaceutical industry’s grand challenge.  Nat Rev Drug Discov. 2010;9(3):203-214. doi:10.1038/nrd3078PubMedGoogle ScholarCrossref
53.
Prasad  V, Mailankody  S.  Research and development spending to bring a single cancer drug to market and revenues after approval.  JAMA Intern Med. 2017;177(11):1569-1575. doi:10.1001/jamainternmed.2017.3601PubMedGoogle ScholarCrossref
54.
Baer  AR, Kelly  CA, Bruinooge  SS, Runowicz  CD, Blayney  DW.  Challenges to National Cancer Institute–supported Cooperative Group Clinical Trial participation: an ASCO survey of cooperative group sites.  J Oncol Pract. 2010;6(3):114-117. doi:10.1200/JOP.200028PubMedGoogle ScholarCrossref
55.
Nass  SJ, Balogh  E, Mendelsohn  J.  A National Cancer Clinical Trials Network: recommendations from the Institute of Medicine.  Am J Ther. 2011;18(5):382-391. doi:10.1097/MJT.0b013e3181ff7e23PubMedGoogle ScholarCrossref
56.
Peters  WP, Ross  M, Vredenburgh  JJ,  et al.  High-dose chemotherapy and autologous bone marrow support as consolidation after standard-dose adjuvant therapy for high-risk primary breast cancer.  J Clin Oncol. 1993;11(6):1132-1143. doi:10.1200/JCO.1993.11.6.1132PubMedGoogle ScholarCrossref
57.
Howard  DH, Kenline  C, Lazarus  HM,  et al.  Abandonment of high-dose chemotherapy/hematopoietic cell transplants for breast cancer following negative trial results.  Health Serv Res. 2011;46(6, pt 1):1762-1777. doi:10.1111/j.1475-6773.2011.01296.xPubMedGoogle ScholarCrossref
58.
Mello  MM, Brennan  TA.  The controversy over high-dose chemotherapy with autologous bone marrow transplant for breast cancer.  Health Aff (Millwood). 2001;20(5):101-117. doi:10.1377/hlthaff.20.5.101PubMedGoogle ScholarCrossref
59.
Rettig  RA, Jacobson  PD, Farquhar  CM,  et al.  False Hope: Bone Marrow Transplantation for Breast Cancer. New York, NY: Oxford University Press; 2007.
60.
Przepiorka  D, Deisseroth  A, Kane  R, Kaminskas  E, Farrell  AT, Pazdur  R.  Gemtuzumab ozogamicin.  J Clin Oncol. 2013;31(13):1699-1700. doi:10.1200/JCO.2012.48.1887PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 1,254
    Original Investigation
    Oncology
    September 4, 2019

    Association of National Cancer Institute–Sponsored Clinical Trial Network Group Studies With Guideline Care and New Drug Indications

    Author Affiliations
    • 1SWOG Statistics and Data Management Center, Seattle, Washington
    • 2Fred Hutchinson Cancer Research Center, Seattle, Washington
    • 3Department of Epidemiology, Mailman School of Public Health, Columbia University Medical Center, New York, New York
    • 4SWOG Group Chair’s Office, Knight Cancer Institute, Oregon Health & Science University, Portland
    JAMA Netw Open. 2019;2(9):e1910593. doi:10.1001/jamanetworkopen.2019.10593
    Key Points español 中文 (chinese)

    Question  What proportion of National Cancer Institute–sponsored, phase 3 Clinical Trial Network program studies are associated with guideline care or new drug indications?

    Findings  In this cohort study based on 182 trials including 148 028 patients, 82 trials (45.1%) were associated with guideline care or new drug indications, including trials with positive and negative findings. The estimated federal investment for each practice-influential trial was $16.6 million.

    Meaning  The National Cancer Institute’s Clinical Trial Network program contributes clinically meaningful, cost-efficient evidence to guide patient care.

    Abstract

    Importance  National Cancer Institute Clinical Trial Network (NCTN) groups serve a vital role in identifying effective new antineoplastic regimens. However, the downstream clinical effect of their trials has not been systematically examined.

    Objective  To examine the association of NCTN trials with guideline care and new drug indications.

    Design, Setting, and Participants  This retrospective cohort study evaluated phase 3 SWOG Cancer Research Network clinical trials from January 1, 1980, through June 30, 2017. Only completed trials with published results were included. To be considered practice influential (PI), a trial must have been associated with guideline care through its inclusion in National Comprehensive Cancer Network (NCCN) clinical guidelines or US Food and Drug Administration (FDA) new drug approvals in favor of a recommended treatment. Data were analyzed from June 15, 2018, through March 29, 2019.

    Main Outcomes and Measures  Estimated overall rate of PI trials, as well as trends over time. The total federal investment supporting the set of trials was also determined.

    Results  In total, 182 trials consisting of 148 028 patients were studied. Eighty-two studies (45.1%; 95% CI, 37.7%-52.6%) were PI, of which 70 (38.5%) influenced NCCN guidelines, 6 (3.3%) influenced FDA new drug approvals, and 6 (3.3%) influenced both. The number of PI trials was 47 of 65 (72.3%) among those with positive findings and 35 of 117 (29.9%) among those with negative findings. Thus, 35 of 82 PI trials (42.7%) were based on studies with negative findings, with nearly half of these studies (17 of 35 [48.6%]) reaffirming standard of care compared with experimental therapy. The total federal investment spent in conducting the trials was $1.36 billion (2017 US dollars), a rate of $7.5 million per study or $16.6 million per PI trial.

    Conclusions and Relevance  Nearly half of all phase 3 trials by one of the NCTN’s largest groups were associated with guideline care or new drug indications, including those with positive and negative findings. Compared with the costs of a new drug approval in pharmaceutical companies, typically estimated at more than $1 billion, the amount of federal funds invested to provide this valuable evidence was modest. These results suggest that the NCTN program contributes clinically meaningful, cost-efficient evidence to guide patient care.

    Introduction

    The National Cancer Institute (NCI) Clinical Trial Network (NCTN) groups serve a vital role in identifying new cancer treatments. Unlike pharmaceutical companies—whose primary aims are to develop new drugs and generate profits—the mandate of the NCTN is to serve the community of patients with cancer more broadly. Network group trials may compare different treatment regimens, combine treatments and treatment modalities, and/or examine whether 1 or more newly approved drugs may work in other cancers.1 Although the network groups have been in existence since the 1950s, an examination of the clinical effects of network group trials as a research process has, to our knowledge, not been conducted to date.2

    The mission of the NCTN groups is to “routinely achieve change in clinical practice” through the execution of well-conducted clinical trials.1(p9) The overall goal is to improve clinical outcomes for patients. Negative as well as positive findings of trials may influence practice; in particular, negative findings may inform health care professionals about which (potentially expensive) treatments do not work while also reducing costs and protecting patients from added treatment toxicity. Practice change may be best measured by the extent to which the results from clinical trials are used to support guideline care recommendations. We examined the association of network group trials with clinical practice guidelines and new drug approvals using study data from one of the largest NCTN groups.

    Methods
    Study Data

    Study data were generated by the SWOG (formerly the Southwest Oncology Group) Cancer Research Network, a member of the NCTN.3 We identified randomized phase 3 SWOG trials designed to test the current standard of care (SOC) compared with experimental therapy using SWOG’s study database web portal (https://www.swog.org). Evaluable studies were those for which the primary end point was reported in a scientific journal from January 1, 1980, through June 30, 2017. Only trials led by SWOG or another collaborating NCI network group (ECOG-ACRIN [formerly the Eastern Cooperative Oncology Group and American College of Radiology Imaging Network], Alliance for Clinical Trials in Oncology, NRG Oncology [formerly the National Surgical Adjuvant Breast and Bowel Project, Radiation Therapy Oncology Group, and Gynecologic Oncology Group], or other) with SWOG centers and/or investigators contributing to patient accrual were included. Information on ethical review and informed consent of participants for each of the trials was included in their primary study reports. Institutional review board approval of this study was not required because this analysis used no patient-level data. Data were evaluated according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.4

    Variable Definitions

    To be considered practice influential (PI), a trial must have been associated with guideline care through its inclusion in clinical care guidelines or US Food and Drug Administration (FDA) new drug approvals in favor of a recommended treatment. We required more than 1 year since study publication to evaluate whether a trial was PI. Clinical care guidelines and FDA new indications were allowed as measures of practice influence because they represent different constructs that may not overlap; in particular, a negative trial finding may influence guideline care without leading to an FDA new indication.5 Importantly, a PI trial can only be considered to be associated with the practice of guideline clinical care because health care professionals do not always follow guideline recommendations.6,7 The categorization of PI trials was made independently by 2 authors (V.T.N. and R.V.), with disagreement resolved by a third author (J.M.U.).

    A trial with positive findings was defined as one in which the experimental regimen achieved a statistically significant better result based on the study’s protocol-specified primary end point. All other studies—including those with negative or null findings—were categorized as negative. Trial designs were determined from protocol statistical sections or primary articles reporting the trial results. Categories of designs (superiority vs noninferiority vs equivalency) were determined using standard criteria.8

    Data Abstraction to Identify PI Trials

    The primary articles for each study—those reporting the results of the analysis of the primary protocol-specified end point by randomized arm—were identified through SWOG’s publication database or manually using PubMed or Google Scholar. Primary article title, study regimens (agent, dose, administration, and sequence), and primary author last name were abstracted to use as search indices; study characteristics were also obtained (Figure 1A). Missing data for any study characteristics were augmented using internal information sources, including SWOG’s clinical data management database, trial protocols, and the regularly updated SWOG formal Report of Studies.

    We used the National Comprehensive Cancer Network (NCCN) clinical practice guidelines from their inception in 1996 until 2019 to assess changes to guideline care (Figure 1B).9 The NCCN guidelines are comprehensive and continuously updated and apply to 97% of cancers affecting US patients.10 Each NCCN guideline version includes updates from prior versions, clinical charts representing primary treatment pathways and therapies, principles of treatment, and a discussion. We searched each cancer site-specific guideline using the search indices. If the trial provided evidence for the NCCN guideline recommendation (regardless of whether the finding was positive or negative) as cited in the clinical charts or principles of treatment sections, the trial was considered PI. For negative studies in particular, a PI trial was one for which the clinical charts or treatment sections indicated that the trial results reaffirmed SOC or, based on secondary evidence in the trial, suggested acceptable alternatives to SOC.

    We also examined whether a trial led to an FDA new drug indication (Figure 1C). We constructed a comprehensive catalog of FDA-approved anticancer drugs based on multiple publicly available sources, including the FDA portal.11,12 If the trial’s experimental agent was included in this catalog, we obtained package inserts for the agent using the Micromedex drug information portal.13 As a secondary examination, we input each agent name into the FDA portal and traced the package insert or the FDA approval documentation that discussed the pivotal trial.14 If the drug was cited as pivotal in the package inserts or the FDA approval documentation, the associated trial was deemed to have resulted in an FDA new drug approval and was categorized as PI.

    For trials not already classified as PI, we conducted a manual search using the Google search engine by using the combination of the first author’s last name and the title of the primary report, along with Food and Drug Administration, FDA, National Comprehensive Cancer Network, or NCCN as key words (Figure 1D). The same criteria were used to define a PI trial with respect to NCCN guidelines or FDA new drug approvals as noted earlier.

    Assessment of Costs

    To estimate the total costs of conducting the trials, we obtained NCTN funding data from the National Institutes of Health’s Research Portfolio Online Reporting Tools by fiscal year starting in 1985.15 We also included funding to support early-stage trials (phases 1 and 2) as well as grants for statistical centers and member institutions supporting the conduct of NCTN trials. Nontreatment trial, biospecimen, and specific database project grants were excluded. Funding estimates were inflated to constant 2017 US dollars based on the Consumer Price Index.16 Funding for years 1980 through 1984, which was not available, was assumed to contribute the same amount in 2017 dollars as the subsequent 5-year period from 1985 through 1989.

    Statistical Analysis

    Data were analyzed from June 15, 2018, to March 29, 2019. We reported the proportion of PI trials among all evaluable studies. Differences in rates of PI trials were compared between levels of study characteristics using χ2 tests (uncorrected) or Fisher exact tests where any cell counts were less than 5 for categorical variables and 2-sample t tests for continuous variables. Statistical significance was specified as 2-sided α ≤ .05. We used SAS, version 9.4 (SAS Institute Inc) and R, version 3.5.1 (R Project for Statistical Computing) to conduct the analyses.17

    Total funding to conduct the trials was calculated as the total estimated funding for each network group, weighted by the estimated proportion that the number of trials led by each group represented of the group’s entire phase 3 trial portfolio. For SWOG, the number of phase 3 trials in the entire portfolio was identified from the study database. For other network groups, the number of phase 3 trials conducted during the period was estimated from ClinicalTrials.gov. Mean funding per trial for studies led by other groups (IBCSG [International Breast Cancer Study Group], EORTC [European Organisation for Research and Treatment of Cancer], etc) was assumed to be the mean amount among trials led by SWOG, Alliance, ECOG-ACRIN, and NRG. Total costs per completed trial, per PI study, and per FDA new drug approval were calculated.

    Results
    Study Design Characteristics

    Two hundred forty-nine phase 3 clinical trials from the SWOG publication portal were identified as closed and published (Figure 2). One hundred eighty-two trials, including 148 028 patients, met the inclusion criteria. Ninety-four trials (51.6%) were led by SWOG and 88 (48.4%) by other groups. Accrual by group included 58 281 patients (39.4%) by SWOG, 22 028 (14.9%) by Alliance, 26 249 (17.7%) by ECOG-ACRIN, 12 051 (8.1%) by NRG, and 29 419 (19.9%) by other groups.

    One hundred thirty-one trials (72.0%) included intergroup participation (Table 1). The most commonly investigated cancers included breast (30 [16.5%]), genitourinary (26 [14.3%]), lung (23 [12.6%]), and gastrointestinal tract (22 [12.1%]) cancers. Superiority designs accounted for 176 trials (96.7%). Advanced disease trials accounted for 120 studies (65.9%). Nearly all trials included systemic therapy as one of the treatments on any arm (177 [97.3%]). Median total accrual was 463 patients per trial (range, 60-7576); mean (SD) total accrual, 814 (1087) patients per trial.

    Overall Rate of PI Trials

    Overall, 82 of 182 trials (45.1%; 95% CI, 37.7%-52.6%) were PI (eTable 1 in the Supplement). Seventy trials (38.5%) influenced NCCN guidelines only; 6 (3.3%), FDA indications only; and 6 (3.3%), both (Figure 3).

    Rates of PI trials differed among cancer types (P = .02), with lower rates observed among brain (0 of 7) and breast (7 of 30 [23.3%]) cancer trials and higher rates among head and neck (5 of 7 [71.4%]), genitourinary (16 of 26 [61.5%]), and gastrointestinal tract (13 of 22 [59.1%]) cancer trials (Table 1). Rates of PI trials were also higher among intergroup trials (68 of 131 [5.19%]; P = .003) and lower among blinded treatment trials (0 of 8; P = .009) and differed over time (P = .01), with the highest rate from 2000 to 2009 (33 of 55 [60.0%]).

    Fourteen trials were cited in NCCN guidelines but were determined not to be PI because they did not support the recommended treatment (eTable 2 in the Supplement). Two trials (S8216 and S0106) (eTable 1 in the Supplement) were identified as PI through the manual search using Google Scholar (Figure 1D).18,19

    Rate of PI Trials Among Positive and Negative Findings

    Among trials with positive findings, 47 of 65 (72.3%; 95% CI, 59.8%-82.7%) were PI, including 36 (55.4%) that influenced NCCN guidelines only, 5 (7.7%) that influenced FDA new drug indications only, and 6 (9.2%) that influenced both (Figure 3). Nine (4.9%) of the 11 trials (6.0%) that supported new FDA indications represented entirely new indications (rather than indications of previously approved regimens in other cancers) for single drugs (n = 7) or for drug combinations (n = 2) (eTable 3 in the Supplement).

    Among trials with negative findings, 35 of 117 (29.9%; 95% CI, 21.8%-39.1%) were PI. Thirty-four trials (29.1%) influenced NCCN guidelines and 1 (0.9%) influenced the FDA to remove a listed indication.20 Overall, 35 of 82 PI trials (42.7%) were based on negative findings. Nearly half the studies with negative findings (17 of 35 [48.6%]) reaffirmed SOC over experimental therapy (eTable 1 in the Supplement). Among the remaining studies with negative findings, 15 provided sufficient evidence to suggest acceptable alternatives for SOC, and 3 influenced guidelines for other reasons.

    Additional Analyses

    Early studies may have been less likely to be identified as PI because the NCCN guidelines only became available in 1996. However, among the 42 trials published before 1996, 16 trials were PI (38.1%); among the 140 trials published in 1996 or after, 66 were PI (47.1%; P = .39).

    An additional 5 trials were identified as potentially PI based on the manual Google search (Figure 1D and eTable 4 in the Supplement) but did not meet the formal criteria of inclusion within NCCN or FDA materials. With these studies included as PI, the overall rate of PI trials was 87 of 182 (47.8%; 95% CI, 40.4%-55.3%).

    The median number of additional studies cited as influential in NCCN guidelines or FDA package inserts—in addition to the SWOG study—was 1 (range, 0-14). The SWOG trial was the only cited study for 23 of the 82 PI trials (28.0%) and was 1 of 2 cited studies for 28 of the 82 PI trials (34.1%). The median number of additional studies cited did not differ between positive vs negative findings (median, 1; range, 0-14; P = .69).

    Cost per Trial

    Total federal investment supporting the set of analyzed trials was estimated to be $1.36 billion in 2017 dollars. Given 182 total trials, this finding suggests the mean costs were $7.5 million per completed phase 3 trial (PI or non-PI), $16.6 million per PI trial, and $123.6 million per FDA new drug approval.

    Discussion

    This study is the first, to our knowledge, to comprehensively examine the practice influence of NCTN-sponsored clinical trials for cancer using objective criteria. Nearly half (45.1%) of the trials we examined were found to be associated with guideline care through their inclusion in clinical care guidelines or FDA new drug indications, including many studies with negative findings, illustrating the value of well-conducted trials even when they do not demonstrate positive results. The investment cost for each PI trial was $16.6 million.

    A few studies in the United Kingdom used physician surveys to show that multicenter randomized clinical trials influence clinical practice, with the effect extending beyond the participating physicians.21,22 Other studies have examined the research underlying cancer clinical practice guidelines across various countries.23-25 We found no evidence in the literature of studies that explicitly estimated how often primary clinical trial results are associated with guideline care.

    The results in Table 1 are limited by small numbers but illuminate potentially informative associations. The high proportion of PI trials in head and neck cancers was due partly to 3 trials26-28 conducted in the 1990s showing that multimodality therapy (chemotherapy and radiotherapy) provided superior outcomes compared with radiotherapy alone, a strategy still commonly used today. This multimodality approach was also successful for trials across different cancer settings, in part accounting for the higher rate of PI trials from 2000 to 2009.29,30 The higher rate of PI trials in genitourinary cancers was associated with efforts to improve on androgen deprivation therapy for prostate cancer by including antiandrogrens,31 by modulating timing of androgen deprivation therapy administration,32 or more recently by adding other systemic therapy or radiotherapy.33,34 In bladder cancer, the discovery of bacillus Calmette-Guérin, a vaccine and early immunotherapeutic approach to the treatment of bladder cancer, was established in a series of trials showing its superiority over other SOCs.18,35,36 The lower rate of PI trials in breast cancer is somewhat misleading because 4 other trials identified in our supplemental search37-40 provided evidence against the use of transplant therapies for metastatic disease but did not meet our end point criteria.

    Trials conducted by intergroups rather than single network groups were more likely to be PI. The collaboration of multiple network groups on a trial could reflect the importance of the original research question and the increased potential for the results to guide practice. Future research establishing a prediction model based on modifiable factors that reliably predict PI trial findings would represent a powerful use of study databases of the kind used in this analysis to influence policy.41

    The success rate of pharmaceutical company phase 3 FDA indication studies has been examined extensively. DiMasi and colleagues42 used data from the largest 50 pharmaceutical companies for drug evaluations from 1993 through 2004 and found that about half of phase 3 trials were submitted for indication. Kola and Landis43 examined data from 10 large pharmaceutical companies from 1991 through 2000 and found the overall success rate from phase 3 trials through registration of about 28%. Hay and colleagues44 examined 835 drug developers from 2003 through 2011 to estimate an overall success probability from phase 3 trials in oncology of 37%. Together, approximately 35% of oncology drugs from phase 3 trials receive an FDA new drug approval in the pharmaceutical setting.45

    Overall, 6.0% of phase 3 trials in this study led to a new FDA indication, including 4.9% leading to entirely new indications. These rates are lower than those observed in pharmaceutical company studies. However, the objective of NCTN trials is rarely to register a new drug indication. As noted by the Institute of Medicine, the primary focus of pharmaceutical company–sponsored trials is “to develop novel therapeutic agents and gain FDA approval for clinical use,”1(p5) whereas the publicly funded NCTN clinical trials address research questions that are “important to patients but less likely to be top priorities of industry.”1(p1) Thus, the focus of the NCTN’s groups is placed on trials that compare the effectiveness of treatment options already in use, that combine novel therapies developed by different companies, that develop therapies for rare diseases, that examine different doses and durations of existing therapies, or that test therapeutic approaches that combine multiple treatment modalities.1 Therefore, the rate at which network group clinical trials result in new drug indications is expected to be lower.

    For pharmaceutical companies, the investment costs for each drug approval are high.42,45-53 One frequently cited statistic indicates a requirement of $1.1 billion (in 2017 US dollars).48 Many studies suggest the cost is even higher,42,45,47,49-52 especially for cancer drugs.46 Indeed, the mean cost estimate for a single new drug indication in 2017 dollars across 10 different studies42,45-53 we reviewed was $1.73 billion, more than 100 times greater than the federal investment cost per PI trial (Table 2). This comparison is imperfect because the pharmaceutical research process supports human trials and initial drug discovery, and the regulatory oversight for early-phase trials is more costly. Although our estimate of costs for NCTN trials also included those supporting early-stage (phases 1 and 2) trials, these trials are not often as closely tied to the eventual phase 3 trial and are not as numerous. Further, federal dollars invested in network group clinical trials frequently do not cover all of the capitation and institutional costs related to trial participation.54 For these reasons, network groups sometimes explicitly collaborate with industry to conduct trials, providing the design, conduct, and analysis of clinical trials, whereas industry may provide support for per-case costs at trial sites and the costs of nonstandard patient care and ancillary studies.55 Nonetheless, the observation highlights the value of the NCI’s network cancer research groups.

    One striking feature of our results is the number of negative trials found to influence guideline care. This factor is important because negative trials are generally perceived as failures. However, the true underlying objective of a well-conducted trial is to reduce uncertainty about the efficacy of a new treatment rather than to achieve any particular outcome. This notion is important given the history of exciting new therapeutic approaches found to be ineffective or outright harmful when tested in comparative clinical trials. In the 1990s, autologous bone marrow transplants for breast cancer were thought to represent an important alternative and potentially curable avenue for women with metastatic disease, supported by the exciting results of early-stage, noncontrolled trials.56 However, multiple randomized phase 3 trials—conducted even as transplants were already in use for many patients with breast cancer—found that transplants were expensive, resulted in consistently more treatment fatalities vs conventional chemotherapy, and showed no meaningful evidence of clinical benefit.37-40 Since the publication of these studies, transplant therapy in this setting has been largely abandoned.57-59 In another example, after the FDA approved gemtuzumab ozogamicin through the accelerated approval program for patients with acute myeloid leukemia, the confirmatory trial S0106 demonstrated increased deaths and no advantage in survival end points associated with this agent, leading to a voluntary postmarket withdrawal of gemtuzumab in 2010.19,60 Recently, this agent was newly approved by the FDA but only at a lower dose, a different schedule, and in the targeted setting of cases expressing the CD33 antigen.20

    Limitations

    Our study was limited to the evaluation of trials directed or contributed to by a single large network group. This criterion could limit the generalizability of the overall estimate of PI trials, although importantly, other network groups led or contributed to almost 3 of every 4 trials. Also, our estimate may be biased high if the rate of intergroup trials was not representative because intergroup trials were more likely to influence practice. Further, although our determination of PI trials relied in part on NCCN guidelines, which are widely used by payers and health care professionals, other compendia may have generated a different set of estimates. Moreover, the identification of a PI trial may be subject to the frequency with which NCCN guidelines are updated for selected cancers, especially more recent trials and those conducted in less common cancers. Although in some instances a SWOG PI trial was solely responsible for influencing guideline care, in others, it was only 1 of multiple studies, suggesting contributions from trials not included in our study sample.

    Conclusions

    Our findings show that nearly half of all randomized phase 3 trials conducted by one of the NCI’s large network clinical trial groups were associated with guideline care and new drug approvals. This rate exceeds the rate of phase 3 trials with positive findings (generally estimated to be 25%-30%) owing to the fact that many trials with negative findings—which are often considered scientific failures—also influence guideline care, especially by identifying what treatments should not be used. Thus, the true value of a well-conducted, controlled randomized clinical trial for cancer treatment is in guiding the cancer clinical treatment community about the best treatment choice regardless of whether the best choice is a new, trial-proven experimental therapy or the existing SOC. Moreover, NCTN trials also benefit research through data sharing and secondary analyses using the clinical study databases, which provide important insights into current treatments and hypotheses for future trials. Finally, the amount invested by federal funders to provide this valuable evidence was modest. The NCTN program contributes clinically meaningful, cost-efficient evidence to guide patient care.

    Back to top
    Article Information

    Accepted for Publication: July 14, 2019.

    Published: September 4, 2019. doi:10.1001/jamanetworkopen.2019.10593

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Unger JM et al. JAMA Network Open.

    Corresponding Author: Joseph M. Unger, PhD, MS, SWOG Statistics and Data Management Center, Fred Hutchinson Cancer Research Center, Mail Stop M3-C102, 1100 Fairview Ave, Seattle, WA 98109 (junger@fredhutch.org).

    Author Contributions: Dr Unger had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Unger, Nghiem, Hershman, Blanke.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Unger, Nghiem, Hershman, Vaidya, Blanke.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Unger, Nghiem, Vaidya, Blanke.

    Obtained funding: Unger, Blanke.

    Administrative, technical, or material support: Unger, Nghiem, LeBlanc, Blanke.

    Supervision: Unger, Blanke.

    Conflict of Interest Disclosures: Dr Unger reported receiving a grant from the Hope Foundation for Cancer Research related to this research. Dr Blanke reported receiving grants from the National Cancer Institute (NCI) of the National Institutes of Health (NIH) related to this research. Dr LeBlanc reported receiving grants from the National Cancer Institute (NCI) of the National Institutes of Health (NIH) related to this research. No other conflicts were reported.

    Funding/Support: This study was supported by grants U10CA180888 (Dr Blanke), U10CA180819 (Dr LeBlanc), and UG1CA189974 (Drs Blanke and Hershman) from the NCI of the NIH and in part by the Hope Foundation for Cancer Research (Dr Unger).

    Role of the Funder/Sponsor: The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    References
    1.
    Institute of Medicine. National Cancer Clinical Trials System for the 21st Century: Reinvigorating the NCI Cooperative Group Program. Washington, DC: National Academies Press; 2010.
    2.
    National Cancer Institute. Cancer Moonshot Blue Ribbon Panel. https://www.cancer.gov/research/key-initiatives/moonshot-cancer-initiative/blue-ribbon-panel. Updated April 26, 2017. Accessed July 3, 2019.
    3.
    SWOG Cancer Research Network. History and impact. https://www.swog.org/about/history-impact. Accessed July 3, 2019.
    4.
    von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.  Lancet. 2007;370(9596):1453-1457. doi:10.1016/S0140-6736(07)61602-XPubMedGoogle ScholarCrossref
    5.
    Wagner  J, Marquart  J, Ruby  J,  et al.  Frequency and level of evidence used in recommendations by the National Comprehensive Cancer Network guidelines beyond approvals of the US Food and Drug Administration: retrospective observational study.  BMJ. 2018;360:k668. doi:10.1136/bmj.k668PubMedGoogle ScholarCrossref
    6.
    Cabana  MD, Rand  CS, Powe  NR,  et al.  Why don’t physicians follow clinical practice guidelines? a framework for improvement.  JAMA. 1999;282(15):1458-1465. doi:10.1001/jama.282.15.1458PubMedGoogle ScholarCrossref
    7.
    Francke  AL, Smit  MC, de Veer  AJE, Mistiaen  P.  Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review.  BMC Med Inform Decis Mak. 2008;8:38. doi:10.1186/1472-6947-8-38PubMedGoogle ScholarCrossref
    8.
    Lesaffre  E.  Superiority, equivalence, and non-inferiority trials.  Bull NYU Hosp Jt Dis. 2008;66(2):150-154.PubMedGoogle Scholar
    9.
    National Comprehensive Cancer Network. About the NCCN clinical practice guidelines in oncology. https://www.nccn.org/professionals/. Accessed April 3, 2019.
    10.
    National Comprehensive Cancer Network. Development and update of the NCCN guidelines. https://www.nccn.org/professionals/development.aspx. Accessed April 3, 2019.
    11.
    National Cancer Institute. A to Z list of cancer drugs. https://www.cancer.gov/about-cancer/treatment/drugs. Accessed April 3, 2019.
    12.
    Sun  J, Wei  Q, Zhou  Y, Wang  J, Liu  Q, Xu  H.  A systematic analysis of FDA-approved anticancer drugs.  BMC Syst Biol. 2017;11(suppl 5):87. doi:10.1186/s12918-017-0464-7PubMedGoogle ScholarCrossref
    13.
    IBM, Inc. IBM Micromedex web applications access. http://www.micromedexsolutions.com/home/dispatch. Updated 2019. Accessed October 1, 2018.
    14.
    US Food and Drug Administration. FDA approved drug products. https://www.accessdata.fda.gov/scripts/cder/daf/. Accessed October 1, 2018.
    15.
    National Institutes of Health. Research Portfolio Online Reporting Tools (RePORT). https://projectreporter.nih.gov/reporter.cfm. Updated July 20, 2019. Accessed October 16, 2018.
    16.
    US Department of Labor Statistics. Consumer Price Index US city average (1982-1984 = 100). https://www.bls.gov/regions/new-england/data/consumerpriceindex_us_table.htm. Updated 2019. Accessed December 17, 2018.
    17.
    R: A Language and Environment for Statistical Computing. Vienna, Austria: R Core Team; 2017. https://www.R-project.org/. Accessed October 4, 2017.
    18.
    Lamm  DL, Blumenstein  BA, Crawford  ED,  et al.  A randomized trial of intravesical doxorubicin and immunotherapy with bacille Calmette-Guérin for transitional-cell carcinoma of the bladder.  N Engl J Med. 1991;325(17):1205-1209. doi:10.1056/NEJM199110243251703PubMedGoogle ScholarCrossref
    19.
    Petersdorf  SH, Kopecky  KJ, Slovak  M,  et al.  A phase 3 study of gemtuzumab ozogamicin during induction and postconsolidation therapy in younger patients with acute myeloid leukemia.  Blood. 2013;121(24):4854-4860. doi:10.1182/blood-2013-01-466706PubMedGoogle ScholarCrossref
    20.
    U.S. Food and Drug Administration. FDA news release: FDA approves Mylotarg for treatment of acute myeloid leukemia. https://www.fda.gov/newsevents/newsroom/pressannouncements/ucm574507.htm. Released September 1, 2017. Accessed February 21, 2019.
    21.
    Benghiat  A.  The influence of clinical trials in the treatment of cancer: results of a survey.  Br J Radiol. 1985;58(690):572-573. doi:10.1259/0007-1285-58-690-572PubMedGoogle ScholarCrossref
    22.
    Stephens  R, Gibson  D.  The impact of clinical trials on the treatment of lung cancer.  Clin Oncol (R Coll Radiol). 1993;5(4):211-219. doi:10.1016/S0936-6555(05)80231-6PubMedGoogle ScholarCrossref
    23.
    Lewison  G, Sullivan  R.  The impact of cancer research: how publications influence UK cancer clinical guidelines.  Br J Cancer. 2008;98(12):1944-1950. doi:10.1038/sj.bjc.6604405PubMedGoogle ScholarCrossref
    24.
    Pallari  E, Fox  AW, Lewison  G.  Differential research impact in cancer practice guidelines’ evidence base: lessons from ESMO, NICE and SIGN.  ESMO Open. 2018;3(1):e000258. doi:10.1136/esmoopen-2017-000258PubMedGoogle ScholarCrossref
    25.
    Poonacha  TK, Go  RS.  Level of scientific evidence underlying recommendations arising from the National Comprehensive Cancer Network clinical practice guidelines.  J Clin Oncol. 2011;29(2):186-191. doi:10.1200/JCO.2010.31.6414PubMedGoogle ScholarCrossref
    26.
    Al-Sarraf  M, LeBlanc  M, Giri  PG,  et al.  Chemoradiotherapy versus radiotherapy in patients with advanced nasopharyngeal cancer: phase III randomized intergroup study 0099.  J Clin Oncol. 1998;16(4):1310-1317. doi:10.1200/JCO.1998.16.4.1310PubMedGoogle ScholarCrossref
    27.
    Adelstein  DJ, Li  Y, Adams  GL,  et al.  An intergroup phase III comparison of standard radiation therapy and two schedules of concurrent chemoradiotherapy in patients with unresectable squamous cell head and neck cancer.  J Clin Oncol. 2003;21(1):92-98. doi:10.1200/JCO.2003.01.008PubMedGoogle ScholarCrossref
    28.
    Cooper  JS, Pajak  TF, Forastiere  AA,  et al; Radiation Therapy Oncology Group 9501/Intergroup.  Postoperative concurrent radiotherapy and chemotherapy for high-risk squamous-cell carcinoma of the head and neck.  N Engl J Med. 2004;350(19):1937-1944. doi:10.1056/NEJMoa032646PubMedGoogle ScholarCrossref
    29.
    Herskovic  A, Martz  K, al-Sarraf  M,  et al.  Combined chemotherapy and radiotherapy compared with radiotherapy alone in patients with cancer of the esophagus.  N Engl J Med. 1992;326(24):1593-1598. doi:10.1056/NEJM199206113262403PubMedGoogle ScholarCrossref
    30.
    Macdonald  JS, Smalley  SR, Benedetti  J,  et al.  Chemoradiotherapy after surgery compared with surgery alone for adenocarcinoma of the stomach or gastroesophageal junction.  N Engl J Med. 2001;345(10):725-730. doi:10.1056/NEJMoa010187PubMedGoogle ScholarCrossref
    31.
    Crawford  ED, Eisenberger  MA, McLeod  DG,  et al.  A controlled trial of leuprolide with and without flutamide in prostatic carcinoma.  N Engl J Med. 1989;321(7):419-424. doi:10.1056/NEJM198908173210702PubMedGoogle ScholarCrossref
    32.
    Messing  EM, Manola  J, Sarosdy  M, Wilding  G, Crawford  ED, Trump  D.  Immediate hormonal therapy compared with observation after radical prostatectomy and pelvic lymphadenectomy in men with node-positive prostate cancer.  N Engl J Med. 1999;341(24):1781-1788. doi:10.1056/NEJM199912093412401PubMedGoogle ScholarCrossref
    33.
    Sweeney  CJ, Chen  YH, Carducci  M,  et al.  Chemohormonal therapy in metastatic hormone-sensitive prostate cancer.  N Engl J Med. 2015;373(8):737-746. doi:10.1056/NEJMoa1503747PubMedGoogle ScholarCrossref
    34.
    Warde  P, Mason  M, Ding  K,  et al; NCIC CTG PR.3/MRC UK PR07 Investigators.  Combined androgen deprivation therapy and radiation therapy for locally advanced prostate cancer: a randomised, phase 3 trial.  Lancet. 2011;378(9809):2104-2111. doi:10.1016/S0140-6736(11)61095-7PubMedGoogle ScholarCrossref
    35.
    Lamm  DL, Blumenstein  BA, David Crawford  E,  et al.  Randomized intergroup comparison of bacillus Calmette-Guerin immunotherapy and mitomycin C chemotherapy prophylaxis in superficial transitional cell carcinoma of the bladder: a Southwest Oncology Group Study.  Urol Oncol. 1995;1(3):119-126. doi:10.1016/1078-1439(95)00041-FPubMedGoogle ScholarCrossref
    36.
    Lamm  DL, Blumenstein  BA, Crissman  JD,  et al.  Maintenance bacillus Calmette-Guerin immunotherapy for recurrent TA, T1 and carcinoma in situ transitional cell carcinoma of the bladder: a randomized Southwest Oncology Group Study.  J Urol. 2000;163(4):1124-1129. doi:10.1016/S0022-5347(05)67707-5PubMedGoogle ScholarCrossref
    37.
    Moore  HC, Green  SJ, Gralow  JR,  et al; Southwest Oncology Group/Intergroup Study 9623.  Intensive dose-dense compared with high-dose adjuvant chemotherapy for high-risk operable breast cancer: Southwest Oncology Group/Intergroup Study 9623.  J Clin Oncol. 2007;25(13):1677-1682. doi:10.1200/JCO.2006.08.9383PubMedGoogle ScholarCrossref
    38.
    Peters  WP, Rosner  GL, Vredenburgh  JJ,  et al.  Prospective, randomized comparison of high-dose chemotherapy with stem-cell support versus intermediate-dose chemotherapy after surgery and adjuvant chemotherapy in women with high-risk primary breast cancer: a report of CALGB 9082, SWOG 9114, and NCIC MA-13.  J Clin Oncol. 2005;23(10):2191-2200. doi:10.1200/JCO.2005.10.202PubMedGoogle ScholarCrossref
    39.
    Stadtmauer  EA, O’Neill  A, Goldstein  LJ,  et al; Philadelphia Bone Marrow Transplant Group.  Conventional-dose chemotherapy compared with high-dose chemotherapy plus autologous hematopoietic stem-cell transplantation for metastatic breast cancer.  N Engl J Med. 2000;342(15):1069-1076. doi:10.1056/NEJM200004133421501PubMedGoogle ScholarCrossref
    40.
    Tallman  MS, Gray  R, Robert  NJ,  et al.  Conventional adjuvant chemotherapy with or without high-dose chemotherapy and autologous stem-cell transplantation in high-risk breast cancer.  N Engl J Med. 2003;349(1):17-26. doi:10.1056/NEJMoa030684PubMedGoogle ScholarCrossref
    41.
    Wong  CH, Siah  KW, Lo  AW.  Estimation of clinical trial success rates and related parameters.  Biostatistics. 2019;20(2):273-286. doi:10.1093/biostatistics/kxx069PubMedGoogle ScholarCrossref
    42.
    DiMasi  JA, Feldman  L, Seckler  A, Wilson  A.  Trends in risks associated with new drug development: success rates for investigational drugs.  Clin Pharmacol Ther. 2010;87(3):272-277. doi:10.1038/clpt.2009.295PubMedGoogle ScholarCrossref
    43.
    Kola  I, Landis  J.  Can the pharmaceutical industry reduce attrition rates?  Nat Rev Drug Discov. 2004;3(8):711-715. doi:10.1038/nrd1470PubMedGoogle ScholarCrossref
    44.
    Hay  M, Thomas  DW, Craighead  JL, Economides  C, Rosenthal  J.  Clinical development success rates for investigational drugs.  Nat Biotechnol. 2014;32(1):40-51. doi:10.1038/nbt.2786PubMedGoogle ScholarCrossref
    45.
    DiMasi  JA, Grabowski  HG, Hansen  RW.  Innovation in the pharmaceutical industry: new estimates of R&D costs.  J Health Econ. 2016;47:20-33. doi:10.1016/j.jhealeco.2016.01.012PubMedGoogle ScholarCrossref
    46.
    Adams  CP, Brantner  VV.  Estimating the cost of new drug development: is it really 802 million dollars?  Health Aff (Millwood). 2006;25(2):420-428. doi:10.1377/hlthaff.25.2.420PubMedGoogle ScholarCrossref
    47.
    Adams  CP, Brantner  VV.  Spending on new drug development.  Health Econ. 2010;19(2):130-141. doi:10.1002/hec.1454PubMedGoogle ScholarCrossref
    48.
    DiMasi  JA, Hansen  RW, Grabowski  HG.  The price of innovation: new estimates of drug development costs.  J Health Econ. 2003;22(2):151-185. doi:10.1016/S0167-6296(02)00126-1PubMedGoogle ScholarCrossref
    49.
    Gilbert  J, Preston  H, Singh  A. Rebuilding big pharma's business model in vivo. Business Med Rep 21. https://invivo.pharmaintelligence.informa.com/IV002027/Rebuilding-Big-Pharmas-Business-Model. Published November 1, 2003. Accessed July 3, 2019.
    50.
    Mestre-Ferrandiz  J, Sussex  J, Towse  A. The R&D cost of a new medicine. Monographs. https://www.ohe.org/publications/rd-cost-new-medicine. Published December 2012. Accessed July 3, 2019.
    51.
    O’Hagan  P, Farkas  C. Bringing pharma R&D back to health. Bain Brief. https://www.bain.com/insights/bringing-pharma-r-and-d-back-to-health/. Published March 30, 2009. Accessed July 3, 2019.
    52.
    Paul  SM, Mytelka  DS, Dunwiddie  CT,  et al.  How to improve R&D productivity: the pharmaceutical industry’s grand challenge.  Nat Rev Drug Discov. 2010;9(3):203-214. doi:10.1038/nrd3078PubMedGoogle ScholarCrossref
    53.
    Prasad  V, Mailankody  S.  Research and development spending to bring a single cancer drug to market and revenues after approval.  JAMA Intern Med. 2017;177(11):1569-1575. doi:10.1001/jamainternmed.2017.3601PubMedGoogle ScholarCrossref
    54.
    Baer  AR, Kelly  CA, Bruinooge  SS, Runowicz  CD, Blayney  DW.  Challenges to National Cancer Institute–supported Cooperative Group Clinical Trial participation: an ASCO survey of cooperative group sites.  J Oncol Pract. 2010;6(3):114-117. doi:10.1200/JOP.200028PubMedGoogle ScholarCrossref
    55.
    Nass  SJ, Balogh  E, Mendelsohn  J.  A National Cancer Clinical Trials Network: recommendations from the Institute of Medicine.  Am J Ther. 2011;18(5):382-391. doi:10.1097/MJT.0b013e3181ff7e23PubMedGoogle ScholarCrossref
    56.
    Peters  WP, Ross  M, Vredenburgh  JJ,  et al.  High-dose chemotherapy and autologous bone marrow support as consolidation after standard-dose adjuvant therapy for high-risk primary breast cancer.  J Clin Oncol. 1993;11(6):1132-1143. doi:10.1200/JCO.1993.11.6.1132PubMedGoogle ScholarCrossref
    57.
    Howard  DH, Kenline  C, Lazarus  HM,  et al.  Abandonment of high-dose chemotherapy/hematopoietic cell transplants for breast cancer following negative trial results.  Health Serv Res. 2011;46(6, pt 1):1762-1777. doi:10.1111/j.1475-6773.2011.01296.xPubMedGoogle ScholarCrossref
    58.
    Mello  MM, Brennan  TA.  The controversy over high-dose chemotherapy with autologous bone marrow transplant for breast cancer.  Health Aff (Millwood). 2001;20(5):101-117. doi:10.1377/hlthaff.20.5.101PubMedGoogle ScholarCrossref
    59.
    Rettig  RA, Jacobson  PD, Farquhar  CM,  et al.  False Hope: Bone Marrow Transplantation for Breast Cancer. New York, NY: Oxford University Press; 2007.
    60.
    Przepiorka  D, Deisseroth  A, Kane  R, Kaminskas  E, Farrell  AT, Pazdur  R.  Gemtuzumab ozogamicin.  J Clin Oncol. 2013;31(13):1699-1700. doi:10.1200/JCO.2012.48.1887PubMedGoogle ScholarCrossref
    ×