[Skip to Content]
[Skip to Content Landing]
Figure.
Unadjusted Rates of Factors Associated With Regulatory Approval by the FDA
Unadjusted Rates of Factors Associated With Regulatory Approval by the FDA

Error bars represent 95% CIs. SMC indicates small or medium-sized company.

Table 1.  
Characteristics of Novel Drugs and Biologics Entering Pivotal Trials, 1998-2008
Characteristics of Novel Drugs and Biologics Entering Pivotal Trials, 1998-2008
Table 2.  
Reasons for Failure of Late-Stage Clinical Development of Experimental Agents, Stratified by Agent Characteristics
Reasons for Failure of Late-Stage Clinical Development of Experimental Agents, Stratified by Agent Characteristics
Table 3.  
Results From Multivariable Logistic Regression Models of Regulatory Approval
Results From Multivariable Logistic Regression Models of Regulatory Approval
Table 4.  
Rates of Publication of Trial Results, by Reason for Failure of Late-Stage Developmenta
Rates of Publication of Trial Results, by Reason for Failure of Late-Stage Developmenta
1.
Institute of Medicine, Forum on Drug Discovery, Development, and Translation. Transforming Clinical Research in the United States: Challenges and Opportunities. Washington, DC: National Academies Press; 2010.
2.
Barter  PJ, Caulfield  M, Eriksson  M,  et al; ILLUMINATE Investigators.  Effects of torcetrapib in patients at high risk for coronary events.  N Engl J Med. 2007;357(21):2109-2122.PubMedGoogle ScholarCrossref
3.
Doody  RS, Raman  R, Farlow  M,  et al; Alzheimer’s Disease Cooperative Study Steering Committee; Semagacestat Study Group.  A phase 3 trial of semagacestat for treatment of Alzheimer’s disease.  N Engl J Med. 2013;369(4):341-350.PubMedGoogle ScholarCrossref
4.
Amiri-Kordestani  L, Fojo  T.  Why do phase III clinical trials in oncology fail so often?  J Natl Cancer Inst. 2012;104(8):568-569.PubMedGoogle ScholarCrossref
5.
Lurie  P, Chahal  HS, Sigelman  DW, Stacy  S, Sclar  J, Ddamulira  B.  Comparison of content of FDA letters not approving applications for new drugs and associated public announcements from sponsors: cross sectional study.  BMJ. 2015;350:h2758.PubMedGoogle ScholarCrossref
6.
Hakala  A, Kimmelman  J, Carlisle  B, Freeman  G, Fergusson  D.  Accessibility of trial reports for drugs stalling in development: a systematic assessment of registered trials.  BMJ. 2015;350:h1116.PubMedGoogle ScholarCrossref
7.
Sacks  LV, Shamsuddin  HH, Yasinskaya  YI, Bouri  K, Lanthier  ML, Sherman  RE.  Scientific and regulatory reasons for delay and denial of FDA approval of initial applications for new drugs, 2000-2012.  JAMA. 2014;311(4):378-384.PubMedGoogle ScholarCrossref
8.
Ross  JS, Dzara  K, Downing  NS.  Efficacy and safety concerns are important reasons why the FDA requires multiple reviews before approval of new drugs.  Health Aff (Millwood). 2015;34(4):681-688.PubMedGoogle ScholarCrossref
9.
Hwang  TJ, Carpenter  D, Kesselheim  AS.  Target small firms for antibiotic innovation.  Science. 2014;344(6187):967-969.PubMedGoogle ScholarCrossref
10.
Carpenter  D, Zucker  EJ, Avorn  J.  Drug-review deadlines and safety problems.  N Engl J Med. 2008;358(13):1354-1361.PubMedGoogle ScholarCrossref
11.
Lincker  H, Ziogas  C, Carr  M, Porta  N, Eichler  HG.  Regulatory watch: where do new medicines originate from in the EU?  Nat Rev Drug Discov. 2014;13(2):92-93.PubMedGoogle ScholarCrossref
12.
Adams  CP, Brantner  VV.  Estimating the cost of new drug development: is it really 802 million dollars?  Health Aff (Millwood). 2006;25(2):420-428.PubMedGoogle ScholarCrossref
13.
Kesselheim  AS, Hwang  TJ, Franklin  JM.  Two decades of new drug development for central nervous system disorders.  Nat Rev Drug Discov. 2015;14(12):815-816.PubMedGoogle ScholarCrossref
14.
Hamburg  MA. Why FDA Supports a Flexible Approach to Drug Development. FDA website. http://blogs.fda.gov/fdavoice/index.php/2014/02/why-fda-supports-a-flexible-approach-to-drug-development/ Accessed February 22, 2016.
15.
Lanthier  M, Miller  KL, Nardinelli  C, Woodcock  J.  An improved approach to measuring drug innovation finds steady rates of first-in-class pharmaceuticals, 1987-2011.  Health Aff (Millwood). 2013;32(8):1433-1439.PubMedGoogle ScholarCrossref
16.
Kesselheim  AS, Wang  B, Franklin  JM, Darrow  JJ.  Trends in utilization of FDA expedited drug development and approval programs, 1987-2014: cohort study.  BMJ. 2015;351:h4633.PubMedGoogle ScholarCrossref
17.
Eder  J, Sedrani  R, Wiesmann  C.  The discovery of first-in-class drugs: origins and evolution.  Nat Rev Drug Discov. 2014;13(8):577-587.PubMedGoogle ScholarCrossref
18.
Hwang  TJ, Lauffenburger  JC, Franklin  JM, Kesselheim  AS.  Temporal trends and factors associated with cardiovascular drug development, 1990 to 2012  [published online August 29, 2016].  J Am Coll Cardiol Basic Trans Science. doi:10.1016/j.jacbts.2016.03.012Google Scholar
19.
Abola  MV, Prasad  V.  The Use of Superlatives in Cancer Research.  JAMA Oncol. 2016;2(1):139-141.PubMedGoogle ScholarCrossref
20.
Djulbegovic  B, Kumar  A, Glasziou  PP,  et al.  New treatments compared to established treatments in randomized trials.  Cochrane Database Syst Rev. 2012;10:MR000024.PubMedGoogle Scholar
21.
O’Day  S, Gonzalez  R, Lawson  D,  et al.  Phase II, randomized, controlled, double-blinded trial of weekly elesclomol plus paclitaxel versus paclitaxel alone for stage IV metastatic melanoma.  J Clin Oncol. 2009;27(32):5452-5458.PubMedGoogle ScholarCrossref
22.
O’Day  SJ, Eggermont  AM, Chiarion-Sileni  V,  et al.  Final results of phase III SYMMETRY study: randomized, double-blind trial of elesclomol plus paclitaxel versus paclitaxel alone as treatment for chemotherapy-naive patients with advanced melanoma.  J Clin Oncol. 2013;31(9):1211-1218.PubMedGoogle ScholarCrossref
23.
Kesselheim  AS, Myers  JA, Avorn  J.  Characteristics of clinical trials to support approval of orphan vs nonorphan drugs for cancer.  JAMA. 2011;305(22):2320-2326.PubMedGoogle ScholarCrossref
24.
Mitsumoto  J, Dorsey  ER, Beck  CA, Kieburtz  K, Griggs  RC.  Pivotal studies of orphan drugs approved for neurological diseases.  Ann Neurol. 2009;66(2):184-190.PubMedGoogle ScholarCrossref
25.
Button  KS, Ioannidis  JP, Mokrysz  C,  et al.  Power failure: why small sample size undermines the reliability of neuroscience.  Nat Rev Neurosci. 2013;14(5):365-376.PubMedGoogle ScholarCrossref
26.
Lee  K, Bacchetti  P, Sim  I.  Publication of clinical trials supporting successful new drug applications: a literature analysis.  PLoS Med. 2008;5(9):e191.PubMedGoogle ScholarCrossref
27.
Rising  K, Bacchetti  P, Bero  L.  Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation.  PLoS Med. 2008;5(11):e217.PubMedGoogle ScholarCrossref
28.
Smithy  JW, Downing  NS, Ross  JS.  Publication of pivotal efficacy trials for novel therapeutic agents approved between 2005 and 2011: a cross-sectional study.  JAMA Intern Med. 2014;174(9):1518-1520.PubMedGoogle ScholarCrossref
29.
Turner  EH, Matthews  AM, Linardatos  E, Tell  RA, Rosenthal  R.  Selective publication of antidepressant trials and its influence on apparent efficacy.  N Engl J Med. 2008;358(3):252-260.PubMedGoogle ScholarCrossref
30.
Hart  B, Lundh  A, Bero  L.  Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses.  BMJ. 2012;344:d7202.PubMedGoogle ScholarCrossref
31.
Becker  RE, Greig  NH.  Lost in translation: neuropsychiatric drug development.  Sci Transl Med. 2010;2(61):61rv6.PubMedGoogle ScholarCrossref
32.
van der Worp  HB, Howells  DW, Sena  ES,  et al.  Can animal models of disease reliably inform human studies?  PLoS Med. 2010;7(3):e1000245.PubMedGoogle ScholarCrossref
33.
O’Collins  VE, Macleod  MR, Donnan  GA, Horky  LL, van der Worp  BH, Howells  DW.  1,026 experimental treatments in acute stroke.  Ann Neurol. 2006;59(3):467-477.PubMedGoogle ScholarCrossref
34.
Chalmers  I.  Underreporting research is scientific misconduct.  JAMA. 1990;263(10):1405-1408.PubMedGoogle ScholarCrossref
35.
Seruga  B, Templeton  AJ, Badillo  FE, Ocana  A, Amir  E, Tannock  IF.  Under-reporting of harm in clinical trials.  Lancet Oncol. 2016;17(5):e209-e219.PubMedGoogle ScholarCrossref
36.
Singhal  S, Mehta  J, Desikan  R,  et al.  Antitumor activity of thalidomide in refractory multiple myeloma.  N Engl J Med. 1999;341(21):1565-1571.PubMedGoogle ScholarCrossref
37.
Hudson  KL, Collins  FS.  Sharing and reporting the results of clinical trials.  JAMA. 2015;313(4):355-356.PubMedGoogle ScholarCrossref
38.
Federal Register. National Institutes of Health: Notice of Proposed Rulemaking on Clinical Trials Registration and Results Submission. https://federalregister.gov/a/2014-26197. Accessed August 11, 2016.
39.
Arrowsmith  J, Miller  P.  Trial watch: phase II and phase III attrition rates 2011-2012.  Nat Rev Drug Discov. 2013;12(8):569.PubMedGoogle ScholarCrossref
40.
Waring  MJ, Arrowsmith  J, Leach  AR,  et al.  An analysis of the attrition of drug candidates from four major pharmaceutical companies.  Nat Rev Drug Discov. 2015;14(7):475-486.PubMedGoogle ScholarCrossref
41.
Hay  M, Thomas  DW, Craighead  JL, Economides  C, Rosenthal  J.  Clinical development success rates for investigational drugs.  Nat Biotechnol. 2014;32(1):40-51.PubMedGoogle ScholarCrossref
42.
Zuckerman  DM, Jury  NJ, Silcox  CE.  21st Century Cures Act and similar policy efforts: at what cost?  BMJ. 2015;351:h6122.PubMedGoogle ScholarCrossref
43.
Redberg  RF.  Faster drug approvals are not always better and can be worse.  JAMA Intern Med. 2015;175(8):1398.PubMedGoogle ScholarCrossref
44.
Gonsalves  G, Zuckerman  D.  Commentary: Will 20th century patient safeguards be reversed in the 21st century?  BMJ. 2015;350:h1500.PubMedGoogle ScholarCrossref
45.
Floyd  JS, Psaty  BM.  The potential risks of expedited approval of drugs for acute bacterial infections.  JAMA Intern Med. 2014;174(9):1436-1437.PubMedGoogle ScholarCrossref
Original Investigation
December 2016

Failure of Investigational Drugs in Late-Stage Clinical Development and Publication of Trial Results

Author Affiliations
  • 1Program on Regulation, Therapeutics, and Law (PORTAL), Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, Massachusetts
  • 2Radcliffe Institute for Advanced Study and Faculty of Arts and Sciences, Harvard University, Cambridge, Massachusetts
 

Copyright 2016 American Medical Association. All Rights Reserved.

JAMA Intern Med. 2016;176(12):1826-1833. doi:10.1001/jamainternmed.2016.6008
Key Points

Question  Why and how often do experimental drugs fail in phase 3 clinical trials, and how often are trial results published?

Findings  Using public sources and commercial databases covering drugs and biologics that started trials between 1998 and 2008, 54% of agents carried into pivotal trials failed, primarily owing to inadequate efficacy or safety concerns. Trial results were published for 40% of these failed agents.

Meaning  Although many drugs fail in late-stage trials, the rate of publication of trial results is poor.

Abstract

Importance  Many investigational drugs fail in late-stage clinical development. A better understanding of why investigational drugs fail can inform clinical practice, regulatory decisions, and future research.

Objective  To assess factors associated with regulatory approval or reasons for failure of investigational therapeutics in phase 3 or pivotal trials and rates of publication of trial results.

Design, Setting, and Participants  Using public sources and commercial databases, we identified investigational therapeutics that entered pivotal trials between 1998 and 2008, with follow-up through 2015. Agents were classified by therapeutic area, orphan designation status, fast track designation, novelty of biological pathway, company size, and as a pharmacologic or biologic product.

Main Outcomes and Measures  For each product, we identified reasons for failure (efficacy, safety, commercial) and assessed the rates of publication of trial results. We used multivariable logistic regression models to evaluate factors associated with regulatory approval.

Results  Among 640 novel therapeutics, 344 (54%) failed in clinical development, 230 (36%) were approved by the US Food and Drug Administration (FDA), and 66 (10%) were approved in other countries but not by the FDA. Most products failed due to inadequate efficacy (n = 195; 57%), while 59 (17%) failed because of safety concerns and 74 (22%) failed due to commercial reasons. The pivotal trial results were published in peer-reviewed journals for 138 of the 344 (40%) failed agents. Of 74 trials for agents that failed for commercial reasons, only 6 (8.1%) were published. In analyses adjusted for therapeutic area, agent type, firm size, orphan designation, fast-track status, trial year, and novelty of biological pathway, orphan-designated drugs were significantly more likely than nonorphan drugs to be approved (46% vs 34%; adjusted odds ratio [aOR], 2.3; 95% CI, 1.4-3.7). Cancer drugs (27% vs 39%; aOR, 0.5; 95% CI, 0.3-0.9) and agents sponsored by small and medium-size companies (28% vs 42%; aOR, 0.4; 95% CI, 0.3-0.7) were significantly less likely to be approved.

Conclusions and Relevance  Roughly half of investigational drugs entering late-stage clinical development fail during or after pivotal clinical trials, primarily because of concerns about safety, efficacy, or both. Results for the majority of studies of investigational drugs that fail are not published in peer-reviewed journals.

Introduction

Phase 3 clinical trials provide the highest level of evidence that an experimental treatment is safe and efficacious. Although these trials, which typically involve large numbers of patients, require substantial investment on the part of participants, investigators, and sponsors, many experimental drugs tested at this stage fail.1 For example, recently, several therapies that demonstrated promise in animal and early testing have failed in larger studies to show clinical benefit, while increasing the risk of serious adverse events and death among participants.2-4

It is difficult to derive lessons from the experiences of unsuccessful experimental drugs. Negative clinical trial findings and the reasons for discontinuing the development of investigational products, including lack of approval by regulators, are often not disclosed.5 Trial data are often not reported publicly in a timely manner and may be worse for unapproved drugs.6 As a result, there are limited systematic data on why and how frequently novel agents fail in late-stage development. Previous studies have found that most new drug applications not approved by the US Food and Drug Administration (FDA) were reported to have efficacy deficiencies, safety deficiencies, or both. However, these studies did not assess the reasons for failure of drugs that did not reach regulatory filing or were not reviewed by the FDA.7,8

Phase 3 trials, even when the agent being tested does not demonstrate efficacy or safety, generate valuable information. Understanding the reasons for development failures can inform clinical practice, regulatory decisions, and future research. We sought to identify reasons that investigational therapeutics fail in late-stage clinical development, the rate of trial publications, and factors associated with regulatory approval in the United States, Europe, Japan, and other countries.

Methods
Data Sources

We constructed a data set of novel drugs and biologics from 2 commercial databases: Pharmaprojects (Informa plc; London, England) and AdisInsight (Springer; Berlin, Germany). These databases, which are the 2 most widely used by industry and regulators, track the development of products from preclinical research to marketing using public and proprietary sources, as well as direct communication with companies. They are assembled into longitudinal development timelines for each product in the database and updated in real-time. Data from these registries have been used in prior studies of pharmaceutical research and development.9-12

We obtained information about regulatory approvals and orphan drug status determinations from public databases maintained by the FDA; European Medicines Agency (EMA) and member states of the European Union, European Economic Area, and Switzerland; and national regulators in Japan, Canada, and Australia. As for a previous study,13 for discontinued products, we manually reviewed regulatory filings, market research reports, press releases, annual reports, published literature, conference abstracts, transcripts of earnings and investor relations calls and stock analyst reports, and other public and commercial sources to ascertain the basis for development failure as well as regulatory pathway (eTable 1 in the Supplement). We obtained company financial information from McGraw Hill Financial and Bloomberg. All data were initially downloaded on June 28, 2013, and updated through December 31, 2015.

Data Extraction

We identified all new drugs, therapeutic biologics, and vaccines that entered phase 3 or other pivotal testing between January 1, 1998, and December 31, 2008, with follow-up through December 31, 2015. We excluded nontherapeutic products, such as diagnostic tests, as well as blood and blood component products. This study period was chosen to allow sufficient time for trial completion (typically 2-5 years), regulatory review (up to 1 year), and publication of trial results, resulting in a total of up to 7 years or more from the start of a phase 3 trial to final approval or discontinuation). We focused on the lead (or first) indication for which the agent was reported to be in development by Pharmaprojects. For all identified agents, we extracted pivotal trial and approval dates (if applicable) from the development histories. A pivotal trial is a clinical study designed to provide adequate data on efficacy and safety to serve as the basis for regulatory approval of the agent for the proposed indication.14 These studies are typically phase 3 trials, but can also be phase 2 trials (representing approximately 5% of products in our study).

We also coded the indication, therapeutic area, World Health Organization Anatomical Therapeutic Chemical (ATC) code, agent type (pharmacologic or biologic), mechanism of action (or putative biological properties if mechanism was unknown), originator (first firm associated with the drug) and sponsor (firm[s] conducting the phase 3 trial) name, orphan designation status (a pathway used by the FDA and European Medicines Agency [EMA] for agents intended to treat rare diseases), and fast track designation by FDA or EMA (designated by the EMA as “accelerated assessment”). We defined small and medium-size companies as those with annual gross revenues less than $1 billion USD at the time of the pivotal trial.

We then assessed whether the drug was directed to a novel pathway, defined as a target or biological pathway for which the FDA had not yet approved a therapeutic agent by the pivotal trial start year, consistent with the definition used by FDA15 and others.16-18 Two investigators (B.W. and J.C.L.) independently assessed novelty (Cohen κ, 0.88), with disagreements resolved by consensus (eTable 2 in the Supplement).

Finally, we matched these data to the lists of approved drugs and biologics. For each discontinued product, the reasons for failure were identified using the data sources listed above. We categorized failures by whether they were primarily owing to safety (eg, imbalance of deaths in the pivotal trial treatment arm, reported serious adverse events, or other safety-related reasons), efficacy (eg, failure to show statistically significant benefit over a comparator), or commercial or other strategic reasons (eg, company went into bankruptcy and ceased development). Successful regulatory approval was defined as approval by the FDA; in sensitivity analyses, we also defined success as approval in the United States or Europe, and as approval in the United States, Europe, Japan, Canada, or Australia. European approval was defined as centralized approval by the EMA; approval through the mutual recognition procedure, which allows approval in 1 member state to be recognized by other European Union countries; or approval by Iceland, Liechtenstein, and Norway, which are European Economic Area countries, or Switzerland.

We searched Medline, EMBASE, and Web of Science for publications of trial results using the product’s chemical, generic, and proprietary names, investigator names, and clinical trial title (if applicable), updated through December 31, 2015.

Statistical Analysis

We used the Fisher exact test, as appropriate, to conduct pairwise comparisons of factors associated with failure of an investigational agent and the publication of trial results.

We then constructed multivariable logistic regression models to examine factors associated with successful regulatory approval. Models included all variables of interest regardless of statistical significance: therapeutic area, agent type (pharmacologic vs biologic), originator and sponsor firm type (small vs large), orphan designation, fast track status, novel pathway, and an indicator variable for trial start year (to account for secular trends over time). In sensitivity analyses, we repeated our analysis using a continuous time variable instead of an indicator variable for trial year and using an alternate threshold of $100 million USD to define small and medium-size companies.

Statistical analyses were performed using Stata version 12 (StataCorp). Two-tailed P values less than .05 were considered statistically significant.

Results

We examined the status of clinical development and basis for failure or regulatory approval for 640 novel therapeutic agents (Table 1): 344 (54%) of the agents failed; 230 (36%) were approved by the FDA, 49 (8%) were granted regulatory approval in Europe, Japan, Canada, or Australia, but not the United States, and 17 (3%) were approved in countries other than the United States, Europe, Japan, Canada, and Australia. The majority of new agents entered pivotal trials during the study period for 3 therapeutic areas: cancer (147 [23%]), cardiovascular disease (102 [16%]), and infectious diseases (100 [16%]). Orphan designation and fast track review were granted to 125 agents (20%) and 118 agents (18%), respectively; 359 (56%) of the agents were categorized as targeting a novel pathway.

Reasons for Failure

Among the 344 unapproved agents, the clinical development for 195 (57%) failed for lack of efficacy, for 59 (17%) due to safety concerns, and for 74 (22%) due to commercial or other reasons (Table 2). We were unable to identify the reasons for failure of 16 (5%) agents.

The failures related to safety included 10 (17%) agents with testing halted due to an increased risk of death; 18 (31%) associated with serious adverse effects such as cancer, stroke, and sepsis; 5 (8%) associated with laboratory test abnormalities; 5 (8%) associated with carcinogenicity or other serious adverse effects in long-term preclinical studies; and 21 (36%) with undisclosed safety issues or a requirement for further safety testing.

In univariable analyses, orphan-designated agents and neurological agents were more likely than nonorphan and non-neurological agents to fail for efficacy-related reasons (Fisher exact P = .02). Commercial reasons were more likely to be cited as the reason for failure of agents developed by small and medium-size companies (Fisher exact P < .001).

Factors Associated With Regulatory Approval

In both unadjusted (Figure) and adjusted (Table 3) analyses, the factors most strongly associated with likelihood of approval by the FDA were orphan designation, cancer drugs, and sponsor size. As compared with nonorphan drugs, orphan drugs were more likely to gain FDA approval than nonorphan drugs (unadjusted rates, 46% vs 34%; adjusted odds ratio [aOR], 2.3; 95% CI, 1.4-3.7; P < .001). Cancer agents were less likely to gain FDA approval than noncancer agents (27% vs 39%; aOR, 0.5; 95% CI, 0.3-0.9; P = .02), and agents sponsored by small and medium-size companies were less likely to gain FDA approval as compared with those sponsored by large companies (28% vs 42%; aOR, 0.4; 95% CI, 0.3-0.7; P < .001). These associations remained significant when defining success as regulatory approval in either the United States or Europe, or as regulatory approval in the United States or any of Europe, Japan, Australia, and Canada (Table 3).

In sensitivity analyses, we obtained similar results using a continuous time variable (eTable 3 in the Supplement) and using a different threshold (annual gross revenues of less than $100 million) to define small and medium-size companies (eTable 4 and eTable 5 in the Supplement).

Trial Results Reporting

The pivotal study results for 138 (40%) of the agents were published in peer-reviewed journals (Table 4). Agents that failed owing to efficacy or safety reasons were more likely than those that failed for commercial reasons to have published trial results (Fisher exact P < .001 for the comparison across categories). Of 74 trials for agents that failed for commercial reasons, only 6 (8.1%) were published. Additional predictors of publication included development by a large company, cardiovascular agents, and neurological agents (Fisher exact P < .001 for all univariable tests).

Discussion

In this study of investigational drugs entering late-stage clinical development between 1998 and 2008 with follow-up through 2015, we found about half of the experimental medicines failed during or after pivotal clinical trials. Most of these development failures were attributable to inadequate evidence of efficacy. The testing for a further 1 in 5 products was halted because of an increased risk of death or other potentially serious harms to patients.

Although many experimental treatments may be publicly described in superlatives during their development process, true breakthroughs are rare.19,20 In several cases, the phase 3 studies reversed the encouraging results of earlier investigations. For example, elesclomol, a first-in-class compound believed to induce oxidative stress, combined with paclitaxel showed a statistically significant improvement in progression-free survival compared with paclitaxel alone in a phase 2 trial of patients with advanced metastatic melanoma.21 Yet, in a larger phase 3 trial, the elesclomol combination did not significantly improve progression-free survival, and the trial was halted when more deaths in the combination arm were observed.22

We found that certain categories of products were more likely to succeed. Orphan-designated drugs, for example, were highly likely to be approved after late-stage trials. Previous studies have shown that orphan drugs are more likely than nonorphan drugs to be approved based on small, single-arm trials, which may be explained by the difficulty of enrolling patients with rare diseases.23,24 However, such trial designs may increase the risk of false-positive results, pointing to the need for timely completion of rigorous postapproval studies.25 Success rates also varied by therapeutic area. For example, our results are consistent with prior work9 showing that infectious disease trials have high success rates. This finding suggests that recent policy efforts to accelerate the approval of infectious disease agents may be better targeted at increasing the number of novel compounds reaching clinical trials, rather than altering the standards for success in such trials.

Despite the importance of the evidence generated by pivotal trials and the large numbers of patients involved, we found that the study results for less than half of the products that failed were eventually published, which is substantially lower than the previously reported trial publication rates of 76% to 86% for approved drugs.26-28 This gap in publication rates has important ethical implications. First, many patients in clinical trials agree to participate to advance scientific understanding of disease. Researchers and sponsors have a responsibility to ensure that the contributions of these patients are honored, even if the development is discontinued, through timely sharing of results in the published literature, where the findings and insights from the trials are accessible to other patients, researchers, and clinicians. Second, negative results can inform clinical practice: for example, trials of an unapproved drug may yield new insights into the safety and pharmacology of other approved agents in that class or in related drug classes.29,30 Third, an incomplete publication record can hinder the translational medicine process.31,32 Without knowledge of safety and efficacy issues found later in the development process, researchers may continue to bring forward investigational agents to clinical trials that are unlikely to show benefit.33 As a result, future research subjects might be more likely to be exposed to harms from toxic or futile treatments.34,35 Such data are also valuable for the repurposing of failed drugs for new indications, such as thalidomide for treatment of patients with multiple myeloma and leprosy.36 Given the increasing cost of clinical trials, lack of information sharing wastes resources and diverts attention from more productive areas of research. To that end, the National Institutes of Health recently proposed a regulation to require that the results of trials for unapproved drugs be deposited in the public ClinicalTrials.gov repository.37,38 If it were to take effect, this rule may promote public accessibility of knowledge gained from clinical trials, even if the results are not yet published in the medical literature.

Our study has several limitations. First, we focused on compounds that failed in late-stage development, and our results may not be generalizable to products discontinued in early-stage testing. Second, although we relied on both public and commercial sources, it is possible that we did not capture all of the reasons for failure or all of the products under development. A prior study found that sponsors often do not adequately disclose the precise reasons why drugs are not approved, and complete response letters issued by the FDA for unapproved products were not available.5 However, we were able to identify broadly stated reasons for failure for most products in our study cohort. Our results are consistent with previous studies of failures of products in phase 3 trials that occurred from 2007 to 201239 and of products developed by large pharmaceutical companies, which have found that the majority of failures are due to inadequate efficacy.40,41 In addition, we used 2 comprehensive databases; our data set of 640 drugs over a 10-year period is larger than that in a FDA study7 that reported 302 new drug applications submitted between 2000 and 2012. Third, although we chose our study period to allow sufficient follow-up time for the products in our cohort, it is possible that some drugs that are currently unapproved may gain approval, and more trial results may be published. Finally, we cannot exclude the possibility of unmeasured confounders, such as other types of regulatory pathways used to expedite approval, which were unavailable in our data sources. These limitations, however, are unlikely to substantially affect our conclusions that many investigational products are discontinued in late-stage development owing to concerns about efficacy and safety and that trial results for unapproved drugs frequently remain unpublished.

Conclusions

Recent policymaking aimed at stimulating pharmaceutical innovation has focused on allowing drugs to be approved on the basis of smaller data sets.42 Some commentators have proposed waiving the need for phase 3 testing, although others have responded that approval before rigorous study could worsen health outcomes by leading to widespread use of toxic or ineffective drugs that would have otherwise been shown to have failed.43-45 As many investigational products fail in late-stage development because of inadequate efficacy or safety, our findings suggest that additional efforts to promote drug development should be directed at improving the validity of preclinical models for use in translational research and increasing the number of innovative products entering trials. The timely publication of trial results for all investigational agents, including those that fail in late-stage clinical development, is imperative.

Back to top
Article Information

Corresponding Author: Aaron S. Kesselheim, MD, JD, MPH, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, 1620 Tremont St, Ste 3030, Boston, MA 02120 (akesselheim@partners.org).

Accepted for Publication: August 16, 2016.

Published Online: October 10, 2016. doi:10.1001/jamainternmed.2016.6008

Author Contributions: Mr Hwang had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis

Concept and design: Hwang, Carpenter, Kesselheim.

Acquisition, analysis, or interpretation of data: All Authors.

Drafting of the manuscript: Hwang.

Critical revision of the manuscript for important intellectual content: All Authors.

Statistical analysis: Hwang, Franklin.

Administrative, technical, or material support: Carpenter, Lauffenburger, Wang.

Study supervision: Carpenter, Franklin, Kesselheim.

Funding/Support: This research was supported by the Laura and John Arnold Foundation, Greenwall Foundation, and Harvard Program in Therapeutic Science.

Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Conflict of Interest Disclosures: Mr Hwang was previously employed by Blackstone and Bain Capital, which have invested in healthcare companies. Dr Lauffenburger has received unrestricted research funding payable to her institution from AstraZeneca. Dr Franklin has received research funding from the Patient-Centered Outcomes Research Institute (PCORI) and Merck & Co, and has consulted for Aetion, Inc, a software company. Dr Kesselheim reports receiving unrelated grants from the US Food and Drug Administration Office of Generic Drugs and Division of Health Communication. No other conflicts are reported.

References
1.
Institute of Medicine, Forum on Drug Discovery, Development, and Translation. Transforming Clinical Research in the United States: Challenges and Opportunities. Washington, DC: National Academies Press; 2010.
2.
Barter  PJ, Caulfield  M, Eriksson  M,  et al; ILLUMINATE Investigators.  Effects of torcetrapib in patients at high risk for coronary events.  N Engl J Med. 2007;357(21):2109-2122.PubMedGoogle ScholarCrossref
3.
Doody  RS, Raman  R, Farlow  M,  et al; Alzheimer’s Disease Cooperative Study Steering Committee; Semagacestat Study Group.  A phase 3 trial of semagacestat for treatment of Alzheimer’s disease.  N Engl J Med. 2013;369(4):341-350.PubMedGoogle ScholarCrossref
4.
Amiri-Kordestani  L, Fojo  T.  Why do phase III clinical trials in oncology fail so often?  J Natl Cancer Inst. 2012;104(8):568-569.PubMedGoogle ScholarCrossref
5.
Lurie  P, Chahal  HS, Sigelman  DW, Stacy  S, Sclar  J, Ddamulira  B.  Comparison of content of FDA letters not approving applications for new drugs and associated public announcements from sponsors: cross sectional study.  BMJ. 2015;350:h2758.PubMedGoogle ScholarCrossref
6.
Hakala  A, Kimmelman  J, Carlisle  B, Freeman  G, Fergusson  D.  Accessibility of trial reports for drugs stalling in development: a systematic assessment of registered trials.  BMJ. 2015;350:h1116.PubMedGoogle ScholarCrossref
7.
Sacks  LV, Shamsuddin  HH, Yasinskaya  YI, Bouri  K, Lanthier  ML, Sherman  RE.  Scientific and regulatory reasons for delay and denial of FDA approval of initial applications for new drugs, 2000-2012.  JAMA. 2014;311(4):378-384.PubMedGoogle ScholarCrossref
8.
Ross  JS, Dzara  K, Downing  NS.  Efficacy and safety concerns are important reasons why the FDA requires multiple reviews before approval of new drugs.  Health Aff (Millwood). 2015;34(4):681-688.PubMedGoogle ScholarCrossref
9.
Hwang  TJ, Carpenter  D, Kesselheim  AS.  Target small firms for antibiotic innovation.  Science. 2014;344(6187):967-969.PubMedGoogle ScholarCrossref
10.
Carpenter  D, Zucker  EJ, Avorn  J.  Drug-review deadlines and safety problems.  N Engl J Med. 2008;358(13):1354-1361.PubMedGoogle ScholarCrossref
11.
Lincker  H, Ziogas  C, Carr  M, Porta  N, Eichler  HG.  Regulatory watch: where do new medicines originate from in the EU?  Nat Rev Drug Discov. 2014;13(2):92-93.PubMedGoogle ScholarCrossref
12.
Adams  CP, Brantner  VV.  Estimating the cost of new drug development: is it really 802 million dollars?  Health Aff (Millwood). 2006;25(2):420-428.PubMedGoogle ScholarCrossref
13.
Kesselheim  AS, Hwang  TJ, Franklin  JM.  Two decades of new drug development for central nervous system disorders.  Nat Rev Drug Discov. 2015;14(12):815-816.PubMedGoogle ScholarCrossref
14.
Hamburg  MA. Why FDA Supports a Flexible Approach to Drug Development. FDA website. http://blogs.fda.gov/fdavoice/index.php/2014/02/why-fda-supports-a-flexible-approach-to-drug-development/ Accessed February 22, 2016.
15.
Lanthier  M, Miller  KL, Nardinelli  C, Woodcock  J.  An improved approach to measuring drug innovation finds steady rates of first-in-class pharmaceuticals, 1987-2011.  Health Aff (Millwood). 2013;32(8):1433-1439.PubMedGoogle ScholarCrossref
16.
Kesselheim  AS, Wang  B, Franklin  JM, Darrow  JJ.  Trends in utilization of FDA expedited drug development and approval programs, 1987-2014: cohort study.  BMJ. 2015;351:h4633.PubMedGoogle ScholarCrossref
17.
Eder  J, Sedrani  R, Wiesmann  C.  The discovery of first-in-class drugs: origins and evolution.  Nat Rev Drug Discov. 2014;13(8):577-587.PubMedGoogle ScholarCrossref
18.
Hwang  TJ, Lauffenburger  JC, Franklin  JM, Kesselheim  AS.  Temporal trends and factors associated with cardiovascular drug development, 1990 to 2012  [published online August 29, 2016].  J Am Coll Cardiol Basic Trans Science. doi:10.1016/j.jacbts.2016.03.012Google Scholar
19.
Abola  MV, Prasad  V.  The Use of Superlatives in Cancer Research.  JAMA Oncol. 2016;2(1):139-141.PubMedGoogle ScholarCrossref
20.
Djulbegovic  B, Kumar  A, Glasziou  PP,  et al.  New treatments compared to established treatments in randomized trials.  Cochrane Database Syst Rev. 2012;10:MR000024.PubMedGoogle Scholar
21.
O’Day  S, Gonzalez  R, Lawson  D,  et al.  Phase II, randomized, controlled, double-blinded trial of weekly elesclomol plus paclitaxel versus paclitaxel alone for stage IV metastatic melanoma.  J Clin Oncol. 2009;27(32):5452-5458.PubMedGoogle ScholarCrossref
22.
O’Day  SJ, Eggermont  AM, Chiarion-Sileni  V,  et al.  Final results of phase III SYMMETRY study: randomized, double-blind trial of elesclomol plus paclitaxel versus paclitaxel alone as treatment for chemotherapy-naive patients with advanced melanoma.  J Clin Oncol. 2013;31(9):1211-1218.PubMedGoogle ScholarCrossref
23.
Kesselheim  AS, Myers  JA, Avorn  J.  Characteristics of clinical trials to support approval of orphan vs nonorphan drugs for cancer.  JAMA. 2011;305(22):2320-2326.PubMedGoogle ScholarCrossref
24.
Mitsumoto  J, Dorsey  ER, Beck  CA, Kieburtz  K, Griggs  RC.  Pivotal studies of orphan drugs approved for neurological diseases.  Ann Neurol. 2009;66(2):184-190.PubMedGoogle ScholarCrossref
25.
Button  KS, Ioannidis  JP, Mokrysz  C,  et al.  Power failure: why small sample size undermines the reliability of neuroscience.  Nat Rev Neurosci. 2013;14(5):365-376.PubMedGoogle ScholarCrossref
26.
Lee  K, Bacchetti  P, Sim  I.  Publication of clinical trials supporting successful new drug applications: a literature analysis.  PLoS Med. 2008;5(9):e191.PubMedGoogle ScholarCrossref
27.
Rising  K, Bacchetti  P, Bero  L.  Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation.  PLoS Med. 2008;5(11):e217.PubMedGoogle ScholarCrossref
28.
Smithy  JW, Downing  NS, Ross  JS.  Publication of pivotal efficacy trials for novel therapeutic agents approved between 2005 and 2011: a cross-sectional study.  JAMA Intern Med. 2014;174(9):1518-1520.PubMedGoogle ScholarCrossref
29.
Turner  EH, Matthews  AM, Linardatos  E, Tell  RA, Rosenthal  R.  Selective publication of antidepressant trials and its influence on apparent efficacy.  N Engl J Med. 2008;358(3):252-260.PubMedGoogle ScholarCrossref
30.
Hart  B, Lundh  A, Bero  L.  Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses.  BMJ. 2012;344:d7202.PubMedGoogle ScholarCrossref
31.
Becker  RE, Greig  NH.  Lost in translation: neuropsychiatric drug development.  Sci Transl Med. 2010;2(61):61rv6.PubMedGoogle ScholarCrossref
32.
van der Worp  HB, Howells  DW, Sena  ES,  et al.  Can animal models of disease reliably inform human studies?  PLoS Med. 2010;7(3):e1000245.PubMedGoogle ScholarCrossref
33.
O’Collins  VE, Macleod  MR, Donnan  GA, Horky  LL, van der Worp  BH, Howells  DW.  1,026 experimental treatments in acute stroke.  Ann Neurol. 2006;59(3):467-477.PubMedGoogle ScholarCrossref
34.
Chalmers  I.  Underreporting research is scientific misconduct.  JAMA. 1990;263(10):1405-1408.PubMedGoogle ScholarCrossref
35.
Seruga  B, Templeton  AJ, Badillo  FE, Ocana  A, Amir  E, Tannock  IF.  Under-reporting of harm in clinical trials.  Lancet Oncol. 2016;17(5):e209-e219.PubMedGoogle ScholarCrossref
36.
Singhal  S, Mehta  J, Desikan  R,  et al.  Antitumor activity of thalidomide in refractory multiple myeloma.  N Engl J Med. 1999;341(21):1565-1571.PubMedGoogle ScholarCrossref
37.
Hudson  KL, Collins  FS.  Sharing and reporting the results of clinical trials.  JAMA. 2015;313(4):355-356.PubMedGoogle ScholarCrossref
38.
Federal Register. National Institutes of Health: Notice of Proposed Rulemaking on Clinical Trials Registration and Results Submission. https://federalregister.gov/a/2014-26197. Accessed August 11, 2016.
39.
Arrowsmith  J, Miller  P.  Trial watch: phase II and phase III attrition rates 2011-2012.  Nat Rev Drug Discov. 2013;12(8):569.PubMedGoogle ScholarCrossref
40.
Waring  MJ, Arrowsmith  J, Leach  AR,  et al.  An analysis of the attrition of drug candidates from four major pharmaceutical companies.  Nat Rev Drug Discov. 2015;14(7):475-486.PubMedGoogle ScholarCrossref
41.
Hay  M, Thomas  DW, Craighead  JL, Economides  C, Rosenthal  J.  Clinical development success rates for investigational drugs.  Nat Biotechnol. 2014;32(1):40-51.PubMedGoogle ScholarCrossref
42.
Zuckerman  DM, Jury  NJ, Silcox  CE.  21st Century Cures Act and similar policy efforts: at what cost?  BMJ. 2015;351:h6122.PubMedGoogle ScholarCrossref
43.
Redberg  RF.  Faster drug approvals are not always better and can be worse.  JAMA Intern Med. 2015;175(8):1398.PubMedGoogle ScholarCrossref
44.
Gonsalves  G, Zuckerman  D.  Commentary: Will 20th century patient safeguards be reversed in the 21st century?  BMJ. 2015;350:h1500.PubMedGoogle ScholarCrossref
45.
Floyd  JS, Psaty  BM.  The potential risks of expedited approval of drugs for acute bacterial infections.  JAMA Intern Med. 2014;174(9):1436-1437.PubMedGoogle ScholarCrossref
×