Failure of Investigational Drugs in Late-Stage Clinical Development and Publication of Trial Results

Key Points

Question  Why and how often do experimental drugs fail in phase 3 clinical trials, and how often are trial results published?

Findings  Using public sources and commercial databases covering drugs and biologics that started trials between 1998 and 2008, 54% of agents carried into pivotal trials failed, primarily owing to inadequate efficacy or safety concerns. Trial results were published for 40% of these failed agents.

Meaning  Although many drugs fail in late-stage trials, the rate of publication of trial results is poor.

Importance  Many investigational drugs fail in late-stage clinical development. A better understanding of why investigational drugs fail can inform clinical practice, regulatory decisions, and future research.

Objective  To assess factors associated with regulatory approval or reasons for failure of investigational therapeutics in phase 3 or pivotal trials and rates of publication of trial results.

Design, Setting, and Participants  Using public sources and commercial databases, we identified investigational therapeutics that entered pivotal trials between 1998 and 2008, with follow-up through 2015. Agents were classified by therapeutic area, orphan designation status, fast track designation, novelty of biological pathway, company size, and as a pharmacologic or biologic product.

Main Outcomes and Measures  For each product, we identified reasons for failure (efficacy, safety, commercial) and assessed the rates of publication of trial results. We used multivariable logistic regression models to evaluate factors associated with regulatory approval.

Results  Among 640 novel therapeutics, 344 (54%) failed in clinical development, 230 (36%) were approved by the US Food and Drug Administration (FDA), and 66 (10%) were approved in other countries but not by the FDA. Most products failed due to inadequate efficacy (n = 195; 57%), while 59 (17%) failed because of safety concerns and 74 (22%) failed due to commercial reasons. The pivotal trial results were published in peer-reviewed journals for 138 of the 344 (40%) failed agents. Of 74 trials for agents that failed for commercial reasons, only 6 (8.1%) were published. In analyses adjusted for therapeutic area, agent type, firm size, orphan designation, fast-track status, trial year, and novelty of biological pathway, orphan-designated drugs were significantly more likely than nonorphan drugs to be approved (46% vs 34%; adjusted odds ratio [aOR], 2.3; 95% CI, 1.4-3.7). Cancer drugs (27% vs 39%; aOR, 0.5; 95% CI, 0.3-0.9) and agents sponsored by small and medium-size companies (28% vs 42%; aOR, 0.4; 95% CI, 0.3-0.7) were significantly less likely to be approved.

Conclusions and Relevance  Roughly half of investigational drugs entering late-stage clinical development fail during or after pivotal clinical trials, primarily because of concerns about safety, efficacy, or both. Results for the majority of studies of investigational drugs that fail are not published in peer-reviewed journals.

Phase 3 clinical trials provide the highest level of evidence that an experimental treatment is safe and efficacious. Although these trials, which typically involve large numbers of patients, require substantial investment on the part of participants, investigators, and sponsors, many experimental drugs tested at this stage fail.¹ For example, recently, several therapies that demonstrated promise in animal and early testing have failed in larger studies to show clinical benefit, while increasing the risk of serious adverse events and death among participants.^2-4

It is difficult to derive lessons from the experiences of unsuccessful experimental drugs. Negative clinical trial findings and the reasons for discontinuing the development of investigational products, including lack of approval by regulators, are often not disclosed.⁵ Trial data are often not reported publicly in a timely manner and may be worse for unapproved drugs.⁶ As a result, there are limited systematic data on why and how frequently novel agents fail in late-stage development. Previous studies have found that most new drug applications not approved by the US Food and Drug Administration (FDA) were reported to have efficacy deficiencies, safety deficiencies, or both. However, these studies did not assess the reasons for failure of drugs that did not reach regulatory filing or were not reviewed by the FDA.^7,8

Phase 3 trials, even when the agent being tested does not demonstrate efficacy or safety, generate valuable information. Understanding the reasons for development failures can inform clinical practice, regulatory decisions, and future research. We sought to identify reasons that investigational therapeutics fail in late-stage clinical development, the rate of trial publications, and factors associated with regulatory approval in the United States, Europe, Japan, and other countries.

We constructed a data set of novel drugs and biologics from 2 commercial databases: Pharmaprojects (Informa plc; London, England) and AdisInsight (Springer; Berlin, Germany). These databases, which are the 2 most widely used by industry and regulators, track the development of products from preclinical research to marketing using public and proprietary sources, as well as direct communication with companies. They are assembled into longitudinal development timelines for each product in the database and updated in real-time. Data from these registries have been used in prior studies of pharmaceutical research and development.^9-12

We obtained information about regulatory approvals and orphan drug status determinations from public databases maintained by the FDA; European Medicines Agency (EMA) and member states of the European Union, European Economic Area, and Switzerland; and national regulators in Japan, Canada, and Australia. As for a previous study,¹³ for discontinued products, we manually reviewed regulatory filings, market research reports, press releases, annual reports, published literature, conference abstracts, transcripts of earnings and investor relations calls and stock analyst reports, and other public and commercial sources to ascertain the basis for development failure as well as regulatory pathway (eTable 1 in the Supplement). We obtained company financial information from McGraw Hill Financial and Bloomberg. All data were initially downloaded on June 28, 2013, and updated through December 31, 2015.

We identified all new drugs, therapeutic biologics, and vaccines that entered phase 3 or other pivotal testing between January 1, 1998, and December 31, 2008, with follow-up through December 31, 2015. We excluded nontherapeutic products, such as diagnostic tests, as well as blood and blood component products. This study period was chosen to allow sufficient time for trial completion (typically 2-5 years), regulatory review (up to 1 year), and publication of trial results, resulting in a total of up to 7 years or more from the start of a phase 3 trial to final approval or discontinuation). We focused on the lead (or first) indication for which the agent was reported to be in development by Pharmaprojects. For all identified agents, we extracted pivotal trial and approval dates (if applicable) from the development histories. A pivotal trial is a clinical study designed to provide adequate data on efficacy and safety to serve as the basis for regulatory approval of the agent for the proposed indication.¹⁴ These studies are typically phase 3 trials, but can also be phase 2 trials (representing approximately 5% of products in our study).

We also coded the indication, therapeutic area, World Health Organization Anatomical Therapeutic Chemical (ATC) code, agent type (pharmacologic or biologic), mechanism of action (or putative biological properties if mechanism was unknown), originator (first firm associated with the drug) and sponsor (firm[s] conducting the phase 3 trial) name, orphan designation status (a pathway used by the FDA and European Medicines Agency [EMA] for agents intended to treat rare diseases), and fast track designation by FDA or EMA (designated by the EMA as “accelerated assessment”). We defined small and medium-size companies as those with annual gross revenues less than $1 billion USD at the time of the pivotal trial.

We then assessed whether the drug was directed to a novel pathway, defined as a target or biological pathway for which the FDA had not yet approved a therapeutic agent by the pivotal trial start year, consistent with the definition used by FDA¹⁵ and others.^16-18 Two investigators (B.W. and J.C.L.) independently assessed novelty (Cohen κ, 0.88), with disagreements resolved by consensus (eTable 2 in the Supplement).

Finally, we matched these data to the lists of approved drugs and biologics. For each discontinued product, the reasons for failure were identified using the data sources listed above. We categorized failures by whether they were primarily owing to safety (eg, imbalance of deaths in the pivotal trial treatment arm, reported serious adverse events, or other safety-related reasons), efficacy (eg, failure to show statistically significant benefit over a comparator), or commercial or other strategic reasons (eg, company went into bankruptcy and ceased development). Successful regulatory approval was defined as approval by the FDA; in sensitivity analyses, we also defined success as approval in the United States or Europe, and as approval in the United States, Europe, Japan, Canada, or Australia. European approval was defined as centralized approval by the EMA; approval through the mutual recognition procedure, which allows approval in 1 member state to be recognized by other European Union countries; or approval by Iceland, Liechtenstein, and Norway, which are European Economic Area countries, or Switzerland.

We searched Medline, EMBASE, and Web of Science for publications of trial results using the product’s chemical, generic, and proprietary names, investigator names, and clinical trial title (if applicable), updated through December 31, 2015.

We used the Fisher exact test, as appropriate, to conduct pairwise comparisons of factors associated with failure of an investigational agent and the publication of trial results.

We then constructed multivariable logistic regression models to examine factors associated with successful regulatory approval. Models included all variables of interest regardless of statistical significance: therapeutic area, agent type (pharmacologic vs biologic), originator and sponsor firm type (small vs large), orphan designation, fast track status, novel pathway, and an indicator variable for trial start year (to account for secular trends over time). In sensitivity analyses, we repeated our analysis using a continuous time variable instead of an indicator variable for trial year and using an alternate threshold of $100 million USD to define small and medium-size companies.

Statistical analyses were performed using Stata version 12 (StataCorp). Two-tailed P values less than .05 were considered statistically significant.

We examined the status of clinical development and basis for failure or regulatory approval for 640 novel therapeutic agents (Table 1): 344 (54%) of the agents failed; 230 (36%) were approved by the FDA, 49 (8%) were granted regulatory approval in Europe, Japan, Canada, or Australia, but not the United States, and 17 (3%) were approved in countries other than the United States, Europe, Japan, Canada, and Australia. The majority of new agents entered pivotal trials during the study period for 3 therapeutic areas: cancer (147 [23%]), cardiovascular disease (102 [16%]), and infectious diseases (100 [16%]). Orphan designation and fast track review were granted to 125 agents (20%) and 118 agents (18%), respectively; 359 (56%) of the agents were categorized as targeting a novel pathway.

Among the 344 unapproved agents, the clinical development for 195 (57%) failed for lack of efficacy, for 59 (17%) due to safety concerns, and for 74 (22%) due to commercial or other reasons (Table 2). We were unable to identify the reasons for failure of 16 (5%) agents.

The failures related to safety included 10 (17%) agents with testing halted due to an increased risk of death; 18 (31%) associated with serious adverse effects such as cancer, stroke, and sepsis; 5 (8%) associated with laboratory test abnormalities; 5 (8%) associated with carcinogenicity or other serious adverse effects in long-term preclinical studies; and 21 (36%) with undisclosed safety issues or a requirement for further safety testing.

In univariable analyses, orphan-designated agents and neurological agents were more likely than nonorphan and non-neurological agents to fail for efficacy-related reasons (Fisher exact P = .02). Commercial reasons were more likely to be cited as the reason for failure of agents developed by small and medium-size companies (Fisher exact P < .001).

In both unadjusted (Figure) and adjusted (Table 3) analyses, the factors most strongly associated with likelihood of approval by the FDA were orphan designation, cancer drugs, and sponsor size. As compared with nonorphan drugs, orphan drugs were more likely to gain FDA approval than nonorphan drugs (unadjusted rates, 46% vs 34%; adjusted odds ratio [aOR], 2.3; 95% CI, 1.4-3.7; P < .001). Cancer agents were less likely to gain FDA approval than noncancer agents (27% vs 39%; aOR, 0.5; 95% CI, 0.3-0.9; P = .02), and agents sponsored by small and medium-size companies were less likely to gain FDA approval as compared with those sponsored by large companies (28% vs 42%; aOR, 0.4; 95% CI, 0.3-0.7; P < .001). These associations remained significant when defining success as regulatory approval in either the United States or Europe, or as regulatory approval in the United States or any of Europe, Japan, Australia, and Canada (Table 3).

In sensitivity analyses, we obtained similar results using a continuous time variable (eTable 3 in the Supplement) and using a different threshold (annual gross revenues of less than $100 million) to define small and medium-size companies (eTable 4 and eTable 5 in the Supplement).

The pivotal study results for 138 (40%) of the agents were published in peer-reviewed journals (Table 4). Agents that failed owing to efficacy or safety reasons were more likely than those that failed for commercial reasons to have published trial results (Fisher exact P < .001 for the comparison across categories). Of 74 trials for agents that failed for commercial reasons, only 6 (8.1%) were published. Additional predictors of publication included development by a large company, cardiovascular agents, and neurological agents (Fisher exact P < .001 for all univariable tests).

In this study of investigational drugs entering late-stage clinical development between 1998 and 2008 with follow-up through 2015, we found about half of the experimental medicines failed during or after pivotal clinical trials. Most of these development failures were attributable to inadequate evidence of efficacy. The testing for a further 1 in 5 products was halted because of an increased risk of death or other potentially serious harms to patients.

Although many experimental treatments may be publicly described in superlatives during their development process, true breakthroughs are rare.^19,20 In several cases, the phase 3 studies reversed the encouraging results of earlier investigations. For example, elesclomol, a first-in-class compound believed to induce oxidative stress, combined with paclitaxel showed a statistically significant improvement in progression-free survival compared with paclitaxel alone in a phase 2 trial of patients with advanced metastatic melanoma.²¹ Yet, in a larger phase 3 trial, the elesclomol combination did not significantly improve progression-free survival, and the trial was halted when more deaths in the combination arm were observed.²²

We found that certain categories of products were more likely to succeed. Orphan-designated drugs, for example, were highly likely to be approved after late-stage trials. Previous studies have shown that orphan drugs are more likely than nonorphan drugs to be approved based on small, single-arm trials, which may be explained by the difficulty of enrolling patients with rare diseases.^23,24 However, such trial designs may increase the risk of false-positive results, pointing to the need for timely completion of rigorous postapproval studies.²⁵ Success rates also varied by therapeutic area. For example, our results are consistent with prior work⁹ showing that infectious disease trials have high success rates. This finding suggests that recent policy efforts to accelerate the approval of infectious disease agents may be better targeted at increasing the number of novel compounds reaching clinical trials, rather than altering the standards for success in such trials.

Despite the importance of the evidence generated by pivotal trials and the large numbers of patients involved, we found that the study results for less than half of the products that failed were eventually published, which is substantially lower than the previously reported trial publication rates of 76% to 86% for approved drugs.^26-28 This gap in publication rates has important ethical implications. First, many patients in clinical trials agree to participate to advance scientific understanding of disease. Researchers and sponsors have a responsibility to ensure that the contributions of these patients are honored, even if the development is discontinued, through timely sharing of results in the published literature, where the findings and insights from the trials are accessible to other patients, researchers, and clinicians. Second, negative results can inform clinical practice: for example, trials of an unapproved drug may yield new insights into the safety and pharmacology of other approved agents in that class or in related drug classes.^29,30 Third, an incomplete publication record can hinder the translational medicine process.^31,32 Without knowledge of safety and efficacy issues found later in the development process, researchers may continue to bring forward investigational agents to clinical trials that are unlikely to show benefit.³³ As a result, future research subjects might be more likely to be exposed to harms from toxic or futile treatments.^34,35 Such data are also valuable for the repurposing of failed drugs for new indications, such as thalidomide for treatment of patients with multiple myeloma and leprosy.³⁶ Given the increasing cost of clinical trials, lack of information sharing wastes resources and diverts attention from more productive areas of research. To that end, the National Institutes of Health recently proposed a regulation to require that the results of trials for unapproved drugs be deposited in the public ClinicalTrials.gov repository.^37,38 If it were to take effect, this rule may promote public accessibility of knowledge gained from clinical trials, even if the results are not yet published in the medical literature.

Our study has several limitations. First, we focused on compounds that failed in late-stage development, and our results may not be generalizable to products discontinued in early-stage testing. Second, although we relied on both public and commercial sources, it is possible that we did not capture all of the reasons for failure or all of the products under development. A prior study found that sponsors often do not adequately disclose the precise reasons why drugs are not approved, and complete response letters issued by the FDA for unapproved products were not available.⁵ However, we were able to identify broadly stated reasons for failure for most products in our study cohort. Our results are consistent with previous studies of failures of products in phase 3 trials that occurred from 2007 to 2012³⁹ and of products developed by large pharmaceutical companies, which have found that the majority of failures are due to inadequate efficacy.^40,41 In addition, we used 2 comprehensive databases; our data set of 640 drugs over a 10-year period is larger than that in a FDA study⁷ that reported 302 new drug applications submitted between 2000 and 2012. Third, although we chose our study period to allow sufficient follow-up time for the products in our cohort, it is possible that some drugs that are currently unapproved may gain approval, and more trial results may be published. Finally, we cannot exclude the possibility of unmeasured confounders, such as other types of regulatory pathways used to expedite approval, which were unavailable in our data sources. These limitations, however, are unlikely to substantially affect our conclusions that many investigational products are discontinued in late-stage development owing to concerns about efficacy and safety and that trial results for unapproved drugs frequently remain unpublished.

Recent policymaking aimed at stimulating pharmaceutical innovation has focused on allowing drugs to be approved on the basis of smaller data sets.⁴² Some commentators have proposed waiving the need for phase 3 testing, although others have responded that approval before rigorous study could worsen health outcomes by leading to widespread use of toxic or ineffective drugs that would have otherwise been shown to have failed.^43-45 As many investigational products fail in late-stage development because of inadequate efficacy or safety, our findings suggest that additional efforts to promote drug development should be directed at improving the validity of preclinical models for use in translational research and increasing the number of innovative products entering trials. The timely publication of trial results for all investigational agents, including those that fail in late-stage clinical development, is imperative.

Corresponding Author: Aaron S. Kesselheim, MD, JD, MPH, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, 1620 Tremont St, Ste 3030, Boston, MA 02120 (akesselheim@partners.org).

Accepted for Publication: August 16, 2016.

Published Online: October 10, 2016. doi:10.1001/jamainternmed.2016.6008

Author Contributions: Mr Hwang had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis

Concept and design: Hwang, Carpenter, Kesselheim.

Acquisition, analysis, or interpretation of data: All Authors.

Drafting of the manuscript: Hwang.

Critical revision of the manuscript for important intellectual content: All Authors.

Statistical analysis: Hwang, Franklin.

Administrative, technical, or material support: Carpenter, Lauffenburger, Wang.

Study supervision: Carpenter, Franklin, Kesselheim.

Funding/Support: This research was supported by the Laura and John Arnold Foundation, Greenwall Foundation, and Harvard Program in Therapeutic Science.

Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Conflict of Interest Disclosures: Mr Hwang was previously employed by Blackstone and Bain Capital, which have invested in healthcare companies. Dr Lauffenburger has received unrestricted research funding payable to her institution from AstraZeneca. Dr Franklin has received research funding from the Patient-Centered Outcomes Research Institute (PCORI) and Merck & Co, and has consulted for Aetion, Inc, a software company. Dr Kesselheim reports receiving unrelated grants from the US Food and Drug Administration Office of Generic Drugs and Division of Health Communication. No other conflicts are reported.