Prevalence, Characteristics, and Publication of Discontinued Randomized Trials

Importance  The discontinuation of randomized clinical trials (RCTs) raises ethical concerns and often wastes scarce research resources. The epidemiology of discontinued RCTs, however, remains unclear.

Objectives  To determine the prevalence, characteristics, and publication history of discontinued RCTs and to investigate factors associated with RCT discontinuation due to poor recruitment and with nonpublication.

Design and Setting  Retrospective cohort of RCTs based on archived protocols approved by 6 research ethics committees in Switzerland, Germany, and Canada between 2000 and 2003. We recorded trial characteristics and planned recruitment from included protocols. Last follow-up of RCTs was April 27, 2013.

Main Outcomes and Measures  Completion status, reported reasons for discontinuation, and publication status of RCTs as determined by correspondence with the research ethics committees, literature searches, and investigator surveys.

Results  After a median follow-up of 11.6 years (range, 8.8-12.6 years), 253 of 1017 included RCTs were discontinued (24.9% [95% CI, 22.3%-27.6%]). Only 96 of 253 discontinuations (37.9% [95% CI, 32.0%-44.3%]) were reported to ethics committees. The most frequent reason for discontinuation was poor recruitment (101/1017; 9.9% [95% CI, 8.2%-12.0%]). In multivariable analysis, industry sponsorship vs investigator sponsorship (8.4% vs 26.5%; odds ratio [OR], 0.25 [95% CI, 0.15-0.43]; P < .001) and a larger planned sample size in increments of 100 (−0.7%; OR, 0.96 [95% CI, 0.92-1.00]; P = .04) were associated with lower rates of discontinuation due to poor recruitment. Discontinued trials were more likely to remain unpublished than completed trials (55.1% vs 33.6%; OR, 3.19 [95% CI, 2.29-4.43]; P < .001).

Conclusions and Relevance  In this sample of trials based on RCT protocols from 6 research ethics committees, discontinuation was common, with poor recruitment being the most frequently reported reason. Greater efforts are needed to ensure the reporting of trial discontinuation to research ethics committees and the publication of results of discontinued trials.

Conducting high-quality randomized clinical trials (RCTs) is challenging and resource-demanding. Trials are often not conducted as planned or are even prematurely discontinued, eg, for reasons of unexpected harm from the intervention, early superiority, futility, administrative problems, or poor recruitment of participants. Trial discontinuation poses ethical concerns, particularly if results remain unreported, and may represent a considerable waste of scarce research resources.^1-3

Currently, little is known about the epidemiology and publication history of discontinued trials. Although studies have highlighted frequent recruitment problems in RCTs, few have empirically addressed factors associated with failure (or success) of patient recruitment, yielding uncertain results.^4-6 We established an international retrospective cohort of RCT protocols to determine: (1) the prevalence of RCT discontinuation and its reasons, (2) differences between RCTs with investigator sponsorship or industry sponsorship, (3) the publication history of discontinued RCTs, (4) factors associated with trial discontinuation due to poor recruitment, and (5) factors associated with nonpublication.

The protocol of this study has been published.³ Briefly, we conducted a retrospective cohort study using RCT protocols approved between 2000 and 2003 by 6 research ethics committees (RECs) in Switzerland (Basel, Lucerne, Zurich, and Lausanne), Germany (Freiburg), and Canada (Hamilton, Ontario). Of these RECs, all but 1 are responsible for human research in large university centers and additional hospitals in their respective catchment areas; the Lucerne REC covers an academic teaching hospital. As a convenience sample we approached the RECs through existing contacts. The participating RECs approved this study or explicitly stated that no ethical approval was necessary.

We considered an RCT discontinued if the investigators indicated discontinuation with a reason in the correspondence with the REC, in a journal publication, or in their response to our survey (see below). If we could not elucidate the reason for trial discontinuation or if poor participant recruitment was mentioned, we used a prespecified cutoff of less than 90% of achieved target sample size (and in a sensitivity analysis of less than 80%) to determine discontinuation.³ Reviewers assessed RCT protocols for industry sponsorship or investigator sponsorship using the following criteria: The protocol clearly named the sponsor; displayed a company or institution logo prominently; mentioned affiliations of protocol authors; or included statements about data ownership or publication rights or statements about full funding by industry or public funding agencies.⁷ Disagreements were resolved by consensus. We regarded peer-reviewed journal publications other than conference abstracts or research letters as full publications. Further details about these and other working definitions of study variables are provided in eTable 1 in Supplement. We a priori defined RCTs involving healthy volunteers as a subgroup, because we anticipated that they would have considerably smaller target sample sizes and, in contrast to RCTs involving patients, use financial incentives, thus leading to different discontinuation patterns.³

We used a web-based database for data extraction and management (http://www.squiekero.org/). Reviewers trained in trial methodology signed confidentiality agreements, completed a calibration process, and then extracted relevant data from RCT protocols.³ The initial 310 protocols (30%) were extracted independently and in duplicate, and disagreements were resolved by discussion; the remaining protocols were extracted by a single investigator, with periodic duplicate agreement checks. We followed up on the completion status and publication history of RCTs as of April 27, 2013, by using information from REC files and by conducting comprehensive searches for corresponding publications in electronic databases (MEDLINE, Embase, Cochrane CENTRAL register, CINAHL, AMED, Google Scholar, and topic-specific databases) and trial registers (ClinicalTrials.gov, WHO International Clinical Trials Registry Platform); details have been reported.³ Two investigators working independently and in duplicate determined whether identified publications matched the corresponding protocol.

If trial completion or publication status remained unclear, the REC in charge contacted the investigators, sending them a standardized questionnaire (eAppendix in Supplement).

Trial completion and reported reasons for discontinuation are presented as frequencies and percentages with 95% CIs, stratified by RCT sponsorship (industry vs investigator) and type of participants (patients vs healthy volunteers). We investigated factors associated with RCT discontinuation due to poor recruitment using complete-case multivariable hierarchical logistic regression with protocol-level variables as fixed effects and the 6 RECs as random intercept. Assuming different recruitment and discontinuation patterns, we excluded from this regression analysis RCTs discontinued for reasons other than poor recruitment, RCTs involving healthy volunteers only, cluster randomized trials, and pilot RCTs. We examined the following prespecified protocol variables in our model: type of control intervention (placebo or no treatment vs active intervention), center status (single vs multicenter), any reported recruitment projection (yes vs no), reported methodological or logistical support by a contract research organization or clinical trial unit (yes vs no), trial sponsor (industry vs investigator), trial design (parallel vs crossover or factorial), and planned sample size (increments of 100).³

Post hoc, we investigated trial discontinuation and type of participants (patients vs healthy volunteers) as risk factors for nonpublication of RCTs in a journal using complete-case multivariable hierarchical logistic regression considering factors associated with nonpublication such as industry sponsorship, larger planned sample size, and single-center status as additional covariables in the model, as previously suggested.⁸ We calculated unadjusted and adjusted odds ratios with 95% CIs. We used the bootstrap procedure with 100 replications to investigate the stability of the estimated standard errors and 95% CIs. In prespecified sensitivity analyses we used an alternate threshold of 80% of the target sample size achieved to define RCT discontinuation and used multiple-imputation techniques to impute missing covariable data.⁹ In a post hoc sensitivity analysis we explored differences in discontinuation rates across countries.

Data analyses were conducted using R version 3.0.1 (R Project for Statistical Computing; http://www.r-project.org/), and Stata version 13.1 (StataCorp). P < .05 (2-sided) was set as level of significance.

Between 2000 and 2003 the collaborating RECs reviewed 3819 study protocols (eFigure in Supplement). Of the 1080 potentially eligible RCT protocols, 53 were never started and 10 RCTs were ongoing (as of April 27, 2013), thus leaving 1017 RCT protocols for inclusion (894 protocols involving patients and 123 involving healthy volunteers).

Most patient RCTs had industry sponsorship and were multicenter, parallel-group, superiority trials in oncology or the cardiovascular field, with a median planned sample size of 260 patients (interquartile range, 100-606) (Table 1, eTable 2 in Supplement). RCTs involving healthy volunteers had predominantly industry sponsorship, used mostly a crossover design, and had a median planned sample size of 20 participants (interquartile range, 12-34).

Overall, 253 of 1017 RCTs (24.9% [95% CI, 22.3%-27.6%]) were discontinued, most frequently because of poor recruitment (101/1017; 9.9% [95% CI, 8.2%-12.0%]). Other reasons for discontinuation are reported in Table 2. Discontinuation information was gathered from REC files alone in 69 trials (27.3%), publications alone in 85 (33.6%), investigator survey alone in 72 (28.5%), and from combined sources in 27 (10.7%).

Assessment of REC files and identified publications showed that information about completion or publication status was missing for 299 RCTs. The RECs sent survey questionnaires to the investigators, of whom 240 responded (response rate, 80.3%). In total, 96 of 253 trial discontinuations (37.9% [95% CI, 32.0%-44.3%]) were reported to RECs. The REC files included the information about trial discontinuation for RCTs discontinued for poor recruitment in 24 of 101 RCTs (23.8% [95% CI, 16.1%-33.5%]); RCTs discontinued for administrative reasons (such as strategic decisions from companies, consequence of new recruitments from regulatory bodies, and change of workplace of principle investigators) in 15 of 39 RCTs (38.5% [95% CI, 23.8%-55.3%]); RCTs discontinued for futility in 16 of 37 RCTs (43.2% [95% CI, 27.5%-60.4%]); and RCTs discontinued for harm in 13 of 24 RCTs (54.2% [95% CI, 33.2%-73.8%]).

Among the 894 RCTs involving patients, 249 were discontinued (27.9% [95% CI, 25.0%-30.9%]), most frequently because of poor recruitment (100/894; 11.2% [95% CI, 9.2%-13.5%]). In contrast, of the 123 RCTs involving healthy volunteers, 4 were discontinued (3.3% [95% CI, 1.2%-8.6%]), 3 for administrative reasons and 1 for poor recruitment.

Trials discontinued because of poor recruitment achieved a median percentage of target sample size of 40.9% (interquartile range, 28.5%-59.8%). Only 3 RCTs recruited more than 80% of the target. Table 3 reports trial characteristics of RCTs discontinued because of poor recruitment and characteristics of completed RCTs. In multivariable analysis, industry sponsorship (8.4% vs 26.5% for investigator sponsorship; adjusted odds ratio, 0.25 [95% CI, 0.15-0.43]), and every increment of 100 patients in the planned sample size (−0.7%; adjusted odds ratio, 0.96 [95% CI, 0.92-1.00]) were associated with less frequent RCT discontinuation. Lack of documentation of any recruitment rate projection (based on retrospective or prospective screening for eligible patients) in the protocol was not associated with discontinuation due to poor recruitment. All 9 RCT protocols that reported performing a full pilot study (ie, including informed consent of patients) were, however, completed.

All sensitivity analyses left our results unchanged (eTable 3 in Supplement). We found no evidence for different discontinuation rates across the 3 countries involved (likelihood ratio test, P = .63).

Of the 1017 RCTs, 567 (55.8%, [95% CI, 52.6%-58.8%]) were published as full journal articles as of April 27, 2013. Publication information was obtained from REC files in 31 of 586 studies (5.5%), from searching electronic databases in 495 (87.3%), and from the investigator survey in 41 (7.2%). Results from all 9 RCTs stopped early for benefit were published, but results were published for only 40 of 101 trials discontinued because of poor recruitment (39.6% [95% CI, 30.2%-49.9%]) (Table 2). Multivariable analysis suggested that discontinued RCTs were more likely to remain unpublished, as were single-center RCTs, those with industry sponsorship, those involving healthy volunteers, and those with smaller sample sizes (Table 4).

Our study found that 25% of initiated RCTs were discontinued. Although discontinuation was common for RCTs involving patients (28%), it was rare for RCTs involving healthy volunteers (3%). The most commonly reported reason for RCT discontinuation was poor recruitment (10% of included RCTs). We found that trials with investigator sponsorship (vs industry sponsorship) and those with smaller planned sample sizes were at higher risk of discontinuation due to poor recruitment. Of discontinued RCTs, up to 60% remained unpublished. Trial investigators rarely informed RECs about trial discontinuation and publication.

A strength of our study was the full access to the files of all trials approved by the collaborating RECs during the study period. We systematically searched all documents to capture any relevant information about the course of the RCT such as issues of recruitment, changes in design, or modification of target sample size. We published our own study protocol,³ involved only trained methodologists in data abstraction, and, to minimize chance associations, considered only a limited number of variables in the statistical models. Our results proved robust in sensitivity analyses applying alternate assumptions and statistical approaches.

A limitation of our study was the low quality of the included RCT protocols, in particular those for RCTs with investigator sponsorship. Elements of trial methodology potentially associated with trial discontinuation due to poor recruitment, eg, recruitment projections, financial or nonfinancial incentives, and study piloting, were frequently not reported and thus limited our risk factor analysis. In our regression analysis, we could not include some well-established factors associated with nonpublication of RCTs, such as results that are statistically nonsignificant or that did not confirm study hypotheses, because we did not ask investigators of unpublished RCTs for study results.¹⁰ We used single data extraction for almost 70% of protocols, thereby potentially increasing extraction errors. However, we used prepiloted extraction forms with detailed written instructions, conducted formal calibration exercises with all data extractors, and checked extractions from a random sample of protocols at several points during the process. Agreement was good, with no more than 2 discrepancies in 30 extracted answers. All outcome data on discontinuation and publication of RCTs were verified by a second investigator. Last, we used a convenience sample of 6 RECs in 3 countries. We cannot say whether they are representative for other RECs in these or other countries; to our knowledge, they are not in any way particular.

The overall RCT discontinuation rate of 25% (253/1017) in our study is identical to estimates reported from Spain (31/123 [25%])¹¹ and Australia (50/197 [25%]).¹² Other studies reported lower rates of discontinuation from the United Kingdom (25/195 [13%]),¹³ France (34/269 [13%]),¹⁴ and Switzerland (57/508 [11%])⁸ or higher rates from the Netherlands (45/135 [33%])⁶ and the United States (37/82 [45%]).⁵ Likely explanations for these differences are (1) the method used to determine trial discontinuation (eg, surveys with 30% or more nonresponders potentially underestimate RCT discontinuation),^13,14 (2) different proportions of industry-sponsored RCTs, (3) selected types of RCTs (eg, those focusing only on drug trials),⁸ and (4) chance.

Various studies suggest that poor recruitment is common and a major problem for clinical trials.^2,13,15-17 Although poor recruitment was the predominantly reported reason for trial discontinuation in our study, the overall frequency of 10% was relatively low.³ The problem appears, however, more severe with investigator-sponsored RCTs involving patients. An analysis of 122 multicenter RCTs funded by 2 public UK health care sponsors found that trial discontinuation due to poor recruitment occurred in up to 20% of trials,⁴ which is similar to our estimate for patient RCTs with investigator sponsorship. The above-mentioned Dutch study found that recruitment was insufficient in 40% of RCTs with investigator sponsorship.⁶ Our findings suggest that sufficient funding and professional planning and conduct of RCTs (ie, common features of RCTs with industry sponsorship) are associated with more successful recruitment. Larger RCTs might be better organized from the outset (eg, within established research networks including multiple centers and experienced investigators) and better able to respond to recruitment challenges.

Overall, 56% of RCTs were published as full journal articles. This publication rate is higher than reported for most previous study cohorts.¹⁰ Reasons likely include our strict focus on RCTs and the longer follow-up period of 9 to 13 years after protocol approval for trials to get published. Those RCTs discontinued for early apparent benefit are frequently published in highly ranked, peer-reviewed journals¹⁸; all 9 RCTs from the present study that were stopped early for benefit were published as full journal articles. Our analysis suggests, however, that RCT discontinuation for other reasons is one of the major factors driving nonpublication of RCTs.

For investigator-sponsored RCTs, stakeholders including trial investigators, funding agencies, and RECs need to develop strategies to prevent trial discontinuation due to poor recruitment. Based on our data and a previous study,¹⁹ retrospective or prospective screening of patients appears to hold little promise in estimating recruitment rates, but conducting a full pilot study including consent procedures might be effective. Further research is necessary to determine the optimal length of pilot studies and to develop reliable prediction models for recruitment performance.²⁰ Recruitment should be closely monitored and contingency plans in place if it is lower than expected. Strategies to improve recruitment have been tested^21,22 and could possibly be combined, depending on the setting and area of investigation. The nonpublication of results from discontinued—or from completed—RCTs represents a waste of valid data that could contribute to systematic reviews and meta-analyses.

In this sample of trials based on RCT protocols from 6 RECs, discontinuation was common, with poor recruitment being the most frequently reported reason. Greater efforts are needed to make certain that trial discontinuation is reported to RECs and that results of discontinued trials are published.

Corresponding Author: Matthias Briel, MD, MSc, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Hebelstrasse 10, 4031 Basel, Switzerland (matthias.briel@usb.ch).

Author Contributions: Drs Kasenda and Briel had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Kasenda and von Elm contributed equally to this article.

Study concept and design: Kasenda, von Elm, Blümle, Saccilotto, Mertz, Burnand, Bucher, Guyatt, Briel.

Acquisition, analysis, and interpretation of data: All authors.

Drafting of the manuscript: Kasenda, von Elm, Carrasco-Labra, Johnston, Briel.

Critical revision of the manuscript for important intellectual content: Kasenda, von Elm, You, Blümle, Tomonaga, Saccilotto, Amstutz, Bengough, Meerphohl, Stegert, Tikkinen, Neumann, Faulhaber, Mulla, Mertz, Akl, Bassler, Busse, Ferreira-González, Lamontagne, Nordmann, Gloy, Raatz, Moja, Rosenthal, Ebrahim, Schandelmaier, Xin, Vandvik, Johnston, Walter, Burnand, Schwenkglenks, Hemkens, Bucher, Guyatt.

Statistical analysis: Kasenda.

Obtained funding: von Elm, Blümle, Bucher, Briel.

Administrative, technical, and material support: Kasenda, Blümle, Tomonaga, Saccilotto, Bengough, Stegert, Neumann, Carrasco-Labra, Faulhaber, Mertz, Nordmann, Gloy, Moja, Walter, Schwenkglenks, Hemkens.

Study supervision: von Elm, Ferreira-González, Burnand, Schwenkglenks, Guyatt, Briel.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: This study was funded by the Swiss National Science Foundation (grant 320030_133540/1) and the German Research Foundation (grant EL 544/1-2). During study preparation, Dr von Elm was supported by the Brocher Foundation. Dr You is supported by a Reseach Early Career Award from Hamilton Health Sciences. Dr Tikkinen is funded by unrestricted grants from the Finnish Cultural Foundation and the Finnish Medical Foundation. Dr Mertz is a recipient of a Research Early Career Award from Hamilton Health Sciences Foundation (Jack Hirsh Fellowship). Dr Busse is funded by a New Investigator Award from the Canadian Institutes of Health Research and Canadian Chiropractic Research Foundation. Drs Nordmann, Gloy, Raatz, Hemkens, Bucher, and Briel are supported by Santésuisse and the Gottfried and Julia Bangerter-Rhyner-Foundation.

Role of the Sponsors: The Swiss National Science Foundation, the German Research Foundation, and the Brocher Foundation had no role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.

Previous Presentation: Presented at the Seventh International Congress on Peer Review and Biomedical Publication; September 8-10, 2013; Chicago, IL.

Additional Contributions: We would like to thank the presidents and staff of participating research ethics committees from Switzerland (Basel, Lausanne, Zurich, Lucerne), Germany (Freiburg), and Canada (Hamilton, Ontario) for their continuous support and cooperation.

Correction: This article was corrected online March 11, 2014, for missing statement of equal contributorship.