Factors Associated With Failure to Publish Large Randomized Trials Presented at an Oncology Meeting

Context Large clinical trials are the criterion standard for making treatment decisions, and nonpublication of the results of such trials can lead to bias in the literature and contribute to inappropriate medical decisions.

Objectives To determine the rate of full publication of large randomized trials presented at annual meetings of the American Society of Clinical Oncology (ASCO), quantify bias against publishing nonsignificant results, and identify factors associated with time to publication.

Design Survey of 510 abstracts from large (sample size, ≥200), phase 3, randomized controlled trials presented at ASCO meetings between 1989 and 1998. Trial results were classified as significant (P≤.05 for the primary outcome measure) or nonsignificant (P>.05 or not reported), and the type of presentation and sponsorship were identified. Subsequent full publication was identified using a search of MEDLINE and EMBASE, completed November 1, 2001; the search was updated in November 2002, using the Cochrane Register of Controlled Trials. Authors were contacted if the searches did not find evidence of publication.

Main Outcome Measures Publication rate at 5 years; time from presentation to full publication.

Results Of 510 randomized trials, 26% were not published in full within 5 years after presentation at the meeting. Eighty-one percent of the studies with significant results had been published by this time compared with 68% of the studies with nonsignificant results (P<.001). Studies with oral or plenary presentation were published sooner than those not presented (P = .002), and studies with pharmaceutical sponsorship were published sooner than studies with cooperative group sponsorship or studies for which sponsorship was not specified (P = .02). These factors remained significant in a multivariable model. The most frequent reason cited by authors for not publishing was lack of time, funds, or other resources.

Conclusions A substantial number of large phase 3 trials presented at an international oncology meeting remain unpublished 5 years after presentation. Bias against publishing nonsignificant results is a problem even for large randomized trials. Nonpublication breaks the contract that investigators make with trial participants, funding agencies, and ethics boards.

Large randomized controlled trials are the criterion standard upon which most treatment decisions are made, and nondissemination of their results is likely to have a negative impact on clinical practice. There are additional ethical implications, since nonpublication of trial results breaks the contract between investigators and study participants and funding agencies.^1,2 This problem may be exaggerated if the likelihood of publication is influenced by study outcome, a phenomenon termed publication bias.^3-10 Not identifying unpublished negative studies can cause bias in systematic reviews or meta-analyses through overestimation of treatment effects⁷; it also can contribute to inappropriate treatment decisions for patients encountered in routine practice, as well as to suboptimal treatment policies.^1,11 Research into reasons for nonpublication suggests that often the authors are influenced by results of the study and lose interest in negative studies.^4,6

Few studies have addressed publication bias in relation to randomized trials.¹² Previous investigators have used ethics committee rosters,^5,6,10,13 funding agency lists,¹⁴ trial registers,^7,15 and conference proceedings^16-18 to identify studies that are followed up for subsequent publication. Only 1 study has evaluated publication bias specific to oncology,¹⁷ and it included all types of study designs. The present study is the first to examine the rate of subsequent full publication of large, randomized cancer trials, which provide the primary evidence upon which most therapeutic decisions in oncology are based.

We undertook the present study to identify important randomized controlled trials that have not yet been published in full but that have been presented at American Society of Clinical Oncology (ASCO) meetings. Our primary objectives were to determine the rate of subsequent full publication of randomized controlled trials presented at ASCO meetings between 1989 and 1998; to quantify bias against publishing nonsignificant results; and to identify factors associated with time to publication. We also sought to generate a compendium of important unpublished cancer trials and to establish guidelines for improved presentation of trial results in abstracts. These will be published separately.

We reviewed proceedings of the annual ASCO meeting for the years 1989 through 1998 to identify abstracts presenting results of phase 3 randomized controlled trials of substantial size (defined as sample size ≥200). Abstracts presenting preliminary findings were included if results were contained within the abstract, and abstracts presenting updated results were included if the results pertained to the main end point of the study. Abstracts reporting analyses of quality of life or toxicity were included only if these were predefined primary end points. We excluded meta-analyses, overviews, abstracts that pooled data from 2 or more trials, and abstracts presenting secondary analyses.

Information on trial characteristics was summarized using a pretested data abstraction form designed for this study. For each abstract we collected the following information: year of meeting; cancer type; whether the primary end point was stated explicitly; number of participants; number of groups; end points addressed; type of analysis; cooperative group involvement; the use of blinding and placebo; pharmaceutical sponsorship; and the format of presentation at the meeting (plenary, oral, poster, or published only). Since all abstracts that are submitted for the ASCO meeting are published in the proceedings, the "published only" category refers to abstracts that were submitted but not accepted for presentation at the meeting.

Results of each trial were classified as significant (P≤.05 for the primary end point) or nonsignificant. Studies for which the primary end point was not defined explicitly were classified as significant or nonsignificant on the basis of the first end point reported. Abstracts that provided results without reporting statistical significance were classified as nonsignificant. The 6 equivalence trials were arbitrarily classified as significant if the P value reported was .05 or less and as nonsignificant otherwise. Classification of results was undertaken without knowledge of publication status. Since there also might be bias for or against publishing trials in which the results favored the standard treatment,¹² we undertook a second analysis in which only trials that indicated significant benefit of the experimental treatment were classified as positive.

Data abstraction was carried out by one investigator (M.K.K.) with a sample of 33 abstracts and completed by a second investigator (I.F.T.) to evaluate consistency. Differences in abstraction were resolved by consensus.

A computer-based search was used to identify full publication of the trials reported in the abstracts. A research assistant with a background in library science conducted the initial literature search using the PubMed, MEDLINE, and/or EMBASE databases. The search was performed using names of the first, second, or presenting authors, and if that did not yield a citation, keywords contained within the title were entered. All retrieved citations were compared with the original published meeting abstract to ensure that they represented the same study. A separate search was undertaken by one of the investigators for abstracts for which a citation was not found. If more than 1 publication was identified the date of the earliest publication was used in analysis. The initial computer search was completed November 1, 2001, but was updated in November 2002, using the Cochrane Register of Controlled Trials.

If our search did not find a publication, authors were contacted to confirm nonpublication and to determine their reasons for not publishing. If an e-mail address was available from the ASCO membership directory, the first, senior, or other author of each abstract was contacted by e-mail; otherwise they were contacted by regular mail. Authors were asked to provide a reference to a published article or to confirm nonpublication and to give reasons for lack of publication. Messages to authors included a checklist of potential reasons for not publishing: (1) insufficient priority to warrant publication; (2) lack of time, funds, or other resources; (3) article submitted but not accepted for publication; and (4) study incomplete with eventual intent to publish. We assumed that all trials for which we did not receive a reply by June 2002 were unpublished.

Kappa statistics were calculated to evaluate consistency in data abstraction between investigators. The main outcome measures of the study were time from presentation to publication of a full report and the probability of publication at 5 years after presentation. A small proportion of the abstracts were published prior to presentation at the meeting; therefore, the time frame for analysis began 1 year preceding the date of presentation. Abstracts that did not result in a publication were censored as of June 1, 2002; if full reports were in press at this time, we assigned a date of publication of July 2002.

Kaplan-Meier methods were used to generate actuarial curves relating probability of publication with time and to estimate probability of publication at 5 years. Cox proportional models were used to investigate factors associated with time to publication. Factors assessed in both univariate and multivariable analyses included the year of presentation, sample size, type of result (each classification tested separately), type of presentation (plenary, oral, poster, or published), whether the primary end point was stated, and whether the study was multicenter, involved a cooperative group, and/or was sponsored by the pharmaceutical industry. Where Kaplan-Meier analysis indicated that the hazard ratio was not constant over time for a variable that might influence publication, an interaction between this variable and time was included in the model. In the multivariable analysis, significant factors were chosen based on a P value less than .05 using backward, stepwise selection techniques. All analyses were carried out using SAS version 8.2 (SAS Institute Inc, Cary, NC).

We identified 539 abstracts that met the inclusion criteria. Twenty-nine abstracts were subsequently excluded (28 duplicates, 1 randomized phase 2 trial). Characteristics of the remaining 510 abstracts are summarized in Table 1, which also includes the percentage of trials in each category that remained unpublished by 5 years. The breast was the most common cancer site (131 trials [26%]) and was the tumor site with the highest proportion of unpublished studies (36%). Lung cancer had the lowest proportion of unpublished studies (16%). The median sample size of the trials was 339. In 223 (44%) trials the results were classified as significant (P≤.05), and 183 (36%) of the studies were classified as positive (significant results favoring experimental treatment). Involvement of a cooperative group was identified in more than half of the trials. Seventeen studies involving cooperative groups also explicitly acknowledged sponsorship from the pharmaceutical industry; these were considered as cooperative group studies in subsequent analyses. The majority of trials were presented as either an oral presentation or a poster. The median follow-up time for unpublished studies was 6.1 (range, 4.1-13.1) years. Trials described in 491 abstracts (96%) were either published or had 5 or more years of follow-up.

For most items (7 of 10) the agreement between investigators was very good (agreement ≥88%, κ>0.7). For 2 items the agreement was above 70% (κ>0.5). The item that caused the highest discrepancy was type of analysis (ie, intent-to-treat, preliminary, per protocol), with a κ less than 0.4; this variable was removed from further analysis.

Figure 1 illustrates the steps used to identify full publication. Our overall search strategy did not disclose publication for 95 of the 510 abstracts (19%). Our search missed 13 publications for the following reasons: in press (n = 3), publication subsequent to our last search date (n = 2), publication in a non–peer-reviewed journal (n = 1), and unknown (n = 7).

The overall probability of publication by 5 years after presentation was 74% regardless of results. The time-dependent probability of publication of studies with significant and nonsignificant results is presented in Figure 2 ; there is a significant difference in time to publication depending on study result (P<.001). Eighty-one percent of trials with significant results, and 68% of trials with nonsignificant results, were published at 5 years. There was a similar difference between rate of publication of trials classified as positive (favoring the experimental group) compared with others (81% vs 70%, respectively, P = .001).

The median time to publication was 2.7 years. The difference between median times to publication for studies with significant (2.2 years) and nonsignificant (3.0 years) results was 0.8 years. The time by which 75% of the studies were published was 4 years for studies with significant results and 6.7 years for those with nonsignificant results. For both types of trial the rate of publication decreased with time. To test the robustness of our results, we repeated the analysis assuming publication of trials for which we did not identify a publication and for which authors did not reply to our questionnaire. Even with this unlikely assumption, a significant proportion of abstracts remained unpublished (24% at 5 years); and the difference between studies with significant and nonsignificant results persisted (17% vs 29% at 5 years).

Among the published studies, 213 (51%) were published in full within 24 months; by 5 years, 374 (90%) of the studies were published. The publication rate beyond 5 years was negligible.

The significant univariate predictors of time to publication were: type of result (significant vs nonsignificant, P<.001); type of presentation (studies with oral or plenary presentation were published in less time compared with studies that were not presented, P = .002); and sponsorship (studies with pharmaceutical sponsorship were published in less time compared with cooperative group studies or studies for which sponsorship was not specified, P = .02). These factors remained significant in the multivariable model (Table 2). However, studies that were presented as a poster did not show evidence of decreased time to publication compared with studies that were included in the conference proceedings, but not presented, after adjustment for type of result and sponsorship (P = .24). Since results using the alternative scheme to classify results (positive vs negative) were similar, they are not presented.

Time to publication was associated with the type of sponsorship (P = .02, Figure 3). To explore the relationship between pharmaceutical sponsorship and type of result on time to publication, we compared the times to publication of 4 groups: pharmaceutically sponsored studies with significant results; pharmaceutically sponsored studies with nonsignificant results; nonpharmaceutically sponsored studies with significant results; and nonpharmaceutically sponsored, nonsignificant studies. Figure 3 shows a difference between the time to publication of significant and nonsignificant pharmaceutically sponsored studies, and a difference between significant and nonsignificant nonpharmaceutically sponsored studies. However, the difference between these differences (ie, the interaction between type of result and sponsorship) was not significant in the final multivariable model (P = .49).

Thirty-four of the 40 authors who confirmed that their study had not been published provided a reason regarding lack of publication (Table 3). The most common reason was lack of time, funds, or other resources.

Our survey of 510 abstracts of large randomized cancer trials revealed that 26% were not published in full 5 years after their presentation, which suggests that a long delay in publication exists for some large trials, and that some trials are never published. Furthermore, there was evidence of publication bias since the actuarial rate of publication was significantly lower for studies with significant as compared with nonsignificant results regardless of the definition used to classify the results. This skews the literature, since large studies with statistically nonsignificant findings contribute as much to the totality of evidence as studies with statistically significant findings.

The publication status of studies first presented as abstracts was the topic of a recent meta-analysis.¹² The mean publication rate in this meta-analysis was 45%. Of the 46 reports included, 5 examined full publication of randomized trials. The mean publication rate in this subgroup was 56% and the median rate was 65%. Since this meta-analysis, 2 additional studies examining the publication of randomized trials first presented as short reports have been published.^18,19 The publication rates in these studies were 52%¹⁸ and 59%,¹⁹ respectively. The high publication rate observed in our study is likely due to the long follow-up and to the fact that we restricted our study to large trials. We chose to study large phase 3 trials, for which results are less likely to be influenced by lack of power, since a delay in or lack of publication is likely to have the greatest impact on evidence-based practice.

The median time to publication in this study was longer than previously reported,¹² which may be due to the fact that complete follow-up and preparation of a full report may take longer for randomized trials than for other study designs. The median time to publication in our study was also longer for studies with nonsignificant compared with significant results, confirming previous findings.²⁰ Trends in the cumulative rate of publication over time in this study were similar to those summarized by Scherer and Langenberg,¹² with half of the studies being published within the first 2 years following publication.

In the meta-analysis by Scherer and Langenberg,¹² factors positively associated with full publication included significant results, sample size greater than or equal to the median, oral presentation of the results (as opposed to poster presentation), and basic science as opposed to clinical research. Of note, definition of what constituted a "positive" study differed between the reports, and the association between results and publication depended on the definition used. In reports for which any study with significant results was classified as positive, the type of result was a predictor of subsequent publication. In contrast, in reports for which only studies favoring the test or new treatment were considered positive, the type of result was not associated with full publication. In our study, we used both of the above definitions to classify study results and found a significant association between type of result and full publication regardless of classification scheme used. Selection for oral presentation was a significant predictor of full publication in our study. Whether it may be viewed as proxy for study quality is not clear. Koren et al²¹ found that the quality of rejected studies presenting negative results was higher than the quality of rejected studies presenting positive results, suggesting that bias may exist in selecting submitted abstracts for presentation. Sample size was not associated with publication rate in our study, probably due to exclusion of trials with fewer than 200 participants from our study. Since our study was limited to randomized trials, we could not make comparisons between publication of basic science vs clinical results.

The role of sponsorship has been explored in a limited number of studies. External funding was associated with a higher probability of full publication in 2 studies.^6,10 The role of pharmaceutical industry sponsorship has been addressed briefly in 2 studies,^4,5 both of which found that pharmaceutically sponsored trials were less likely to be published. Our data suggest that sponsorship is a significant predictor of time to publication, and that pharmaceutically sponsored studies have the fastest rate of publication. The discrepancy between our findings and those of previous studies may be due to trends in time and/or to the inclusion in our study of only large randomized trials, whereas the previous reports included pilot or other small observational trials. None of the prior studies explored the relationship between study results, sponsorship, and subsequent publication. While previous studies have shown that the majority of pharmaceutically sponsored studies that are published have significant findings,^22,23 in our study the type of result did not have a differential effect on time to publication between the pharmaceutically sponsored studies and studies that had other sponsorship.

Previous reports have indicated that lack of subsequent publication is often due to nonsubmission rather than rejection of manuscripts^4-6,17,18,24 and is usually due to lack of resources such as time^17,18,24 or to loss of interest in the study due to negative results.^4-6 Our survey of authors is consistent with these findings.

Potential limitations of our study include the use of abstracts to identify trials, only a 50% rate of response from authors of studies for which we were unable to determine publication status, and difficulties in controlling for study quality. The downside of using conference abstracts as a source of trials is that authors may be less inclined to submit abstracts describing nonsignificant results, thus leading us to underestimate the extent of publication bias. However, ASCO includes almost all abstracts that are submitted in its proceedings, not just the ones that are selected for presentation. While only half of the authors we contacted replied to our letters or e-mails, the majority confirmed that an article had not followed the abstract, suggesting that our search methodology was accurate and that our assumption that no reply meant nonpublication was reasonable. Moreover, when we repeated the analysis with the unlikely assumption that no reply indicated publication, a substantial proportion of abstracts remained unpublished, and the difference between studies with significant and nonsignificant results persisted. Lastly, previous investigators have found that assessment of study quality on the basis of meeting abstracts is difficult,^18,19,25 thus making it challenging to determine whether nonpublication is merely a reflection of poor design. We only included large studies, which generally have substantial time and effort invested in their design and execution. Also, many of the unpublished trials in our study addressed important questions such as the optimal adjuvant therapy for colorectal and breast cancers, and were multicenter or sponsored by cooperative groups.

Our study has several implications. First, lack of publication of some studies, especially those with nonsignificant results, can lead to overestimation of treatment effects,⁷ which in turn can contribute to inappropriate treatment decisions. Second, abstracts are not substitutes for a full report. Our study team was able to contact only half of the authors approached, leaving the publication status of 55 large phase 3 randomized controlled trials unaccounted for. Third, nonpublication violates the agreement that investigators make with patients, funding agencies, and ethics boards, since there is an implicit understanding between these parties that results of clinical trials will be disseminated. Thus, nonpublication can be seen as a breach of trust or even as scientific misconduct.^1,2 According to our study approximately 47 000 patients with cancer participated in studies that did not result in full publication.

Can publication bias be remedied? While methods exist both to detect and to correct for publication bias in meta-analyses, the best solution is prevention. The most commonly advocated solution has been the establishment of trial registers,⁷ but a recent report²⁶ indicates that many ongoing trials, especially those sponsored by industry, are not being registered. Other advantages of trial registers include prevention of duplicate studies and promotion of collaboration.² Research ethics boards and funding agencies could ensure the registration of trials: ethics approval is mandatory for all studies involving patients, so that linking registration to approval should ensure compliance with registration.² Giving ethics committees or funding agencies the responsibility of ensuring publication also has been proposed.² Medical journals also might reduce publication bias through initiatives such as short reports of negative trials, amnesty for previously unpublished research, acceptance for publication on the basis of methodology alone, and establishment of journals dedicated to publishing studies with nonsignificant results.^27,28

In summary, bias against publishing nonsignificant results applies even to large, randomized, phase 3 oncology trials. This can lead to incorrect practice guidelines and inappropriate management of cancer patients. Investigators, funding agencies, ethics boards, and publishers have a joint responsibility to minimize this problem.