Krzyzanowska MK, Pintilie M, Tannock IF. Factors Associated With Failure to Publish Large Randomized Trials Presented at an Oncology Meeting. JAMA. 2003;290(4):495–501. doi:10.1001/jama.290.4.495
Context Large clinical trials are the criterion standard for making treatment
decisions, and nonpublication of the results of such trials can lead to bias
in the literature and contribute to inappropriate medical decisions.
Objectives To determine the rate of full publication of large randomized trials
presented at annual meetings of the American Society of Clinical Oncology
(ASCO), quantify bias against publishing nonsignificant results, and identify
factors associated with time to publication.
Design Survey of 510 abstracts from large (sample size, ≥200), phase 3,
randomized controlled trials presented at ASCO meetings between 1989 and 1998.
Trial results were classified as significant (P≤.05
for the primary outcome measure) or nonsignificant (P>.05
or not reported), and the type of presentation and sponsorship were identified.
Subsequent full publication was identified using a search of MEDLINE and EMBASE,
completed November 1, 2001; the search was updated in November 2002, using
the Cochrane Register of Controlled Trials. Authors were contacted if the
searches did not find evidence of publication.
Main Outcome Measures Publication rate at 5 years; time from presentation to full publication.
Results Of 510 randomized trials, 26% were not published in full within 5 years
after presentation at the meeting. Eighty-one percent of the studies with
significant results had been published by this time compared with 68% of the
studies with nonsignificant results (P<.001).
Studies with oral or plenary presentation were published sooner than those
not presented (P = .002), and studies with pharmaceutical
sponsorship were published sooner than studies with cooperative group sponsorship
or studies for which sponsorship was not specified (P =
.02). These factors remained significant in a multivariable model. The most
frequent reason cited by authors for not publishing was lack of time, funds,
or other resources.
Conclusions A substantial number of large phase 3 trials presented at an international
oncology meeting remain unpublished 5 years after presentation. Bias against
publishing nonsignificant results is a problem even for large randomized trials.
Nonpublication breaks the contract that investigators make with trial participants,
funding agencies, and ethics boards.
Large randomized controlled trials are the criterion standard upon which
most treatment decisions are made, and nondissemination of their results is
likely to have a negative impact on clinical practice. There are additional
ethical implications, since nonpublication of trial results breaks the contract
between investigators and study participants and funding agencies.1,2 This problem may be exaggerated if
the likelihood of publication is influenced by study outcome, a phenomenon
termed publication bias.3- 10 Not
identifying unpublished negative studies can cause bias in systematic reviews
or meta-analyses through overestimation of treatment effects7;
it also can contribute to inappropriate treatment decisions for patients encountered
in routine practice, as well as to suboptimal treatment policies.1,11 Research into reasons for nonpublication
suggests that often the authors are influenced by results of the study and
lose interest in negative studies.4,6
Few studies have addressed publication bias in relation to randomized
trials.12 Previous investigators have used
ethics committee rosters,5,6,10,13 funding
agency lists,14 trial registers,7,15 and
conference proceedings16- 18 to
identify studies that are followed up for subsequent publication. Only 1 study
has evaluated publication bias specific to oncology,17 and
it included all types of study designs. The present study is the first to
examine the rate of subsequent full publication of large, randomized cancer
trials, which provide the primary evidence upon which most therapeutic decisions
in oncology are based.
We undertook the present study to identify important randomized controlled
trials that have not yet been published in full but that have been presented
at American Society of Clinical Oncology (ASCO) meetings. Our primary objectives
were to determine the rate of subsequent full publication of randomized controlled
trials presented at ASCO meetings between 1989 and 1998; to quantify bias
against publishing nonsignificant results; and to identify factors associated
with time to publication. We also sought to generate a compendium of important
unpublished cancer trials and to establish guidelines for improved presentation
of trial results in abstracts. These will be published separately.
We reviewed proceedings of the annual ASCO meeting for the years 1989
through 1998 to identify abstracts presenting results of phase 3 randomized
controlled trials of substantial size (defined as sample size ≥200). Abstracts
presenting preliminary findings were included if results were contained within
the abstract, and abstracts presenting updated results were included if the
results pertained to the main end point of the study. Abstracts reporting
analyses of quality of life or toxicity were included only if these were predefined
primary end points. We excluded meta-analyses, overviews, abstracts that pooled
data from 2 or more trials, and abstracts presenting secondary analyses.
Information on trial characteristics was summarized using a pretested
data abstraction form designed for this study. For each abstract we collected
the following information: year of meeting; cancer type; whether the primary
end point was stated explicitly; number of participants; number of groups;
end points addressed; type of analysis; cooperative group involvement; the
use of blinding and placebo; pharmaceutical sponsorship; and the format of
presentation at the meeting (plenary, oral, poster, or published only). Since
all abstracts that are submitted for the ASCO meeting are published in the
proceedings, the "published only" category refers to abstracts that were submitted
but not accepted for presentation at the meeting.
Results of each trial were classified as significant (P≤.05 for the primary end point) or nonsignificant. Studies for
which the primary end point was not defined explicitly were classified as
significant or nonsignificant on the basis of the first end point reported.
Abstracts that provided results without reporting statistical significance
were classified as nonsignificant. The 6 equivalence trials were arbitrarily
classified as significant if the P value reported
was .05 or less and as nonsignificant otherwise. Classification of results
was undertaken without knowledge of publication status. Since there also might
be bias for or against publishing trials in which the results favored the
standard treatment,12 we undertook a second
analysis in which only trials that indicated significant benefit of the experimental
treatment were classified as positive.
Data abstraction was carried out by one investigator (M.K.K.) with a
sample of 33 abstracts and completed by a second investigator (I.F.T.) to
evaluate consistency. Differences in abstraction were resolved by consensus.
A computer-based search was used to identify full publication of the
trials reported in the abstracts. A research assistant with a background in
library science conducted the initial literature search using the PubMed,
MEDLINE, and/or EMBASE databases. The search was performed using names of
the first, second, or presenting authors, and if that did not yield a citation,
keywords contained within the title were entered. All retrieved citations
were compared with the original published meeting abstract to ensure that
they represented the same study. A separate search was undertaken by one of
the investigators for abstracts for which a citation was not found. If more
than 1 publication was identified the date of the earliest publication was
used in analysis. The initial computer search was completed November 1, 2001,
but was updated in November 2002, using the Cochrane Register of Controlled
If our search did not find a publication, authors were contacted to
confirm nonpublication and to determine their reasons for not publishing.
If an e-mail address was available from the ASCO membership directory, the
first, senior, or other author of each abstract was contacted by e-mail; otherwise
they were contacted by regular mail. Authors were asked to provide a reference
to a published article or to confirm nonpublication and to give reasons for
lack of publication. Messages to authors included a checklist of potential
reasons for not publishing: (1) insufficient priority to warrant publication;
(2) lack of time, funds, or other resources; (3) article submitted but not
accepted for publication; and (4) study incomplete with eventual intent to
publish. We assumed that all trials for which we did not receive a reply by
June 2002 were unpublished.
Kappa statistics were calculated to evaluate consistency in data abstraction
between investigators. The main outcome measures of the study were time from
presentation to publication of a full report and the probability of publication
at 5 years after presentation. A small proportion of the abstracts were published
prior to presentation at the meeting; therefore, the time frame for analysis
began 1 year preceding the date of presentation. Abstracts that did not result
in a publication were censored as of June 1, 2002; if full reports were in
press at this time, we assigned a date of publication of July 2002.
Kaplan-Meier methods were used to generate actuarial curves relating
probability of publication with time and to estimate probability of publication
at 5 years. Cox proportional models were used to investigate factors associated
with time to publication. Factors assessed in both univariate and multivariable
analyses included the year of presentation, sample size, type of result (each
classification tested separately), type of presentation (plenary, oral, poster,
or published), whether the primary end point was stated, and whether the study
was multicenter, involved a cooperative group, and/or was sponsored by the
pharmaceutical industry. Where Kaplan-Meier analysis indicated that the hazard
ratio was not constant over time for a variable that might influence publication,
an interaction between this variable and time was included in the model. In
the multivariable analysis, significant factors were chosen based on a P value less than .05 using backward, stepwise selection
techniques. All analyses were carried out using SAS version 8.2 (SAS Institute
Inc, Cary, NC).
We identified 539 abstracts that met the inclusion criteria. Twenty-nine
abstracts were subsequently excluded (28 duplicates, 1 randomized phase 2
trial). Characteristics of the remaining 510 abstracts are summarized in Table 1, which also includes the percentage
of trials in each category that remained unpublished by 5 years. The breast
was the most common cancer site (131 trials [26%]) and was the tumor site
with the highest proportion of unpublished studies (36%). Lung cancer had
the lowest proportion of unpublished studies (16%). The median sample size
of the trials was 339. In 223 (44%) trials the results were classified as
significant (P≤.05), and 183 (36%) of the studies
were classified as positive (significant results favoring experimental treatment).
Involvement of a cooperative group was identified in more than half of the
trials. Seventeen studies involving cooperative groups also explicitly acknowledged
sponsorship from the pharmaceutical industry; these were considered as cooperative
group studies in subsequent analyses. The majority of trials were presented
as either an oral presentation or a poster. The median follow-up time for
unpublished studies was 6.1 (range, 4.1-13.1) years. Trials described in 491
abstracts (96%) were either published or had 5 or more years of follow-up.
For most items (7 of 10) the agreement between investigators was very
good (agreement ≥88%, κ>0.7). For 2 items the agreement was above
70% (κ>0.5). The item that caused the highest discrepancy was type of
analysis (ie, intent-to-treat, preliminary, per protocol), with a κ
less than 0.4; this variable was removed from further analysis.
Figure 1 illustrates the steps
used to identify full publication. Our overall search strategy did not disclose
publication for 95 of the 510 abstracts (19%). Our search missed 13 publications
for the following reasons: in press (n = 3), publication subsequent to our
last search date (n = 2), publication in a non–peer-reviewed journal
(n = 1), and unknown (n = 7).
The overall probability of publication by 5 years after presentation
was 74% regardless of results. The time-dependent probability of publication
of studies with significant and nonsignificant results is presented in Figure
2 ; there is a significant difference in time to publication depending on
study result (P<.001). Eighty-one percent of trials
with significant results, and 68% of trials with nonsignificant results, were
published at 5 years. There was a similar difference between rate of publication
of trials classified as positive (favoring the experimental group) compared
with others (81% vs 70%, respectively, P = .001).
The median time to publication was 2.7 years. The difference between
median times to publication for studies with significant (2.2 years) and nonsignificant
(3.0 years) results was 0.8 years. The time by which 75% of the studies were
published was 4 years for studies with significant results and 6.7 years for
those with nonsignificant results. For both types of trial the rate of publication
decreased with time. To test the robustness of our results, we repeated the
analysis assuming publication of trials for which we did not identify a publication
and for which authors did not reply to our questionnaire. Even with this unlikely
assumption, a significant proportion of abstracts remained unpublished (24%
at 5 years); and the difference between studies with significant and nonsignificant
results persisted (17% vs 29% at 5 years).
Among the published studies, 213 (51%) were published in full within
24 months; by 5 years, 374 (90%) of the studies were published. The publication
rate beyond 5 years was negligible.
The significant univariate predictors of time to publication were: type
of result (significant vs nonsignificant, P<.001);
type of presentation (studies with oral or plenary presentation were published
in less time compared with studies that were not presented, P = .002); and sponsorship (studies with pharmaceutical sponsorship
were published in less time compared with cooperative group studies or studies
for which sponsorship was not specified, P = .02).
These factors remained significant in the multivariable model (Table 2). However, studies that were presented as a poster did not
show evidence of decreased time to publication compared with studies that
were included in the conference proceedings, but not presented, after adjustment
for type of result and sponsorship (P = .24). Since
results using the alternative scheme to classify results (positive vs negative)
were similar, they are not presented.
Time to publication was associated with the type of sponsorship (P = .02, Figure 3).
To explore the relationship between pharmaceutical sponsorship and type of
result on time to publication, we compared the times to publication of 4 groups:
pharmaceutically sponsored studies with significant results; pharmaceutically
sponsored studies with nonsignificant results; nonpharmaceutically sponsored
studies with significant results; and nonpharmaceutically sponsored, nonsignificant
studies. Figure 3 shows a difference
between the time to publication of significant and nonsignificant pharmaceutically
sponsored studies, and a difference between significant and nonsignificant
nonpharmaceutically sponsored studies. However, the difference between these
differences (ie, the interaction between type of result and sponsorship) was
not significant in the final multivariable model (P =
Thirty-four of the 40 authors who confirmed that their study had not
been published provided a reason regarding lack of publication (Table 3). The most common reason was lack of time, funds, or other
Our survey of 510 abstracts of large randomized cancer trials revealed
that 26% were not published in full 5 years after their presentation, which
suggests that a long delay in publication exists for some large trials, and
that some trials are never published. Furthermore, there was evidence of publication
bias since the actuarial rate of publication was significantly lower for studies
with significant as compared with nonsignificant results regardless of the
definition used to classify the results. This skews the literature, since
large studies with statistically nonsignificant findings contribute as much
to the totality of evidence as studies with statistically significant findings.
The publication status of studies first presented as abstracts was the
topic of a recent meta-analysis.12 The mean
publication rate in this meta-analysis was 45%. Of the 46 reports included,
5 examined full publication of randomized trials. The mean publication rate
in this subgroup was 56% and the median rate was 65%. Since this meta-analysis,
2 additional studies examining the publication of randomized trials first
presented as short reports have been published.18,19 The
publication rates in these studies were 52%18 and
59%,19 respectively. The high publication rate
observed in our study is likely due to the long follow-up and to the fact
that we restricted our study to large trials. We chose to study large phase
3 trials, for which results are less likely to be influenced by lack of power,
since a delay in or lack of publication is likely to have the greatest impact
on evidence-based practice.
The median time to publication in this study was longer than previously
reported,12 which may be due to the fact that
complete follow-up and preparation of a full report may take longer for randomized
trials than for other study designs. The median time to publication in our
study was also longer for studies with nonsignificant compared with significant
results, confirming previous findings.20 Trends
in the cumulative rate of publication over time in this study were similar
to those summarized by Scherer and Langenberg,12 with
half of the studies being published within the first 2 years following publication.
In the meta-analysis by Scherer and Langenberg,12 factors
positively associated with full publication included significant results,
sample size greater than or equal to the median, oral presentation of the
results (as opposed to poster presentation), and basic science as opposed
to clinical research. Of note, definition of what constituted a "positive"
study differed between the reports, and the association between results and
publication depended on the definition used. In reports for which any study
with significant results was classified as positive, the type of result was
a predictor of subsequent publication. In contrast, in reports for which only
studies favoring the test or new treatment were considered positive, the type
of result was not associated with full publication. In our study, we used
both of the above definitions to classify study results and found a significant
association between type of result and full publication regardless of classification
scheme used. Selection for oral presentation was a significant predictor of
full publication in our study. Whether it may be viewed as proxy for study
quality is not clear. Koren et al21 found that
the quality of rejected studies presenting negative results was higher than
the quality of rejected studies presenting positive results, suggesting that
bias may exist in selecting submitted abstracts for presentation. Sample size
was not associated with publication rate in our study, probably due to exclusion
of trials with fewer than 200 participants from our study. Since our study
was limited to randomized trials, we could not make comparisons between publication
of basic science vs clinical results.
The role of sponsorship has been explored in a limited number of studies.
External funding was associated with a higher probability of full publication
in 2 studies.6,10 The role of
pharmaceutical industry sponsorship has been addressed briefly in 2 studies,4,5 both of which found that pharmaceutically
sponsored trials were less likely to be published. Our data suggest that sponsorship
is a significant predictor of time to publication, and that pharmaceutically
sponsored studies have the fastest rate of publication. The discrepancy between
our findings and those of previous studies may be due to trends in time and/or
to the inclusion in our study of only large randomized trials, whereas the
previous reports included pilot or other small observational trials. None
of the prior studies explored the relationship between study results, sponsorship,
and subsequent publication. While previous studies have shown that the majority
of pharmaceutically sponsored studies that are published have significant
findings,22,23 in our study the
type of result did not have a differential effect on time to publication between
the pharmaceutically sponsored studies and studies that had other sponsorship.
Previous reports have indicated that lack of subsequent publication
is often due to nonsubmission rather than rejection of manuscripts4- 6,17,18,24 and
is usually due to lack of resources such as time17,18,24 or
to loss of interest in the study due to negative results.4- 6 Our
survey of authors is consistent with these findings.
Potential limitations of our study include the use of abstracts to identify
trials, only a 50% rate of response from authors of studies for which we were
unable to determine publication status, and difficulties in controlling for
study quality. The downside of using conference abstracts as a source of trials
is that authors may be less inclined to submit abstracts describing nonsignificant
results, thus leading us to underestimate the extent of publication bias.
However, ASCO includes almost all abstracts that are submitted in its proceedings,
not just the ones that are selected for presentation. While only half of the
authors we contacted replied to our letters or e-mails, the majority confirmed
that an article had not followed the abstract, suggesting that our search
methodology was accurate and that our assumption that no reply meant nonpublication
was reasonable. Moreover, when we repeated the analysis with the unlikely
assumption that no reply indicated publication, a substantial proportion of
abstracts remained unpublished, and the difference between studies with significant
and nonsignificant results persisted. Lastly, previous investigators have
found that assessment of study quality on the basis of meeting abstracts is
making it challenging to determine whether nonpublication is merely a reflection
of poor design. We only included large studies, which generally have substantial
time and effort invested in their design and execution. Also, many of the
unpublished trials in our study addressed important questions such as the
optimal adjuvant therapy for colorectal and breast cancers, and were multicenter
or sponsored by cooperative groups.
Our study has several implications. First, lack of publication of some
studies, especially those with nonsignificant results, can lead to overestimation
of treatment effects,7 which in turn can contribute
to inappropriate treatment decisions. Second, abstracts are not substitutes
for a full report. Our study team was able to contact only half of the authors
approached, leaving the publication status of 55 large phase 3 randomized
controlled trials unaccounted for. Third, nonpublication violates the agreement
that investigators make with patients, funding agencies, and ethics boards,
since there is an implicit understanding between these parties that results
of clinical trials will be disseminated. Thus, nonpublication can be seen
as a breach of trust or even as scientific misconduct.1,2 According
to our study approximately 47 000 patients with cancer participated in
studies that did not result in full publication.
Can publication bias be remedied? While methods exist both to detect
and to correct for publication bias in meta-analyses, the best solution is
prevention. The most commonly advocated solution has been the establishment
of trial registers,7 but a recent report26 indicates that many ongoing trials, especially those
sponsored by industry, are not being registered. Other advantages of trial
registers include prevention of duplicate studies and promotion of collaboration.2 Research ethics boards and funding agencies could
ensure the registration of trials: ethics approval is mandatory for all studies
involving patients, so that linking registration to approval should ensure
compliance with registration.2 Giving ethics
committees or funding agencies the responsibility of ensuring publication
also has been proposed.2 Medical journals also
might reduce publication bias through initiatives such as short reports of
negative trials, amnesty for previously unpublished research, acceptance for
publication on the basis of methodology alone, and establishment of journals
dedicated to publishing studies with nonsignificant results.27,28
In summary, bias against publishing nonsignificant results applies even
to large, randomized, phase 3 oncology trials. This can lead to incorrect
practice guidelines and inappropriate management of cancer patients. Investigators,
funding agencies, ethics boards, and publishers have a joint responsibility
to minimize this problem.