From the Division of Emergency Medicine (Drs Weber, Callaham, and Young), University of California, San Francisco; the Division of Emergency Medicine, University of Florida Health Center, Jacksonville (Dr Wears); the Department of Emergency Medicine, University of North Carolina, Chapel Hill (Dr Barton); and the Department of Emergency Medicine, Highland Hospital, Alameda County Medical Center, Oakland, Calif (Dr Young). Dr Young is now with Sacred Heart Medical Center, Eugene, Ore.
Context.— Studies with positive results are more likely to be published in biomedical
journals than are studies with negative results. However, many studies submitted
for consideration at scientific meetings are never published in full; bias
in this setting is poorly studied.
Objective.— To identify features associated with the fate of research abstracts
submitted to a scientific meeting.
Design and Setting.— Prospective observational cohort, with 5-year follow-up of all research
submitted for consideration to the major annual 1991 US research meeting in
the specialty of emergency medicine.
Participants.— All research abstracts submitted for consideration at the meeting for
Main Outcome Measures.— Characteristics associated with acceptance for presentation at the meeting
and subsequent publication as a full manuscript.
Results.— A total of 492 research abstracts were submitted from programs in emergency
medicine and other specialies affiliated with 103 US medical schools. A total
of 179 (36%) were accepted for presentation and 214 (43%) were published in
44 journals. Of the 179 abstracts accepted for presentation, 111 studies were
published. Scientific quality of abstracts or prestige of the journal in which
the study was eventually published did not predict either of these outcomes.
The best predictors (by logistic regression) of meeting acceptance were a
subjective "originality" factor (odds ratio [OR], 2.07; 95% confidence interval
[CI], 1.13-3.89) and positive results (OR, 1.99; 95% CI, 1.07-3.84), and,
for publication, meeting acceptance (OR, 2.49; 95% CI, 1.49-4.35) and large
sample size (OR, 2.26; 95% CI, 1.23-4.31). Forty-nine percent (241) of abstracts
did not report on blinding, and 24% (118) did not report on randomization.
Acceptance and publication were both more likely for positive outcomes (P=.03). Funnel plots showed the classic distribution of
positive-outcome ("publication") bias at each of the submission, acceptance,
and publication phases. Meeting acceptance predicted publication with a sensitivity
of only 51%, specificity of 71%, positive predictive value of 57%, and negative
predictive value of 66%.
Conclusions.— Positive-outcome bias was evident when studies were submitted for consideration
and was amplified in the selection of abstracts for both presentation and
publication, neither of which was strongly related to study design or quality.
POSITIVE-OUTCOME (also known as "publication") bias refers to the fact
that research with positive outcomes is much more likely to be published than
that with negative outcomes.1- 4
Presentation of results in abstracts at scientific meetings is the first and
often only publication for most biomedical research studies.5
However, the abstract selection process for meetings rarely has been studied.
We, therefore, examined all research submitted for presentation at a national
meeting to determine if positive-outcome bias was present in this process
and what characteristics determined successful subsequent publication in a
peer review journal.
The Society for Academic Emergency Medicine (SAEM) meeting is comparable
to the meetings of 31 other societies of the Council of Academic Societies.6 Abstracts with mandatory structured formats were submitted
and, independent of our study (and similar to other specialty meetings), each
submission was evaluated by 5 to 7 blinded members of the SAEM screening committee
(selected for their relevant expertise) and ranked on a 5-point Likert scale.
Selection was based on an average score, and no journal had right of first
Four years after the 21st annual meeting of SAEM in 1991, all SAEM data
were obtained by the authors and each submitted study was categorized according
to design by a blinded Delphi panel. Since no established system exists for
abstract classification, we modified a previously published approach.7 In addition, the review panel ranked each study for
scientific quality and "originality" ("newsworthiness") on a Likert scale
like the one previously validated.8 Institutional
review board approval was obtained, and a detailed description of the methods
used is available from the authors.
All authors' names were searched in MEDLINE in late 1995 to determine
if the study had been published in any listed journal.9
For papers not found, the search was repeated in early 1996; if still not
found, a questionnaire was sent to the authors and EMBASE and the Cochrane
Collaboration databases were also searched.9
Journal impact factor was derived from the Science Citation
Index for the year of publication.10- 13
The authors' institutions were ranked according to a system14,15
based on total dollars of National Institutes of Health (NIH) grant support.
There is no standardized definition of positive results.16
We used one of the more common definitions—that results were positive
if the studied variable produced positive (beneficial) results.1,3,17,18
Some authors have defined positive results as those
reporting statistically significant results (regardless of direction),4,5,16,19- 21
so we also used this definition.
We performed the major logistic regression analysis on those studies
in which the subjects were either humans or animals, and the design was a
prospective interventional trial, a prospective observational study, or a
retrospective observational study. We also separately examined prospective
studies with controls, excluding retrospective studies. We used a general
iterative model-building strategy, as suggested by Hosmer and Lemeshow22 and Harrell,23 assessed
for goodness of fit and subjected to bootstrap validation using S-Plus version
4.0, release 3 (Mathsoft Inc, Seattle, Wash) and Harrell extensions
Correlation coefficients were calculated using JMP software 3.2.2 for Macintosh
(SAS Institute, Inc, Cary, NC). We calculated the effect size of all interventions
for the subgroup of prospective studies with controls in the usual fashion,
except that SDs were not available.
Five hundred research abstracts were submitted to the SAEM selection
committee from a total of 144 institutions, 103 of which had formal US medical
school affiliations. Eight duplicates were deleted, leaving 492 abstracts
as the basis of our study. The submitting schools averaged an NIH funding
rank of 55 (of all US medical schools) (95% confidence interval [CI], 49-62).
Thirty percent of studies did not state a hypothesis, and 49% did not
report on blinding, 24% on randomization, and 74% on exclusion criteria. Seventy-six
percent of the studies were conducted on humans, 10% on animals, and the remainder
on other models. Twenty-nine percent of the studies were retrospective, 27%
prospective observational with control groups, and 26% prospective interventional
trials. Sixty-six percent of all submitted interventional trials had positive
outcome by our initial definition, 83% by the positive P value definition, and 80% by effect size. Respective figures for
all submitted observational prospective studies were 70%, 92%, and 80%.
Three hundred eighty studies met the criteria for logistic regression
(see "Methods"). Most measures of scientific merit did not predict the decision
to accept an abstract for presentation. Instead, this decision was most strongly
related to positive results and the reviewers' subjective originality score,
while controlling for institutional funding, study design, randomization,
blinding, controls, exclusion criteria, and sample size (Table 1). Results were similar for the subgroup of 166 prospective
studies with control groups except that sample size greater than 50 was also
predictive (odds ratio [OR], 2.4; 95% CI, 1.0-6.5) and similarly controlled.
Full regression results are available from the authors.
One hundred seventy-nine (36%) of the submitted abstracts were accepted
for presentation at the meeting (11 reports on teaching methods were excluded).
The SAEM committee scores determining acceptance correlated best with our
subjective quality scale (R=0.57) and originality
factor (R=0.49). Two hundred fourteen (43%) of the
492 studies submitted were published, an average of 18 months after presentation,
in 44 journals with impact factors ranging from 0.23 to 24.5. A follow-up
questionnaire to authors of unpublished papers was returned by 226 authors
and identified 21 publications not found in MEDLINE.9
One hundred four (49%) of the 214 studies ultimately published were rejected
for presentation at the meeting. The mean impact factor of the publishing
journal did not differ for those papers rejected for the meeting vs those
accepted (1.48 vs 1.19; P =.47), nor did time to
One hundred forty-seven studies (70%) were published in emergency medicine
specialty journals. The remaining studies were published in 39 other journals,
including American Journal of Public Health, Annals of Internal Medicine, JAMA , The New England Journal of Medicine, Pediatrics,
and Stroke. Of all the studies published after the
meeting, 38% had been published 1 year later, 68% in 2 years, 88% in 3 years,
and 95% in 4 years.
Publication of a full manuscript in the main group of 380 studies was
related most strongly to abstract acceptance and sample size, controlling
for the previous variables by logistic regression (Table 2). Results in the 166 prospective studies with control groups
were identical. Results of these analyses did not differ using either of the
2 definitions of positive outcome ("Methods").
Data allowing calculation of effect size was reported in only 122 (66%)
of 186 prospective studies. Positive-outcome bias was evident for all studies
submitted before any screening (Figure 1). Submissions with less positive effect size were then disproportionately
rejected for presentation. At the level of publication of full manuscripts,
the same bias again appeared. A funnel plot (Figure 1) shows the absence of expected negative effect sizes at
low sample size, which is the hallmark of positive-outcome bias.
Numerical testing confirms the funnel plots. The mean effect size of
all submitted papers was 0.71 (95% CI, 0.40-1.01). The mean effect size of
papers accepted for the meeting was 0.92 vs 0.45 for those rejected. The mean
effect size of papers eventually published was 0.96 vs 0.45 for those never
published (P=.03, Kruskal-Wallis analysis of variance).
Effect size contributed much more to acceptance or publication than study
Our study was limited to 1 specialty, but 103 medical schools contributed
and the publication rate was comparable with 31 other academic society meetings.6 Research from this meeting was published in 44 journals,
39 of them outside this specialty. The emergency medicine literature, the
abstracts we studied, and the general medicine literature are identical in
the proportion of studies with positive outcomes.1
Effect size data were available for only a minority of studies, and most studies
were not randomized controlled trials. These subgroups might not be representative
of all 492 abstracts, but the results between groups were similar and consistent.
Presentation of scientific studies at meetings is an important part
of the dissemination of knowledge, but half of these studies appear only as
abstracts and never undergo any other peer review.5
Whether the abbreviated peer review used to select abstracts for meetings
actually identifies scientific merit is unknown, yet abstracts are cited as
often as fully published papers.24
We reviewed all submitted research, not just studies accepted for presentation,
assessing those characteristics previously suggested to predict publication.2- 5,14,15
Our results show that acceptance of an abstract for presentation at the meeting
was not strongly related to study design, methods, sample size, or even a
subjective quality score. Instead, a subjective "originality" factor and presence
of positive results best predicted acceptance (ORs, 2.07 and 1.99, respectively),
regardless of study design.
Publication as a full manuscript was best predicted by whether the abstract
had been accepted at the meeting (OR, 2.49) and large sample size (OR, 2.26),
again independent of study design or scientific quality. Positive studies
were preferentially accepted during both the acceptance and publication decisions
(P=.03), which is illustrated in the funnel plots
Positive-outcome bias has been documented previously in publication
of full journal articles, but not in detail at the meeting acceptance level.3,4,18 Four studies examined,
in limited ways, the publication of studies after acceptance for presentation
at meetings.5,25- 27
Abstracts submitted on the single subject of gestational exposure to cocaine
demonstrated positive-outcome bias in acceptance.28
The impact of sample size and positive outcome on acceptance and publication
was reported for cancer abstracts.24
Our study examined all submitted research from a broad cross-section
of institutions, with subsequent publication in a broad variety of journals.
Positive-effect (or publication) bias was already present when studies were
first submitted for consideration (Figure
1). Presumably this was due to authors who did not complete or submit
smaller studies with negative effects, perhaps after experiencing a tradition
of publication bias by meeting selection committees and scientific journals.
The selection process for presentation at the meeting further increased
this positive-outcome bias (Figure 1).
Logistic regression showed that an intangible "originality" ("newsworthiness")
factor and positive outcome were more strongly associated with acceptance
than traditional measures of scientific quality, such as study design, randomization,
sample size, and blinding (Table 1
and Table 2).
Positive-outcome bias appeared again in the selection process for publication
in a journal. Full publication was best predicted by acceptance at the meeting
and study size, rather than study methods or quality. Our results confirm
a smaller study of the cancer literature, which did not control for scientific
A number of potential solutions, such as trials registries, have been
proposed to remedy positive-outcome bias.21,29,30
We offer one more solution: that all studies submitted to scientific meetings
be published as abstracts, indicating whether or not they were chosen to be
presented. This might encourage researchers to submit studies with negative
findings, and readers and researchers could more easily identify the entire
spectrum of research. Journals might adopt a similar practice, publishing
the abstracts of all submitted manuscripts.
Despite the mandatory structured format, 49% of SAEM abstracts failed
to report adequately about blinding, 74% about exclusion criteria, 24% about
randomization, and 14% about sample size. Perhaps because these deficiencies
made the merit of the research difficult to evaluate, acceptance for presentation
at the meeting predicted publication as a full manuscript with a sensitivity
of only 51%, a specificity of 71%, a positive predictive value of 57%, and
a negative predictive value of 66%.
Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive-Outcome Bias and Other Limitations in the Outcome of Research
Abstracts Submitted to a Scientific Meeting. JAMA. 1998;280(3):254-257. doi:10.1001/jama.280.3.254