Positive-Outcome Bias and Other Limitations in the Outcome of Research Abstracts Submitted to a Scientific Meeting

Context.— Studies with positive results are more likely to be published in biomedical journals than are studies with negative results. However, many studies submitted for consideration at scientific meetings are never published in full; bias in this setting is poorly studied.

Objective.— To identify features associated with the fate of research abstracts submitted to a scientific meeting.

Design and Setting.— Prospective observational cohort, with 5-year follow-up of all research submitted for consideration to the major annual 1991 US research meeting in the specialty of emergency medicine.

Participants.— All research abstracts submitted for consideration at the meeting for possible presentation.

Main Outcome Measures.— Characteristics associated with acceptance for presentation at the meeting and subsequent publication as a full manuscript.

Results.— A total of 492 research abstracts were submitted from programs in emergency medicine and other specialies affiliated with 103 US medical schools. A total of 179 (36%) were accepted for presentation and 214 (43%) were published in 44 journals. Of the 179 abstracts accepted for presentation, 111 studies were published. Scientific quality of abstracts or prestige of the journal in which the study was eventually published did not predict either of these outcomes. The best predictors (by logistic regression) of meeting acceptance were a subjective "originality" factor (odds ratio [OR], 2.07; 95% confidence interval [CI], 1.13-3.89) and positive results (OR, 1.99; 95% CI, 1.07-3.84), and, for publication, meeting acceptance (OR, 2.49; 95% CI, 1.49-4.35) and large sample size (OR, 2.26; 95% CI, 1.23-4.31). Forty-nine percent (241) of abstracts did not report on blinding, and 24% (118) did not report on randomization. Acceptance and publication were both more likely for positive outcomes (P=.03). Funnel plots showed the classic distribution of positive-outcome ("publication") bias at each of the submission, acceptance, and publication phases. Meeting acceptance predicted publication with a sensitivity of only 51%, specificity of 71%, positive predictive value of 57%, and negative predictive value of 66%.

Conclusions.— Positive-outcome bias was evident when studies were submitted for consideration and was amplified in the selection of abstracts for both presentation and publication, neither of which was strongly related to study design or quality.

POSITIVE-OUTCOME (also known as "publication") bias refers to the fact that research with positive outcomes is much more likely to be published than that with negative outcomes.^1-4 Presentation of results in abstracts at scientific meetings is the first and often only publication for most biomedical research studies.⁵ However, the abstract selection process for meetings rarely has been studied. We, therefore, examined all research submitted for presentation at a national meeting to determine if positive-outcome bias was present in this process and what characteristics determined successful subsequent publication in a peer review journal.

The Society for Academic Emergency Medicine (SAEM) meeting is comparable to the meetings of 31 other societies of the Council of Academic Societies.⁶ Abstracts with mandatory structured formats were submitted and, independent of our study (and similar to other specialty meetings), each submission was evaluated by 5 to 7 blinded members of the SAEM screening committee (selected for their relevant expertise) and ranked on a 5-point Likert scale. Selection was based on an average score, and no journal had right of first refusal.

Four years after the 21st annual meeting of SAEM in 1991, all SAEM data were obtained by the authors and each submitted study was categorized according to design by a blinded Delphi panel. Since no established system exists for abstract classification, we modified a previously published approach.⁷ In addition, the review panel ranked each study for scientific quality and "originality" ("newsworthiness") on a Likert scale like the one previously validated.⁸ Institutional review board approval was obtained, and a detailed description of the methods used is available from the authors.

All authors' names were searched in MEDLINE in late 1995 to determine if the study had been published in any listed journal.⁹ For papers not found, the search was repeated in early 1996; if still not found, a questionnaire was sent to the authors and EMBASE and the Cochrane Collaboration databases were also searched.⁹ Journal impact factor was derived from the Science Citation Index for the year of publication.^10-13 The authors' institutions were ranked according to a system^14,15 based on total dollars of National Institutes of Health (NIH) grant support.

There is no standardized definition of positive results.¹⁶ We used one of the more common definitions—that results were positive if the studied variable produced positive (beneficial) results.^1,3,17,18 Some authors have defined positive results as those reporting statistically significant results (regardless of direction),^4,5,16,19-21 so we also used this definition.

We performed the major logistic regression analysis on those studies in which the subjects were either humans or animals, and the design was a prospective interventional trial, a prospective observational study, or a retrospective observational study. We also separately examined prospective studies with controls, excluding retrospective studies. We used a general iterative model-building strategy, as suggested by Hosmer and Lemeshow²² and Harrell,²³ assessed for goodness of fit and subjected to bootstrap validation using S-Plus version 4.0, release 3 (Mathsoft Inc, Seattle, Wash) and Harrell extensions (http://www.lib.stat.cmu.edu). Correlation coefficients were calculated using JMP software 3.2.2 for Macintosh (SAS Institute, Inc, Cary, NC). We calculated the effect size of all interventions for the subgroup of prospective studies with controls in the usual fashion, except that SDs were not available.

Five hundred research abstracts were submitted to the SAEM selection committee from a total of 144 institutions, 103 of which had formal US medical school affiliations. Eight duplicates were deleted, leaving 492 abstracts as the basis of our study. The submitting schools averaged an NIH funding rank of 55 (of all US medical schools) (95% confidence interval [CI], 49-62).

Thirty percent of studies did not state a hypothesis, and 49% did not report on blinding, 24% on randomization, and 74% on exclusion criteria. Seventy-six percent of the studies were conducted on humans, 10% on animals, and the remainder on other models. Twenty-nine percent of the studies were retrospective, 27% prospective observational with control groups, and 26% prospective interventional trials. Sixty-six percent of all submitted interventional trials had positive outcome by our initial definition, 83% by the positive P value definition, and 80% by effect size. Respective figures for all submitted observational prospective studies were 70%, 92%, and 80%.

Three hundred eighty studies met the criteria for logistic regression (see "Methods"). Most measures of scientific merit did not predict the decision to accept an abstract for presentation. Instead, this decision was most strongly related to positive results and the reviewers' subjective originality score, while controlling for institutional funding, study design, randomization, blinding, controls, exclusion criteria, and sample size (Table 1). Results were similar for the subgroup of 166 prospective studies with control groups except that sample size greater than 50 was also predictive (odds ratio [OR], 2.4; 95% CI, 1.0-6.5) and similarly controlled. Full regression results are available from the authors.

One hundred seventy-nine (36%) of the submitted abstracts were accepted for presentation at the meeting (11 reports on teaching methods were excluded). The SAEM committee scores determining acceptance correlated best with our subjective quality scale (R=0.57) and originality factor (R=0.49). Two hundred fourteen (43%) of the 492 studies submitted were published, an average of 18 months after presentation, in 44 journals with impact factors ranging from 0.23 to 24.5. A follow-up questionnaire to authors of unpublished papers was returned by 226 authors and identified 21 publications not found in MEDLINE.⁹ One hundred four (49%) of the 214 studies ultimately published were rejected for presentation at the meeting. The mean impact factor of the publishing journal did not differ for those papers rejected for the meeting vs those accepted (1.48 vs 1.19; P =.47), nor did time to publication.

One hundred forty-seven studies (70%) were published in emergency medicine specialty journals. The remaining studies were published in 39 other journals, including American Journal of Public Health, Annals of Internal Medicine, JAMA , The New England Journal of Medicine, Pediatrics, and Stroke. Of all the studies published after the meeting, 38% had been published 1 year later, 68% in 2 years, 88% in 3 years, and 95% in 4 years.

Publication of a full manuscript in the main group of 380 studies was related most strongly to abstract acceptance and sample size, controlling for the previous variables by logistic regression (Table 2). Results in the 166 prospective studies with control groups were identical. Results of these analyses did not differ using either of the 2 definitions of positive outcome ("Methods").

Data allowing calculation of effect size was reported in only 122 (66%) of 186 prospective studies. Positive-outcome bias was evident for all studies submitted before any screening (Figure 1). Submissions with less positive effect size were then disproportionately rejected for presentation. At the level of publication of full manuscripts, the same bias again appeared. A funnel plot (Figure 1) shows the absence of expected negative effect sizes at low sample size, which is the hallmark of positive-outcome bias.

Numerical testing confirms the funnel plots. The mean effect size of all submitted papers was 0.71 (95% CI, 0.40-1.01). The mean effect size of papers accepted for the meeting was 0.92 vs 0.45 for those rejected. The mean effect size of papers eventually published was 0.96 vs 0.45 for those never published (P=.03, Kruskal-Wallis analysis of variance). Effect size contributed much more to acceptance or publication than study sample size.

Our study was limited to 1 specialty, but 103 medical schools contributed and the publication rate was comparable with 31 other academic society meetings.⁶ Research from this meeting was published in 44 journals, 39 of them outside this specialty. The emergency medicine literature, the abstracts we studied, and the general medicine literature are identical in the proportion of studies with positive outcomes.¹ Effect size data were available for only a minority of studies, and most studies were not randomized controlled trials. These subgroups might not be representative of all 492 abstracts, but the results between groups were similar and consistent.

Presentation of scientific studies at meetings is an important part of the dissemination of knowledge, but half of these studies appear only as abstracts and never undergo any other peer review.⁵ Whether the abbreviated peer review used to select abstracts for meetings actually identifies scientific merit is unknown, yet abstracts are cited as often as fully published papers.²⁴

We reviewed all submitted research, not just studies accepted for presentation, assessing those characteristics previously suggested to predict publication.^2-5,14,15 Our results show that acceptance of an abstract for presentation at the meeting was not strongly related to study design, methods, sample size, or even a subjective quality score. Instead, a subjective "originality" factor and presence of positive results best predicted acceptance (ORs, 2.07 and 1.99, respectively), regardless of study design.

Publication as a full manuscript was best predicted by whether the abstract had been accepted at the meeting (OR, 2.49) and large sample size (OR, 2.26), again independent of study design or scientific quality. Positive studies were preferentially accepted during both the acceptance and publication decisions (P=.03), which is illustrated in the funnel plots (Figure 1).

Positive-outcome bias has been documented previously in publication of full journal articles, but not in detail at the meeting acceptance level.^3,4,18 Four studies examined, in limited ways, the publication of studies after acceptance for presentation at meetings.^5,25-27 Abstracts submitted on the single subject of gestational exposure to cocaine demonstrated positive-outcome bias in acceptance.²⁸ The impact of sample size and positive outcome on acceptance and publication was reported for cancer abstracts.²⁴

Our study examined all submitted research from a broad cross-section of institutions, with subsequent publication in a broad variety of journals. Positive-effect (or publication) bias was already present when studies were first submitted for consideration (Figure 1). Presumably this was due to authors who did not complete or submit smaller studies with negative effects, perhaps after experiencing a tradition of publication bias by meeting selection committees and scientific journals.

The selection process for presentation at the meeting further increased this positive-outcome bias (Figure 1). Logistic regression showed that an intangible "originality" ("newsworthiness") factor and positive outcome were more strongly associated with acceptance than traditional measures of scientific quality, such as study design, randomization, sample size, and blinding (Table 1 and Table 2).

Positive-outcome bias appeared again in the selection process for publication in a journal. Full publication was best predicted by acceptance at the meeting and study size, rather than study methods or quality. Our results confirm a smaller study of the cancer literature, which did not control for scientific quality.²⁴

A number of potential solutions, such as trials registries, have been proposed to remedy positive-outcome bias.^21,29,30 We offer one more solution: that all studies submitted to scientific meetings be published as abstracts, indicating whether or not they were chosen to be presented. This might encourage researchers to submit studies with negative findings, and readers and researchers could more easily identify the entire spectrum of research. Journals might adopt a similar practice, publishing the abstracts of all submitted manuscripts.

Despite the mandatory structured format, 49% of SAEM abstracts failed to report adequately about blinding, 74% about exclusion criteria, 24% about randomization, and 14% about sample size. Perhaps because these deficiencies made the merit of the research difficult to evaluate, acceptance for presentation at the meeting predicted publication as a full manuscript with a sensitivity of only 51%, a specificity of 71%, a positive predictive value of 57%, and a negative predictive value of 66%.