Publication Bias in Editorial Decision Making

Context Studies with positive results are more likely to be published than studies with negative results (publication bias). One reason this occurs is that authors are less likely to submit manuscripts reporting negative results to journals. There is no evidence that publication bias occurs once manuscripts have been submitted to a medical journal. We assessed whether submitted manuscripts that report results of controlled trials are more likely to be published if they report positive results.

Methods Prospective cohort study of manuscripts submitted to JAMA from February 1996 through August 1999. We classified results as positive if there was a statistically significant difference (P<.05) reported for the primary outcome. Study characteristics and indicators for quality were also appraised. We included manuscripts that reported prospective studies in which participants were assigned to an intervention or comparison group and statistical tests compared differences between groups.

Results Among 745 manuscripts, 133 (17.9%) were published: 78 (20.4%) of 383 with positive results, 51 (15.0%) of 341 with negative results, and 4 (19.0%) of 21 with unclear results. The crude relative risk for publication of studies with positive results compared with negative results was 1.36 (95% confidence interval [CI], 0.99-1.88). After being adjusted simultaneously for study characteristics and quality indicators, the odds ratio for publishing studies with positive results was 1.30 (95% CI, 0.87-1.96).

Conclusions Among submitted manuscripts, we did not find a statistically significant difference in publication rates between those with positive vs negative results.

Publication bias refers to the greater likelihood that studies with positive results will be published.^1-3 Publication bias has been demonstrated in several cohort studies that followed up protocols approved by research ethics committees,^2,4,5 ongoing trials funded by the National Institutes of Health,⁶ medical doctoral dissertations,⁷ and abstracts presented at scientific meetings.^1,8

Although such studies show that researchers are more likely to submit reports with positive results, it is less clear whether journal editors are more likely to publish them. Researchers may assume that reports of research with negative results will be rejected,⁹ but few researchers associated with unpublished results actually submitted a manuscript.^2,4,10 One case-control study¹¹ examined 100 published and 100 rejected manuscripts submitted to either of 2 Spanish medical journals and found no evidence for publication bias. However, that study may have been too small to identify a meaningful association. We are unaware of any evidence that editors are more likely to publish studies with positive results.

Our objective was to examine whether publication bias operates in editorial decision making. We hypothesized that editors preferentially publish studies with positive results. We performed a prospective cohort study to learn which manuscripts submitted to JAMA were published.

About 4000 manuscripts are submitted to JAMA annually. Each is assigned to a reviewing editor. About half are rejected after internal review and half undergo external peer review. Although the reviewing editor may reject a manuscript at any point, manuscripts are accepted only by the editor-in-chief or designated deputy. Four JAMA editors participated in this investigation; none had ultimate responsibility for accepting manuscripts at JAMA.

Manuscripts were included in the cohort if they reported results of a study that was prospective, assigned participants to an intervention, had at least 1 comparison group, and used a statistical test to compare differences in outcomes between groups. Such studies were typically randomized. Eligibility for inclusion was assessed in 2 steps: an editor-investigator working in the JAMA office screened all manuscripts for adherence to the first 3 criteria. Copies of manuscripts were then sent to 2 of 3 editor-investigators, who verified that the manuscript met all 4 criteria.

Data were abstracted during the editorial process so that investigators were unaware of the publication status of the manuscript when they abstracted data. Data were abstracted independently by 2 of 3 investigators who completed data forms by consensus and forwarded them to other investigators for data entry and analysis. Other investigators independently extracted information on publication status from JAMA's database and sent it directly for data entry and analysis.

Our primary outcome was publication in JAMA. The primary independent variable was whether study results were positive, as determined by applying previously published sequential steps.¹² The primary predictor of publication was significance of results. We classified results as positive if they showed a statistically significant effect (P<.05; 95% confidence interval [CI] for difference excluding 0 or 95% CI for ratio excluding 1) on the primary outcome in that study and negative if they did not. If no primary outcome was stated or discernible, we classified results based on most outcomes. Results were unclear if they could not be classified as positive or negative, typical when many outcomes were reported and an equal number were positive and negative but none was primary.

We abstracted information on study characteristics examined by others for association with publication^{1,2,4-6,13-17} and on objective indicators of study quality and reporting transparency^2,13 (Table 1). We also recorded the clinical topic, sex and age of study participants, and the funding sources as reported in the manuscript. If an industry supplied only drugs or devices, we did not count it as having funded the study.

Before our investigation, JAMA editors agreed to take part in studies relating to peer review and editorial decision making. We did not tell editors the details of this investigation or request informed consent for participation, since their awareness might have influenced their publication decisions.¹⁸ For several years, JAMA's Instructions to Authors have told authors that their work might be included in a study.¹⁹ We did not tell authors about this investigation, because the standard editorial process was unchanged and the confidentiality of the author-editor relationship was maintained. The University of Washington's Human Subjects Review Committee approved the protocol by waiver of consent, under the condition that investigators inform the other editors about the study after all publication decisions had been made.

We estimated the number of manuscripts needed for this investigation, assuming that manuscripts with positive and negative results would be submitted in equal proportions and the overall publication rate would be 16%. Using a 2-sided test for significance of .05 and power of 0.80, 708 manuscripts would be required to detect a difference between 12% and 20% publication rates. We increased the sample size to 750 to allow for manuscripts with unclear significance of results.

Proportions of studies published were examined by significance of results and other variables. Associations between independent variables and publication were estimated with relative risks (RRs) and 95% CIs. The P values were not adjusted for multiple comparisons, and P<.05 was considered statistically significant. To adjust for several variables simultaneously, we used multiple logistic regression and calculated the odds ratio (OR) as the measure of association. Data were entered into an Access (version 2.0, Microsoft Corporation, Redmond, Wash) database and analyzed with SAS software (version 6.12, SAS Institute, Cary, NC).

Since awareness of the investigation may have influenced the decision making of the 4 JAMA editors who were also investigators, we performed additional analyses that excluded their assigned manuscripts. Editors who were not investigators were asked whether they had been aware of the investigation. They rated their awareness with a continuous scale, with 0 indicating they were unaware that any investigation was in progress, 5 indicating they knew some investigation was in progress but were unaware that it was about publication bias, and 10 indicating they knew everything about the investigation, including its hypothesis.

From February 1996 through August 1999, 13 569 manuscripts were submitted to JAMA; 745 met all inclusion criteria. Manuscripts were distributed among 20 JAMA editors for primary responsibility. Among the 745 manuscripts, 383 (51.4%) had positive results, 341 (45.7%) had negative results, and 21 (2.8%) had unclear significance of results.

JAMA published 133 (17.9%) of these manuscripts: 78 (20.4%) of 383 with positive results, 51 (15.0%) of 341 with negative results, and 4 (19.0%) of 21 with unclear significance of results. After studies with an unclear significance of results were excluded, those with positive results were not significantly more likely to be published (unadjusted RR, 1.36; 95% CI, 0.99-1.88). The association of other factors with publication is shown in Table 1.

After all study characteristics and quality indicators were adjusted for, an OR of 1.30 (95% CI, 0.87-1.96) was achieved for publishing manuscripts with positive vs negative results (Table 2). Being multicenter trials, enrolling in the United States, and having a sample size calculation were significantly associated with publication (Table 2).

After manuscripts for which any of the investigators were editors were excluded, the adjusted OR for publishing manuscripts with positive vs negative results was 1.32 (95% CI, 0.84-2.07). Of the 16 JAMA editors who were not investigators, 14 responded to the questionnaire about their awareness of the study. The median score was 4 (range, 0-10) and the mean score was 3.7, indicating editors were generally aware some investigation was in progress but were unaware of its hypothesis.

Submitted manuscripts reporting results of controlled trials are not significantly more likely to be published if they report positive results.

Studies with indicators for higher quality were more likely to be published. In a case-control study¹¹ of submitted manuscripts, higher study quality was associated with publication. In contrast, methodologic quality of abstracts, letters, short reports,¹³ and meta-analyses submitted to JAMA²⁰ was not associated with publication of full results.

We also found that studies enrolling some participants in the United States had an increased adjusted OR of publication. Previous studies^21,22 found an association between journals' nationality and the national origin of the reports they publish. However, those studies did not account for differences in submissions to journals by nationality. Another study¹⁵ found that manuscripts submitted to Gastroenterology, a US journal, were more likely to be published if authors were from the United States. However, that study did not adjust for quality of the submitted manuscripts.

In our investigation, a statistically significant difference (P<.05) in outcomes between groups was used to determine a positive study. Some definitions of positive require that the significant result favor the experimental intervention¹³ or be beneficial.¹ Another definition is that a positive result will change current thinking or the standard of care.^3,23

Researchers may interpret our results as confirming editors' publication bias, but any such bias is small compared with that demonstrated repeatedly for researchers. We found an adjusted OR of 1.30 for publication of controlled trials with positive results. A meta-analysis¹⁰ of controlled trials identified at funding or approval by an ethics committee found an OR for publication of 5.96 (95% CI, 2.33-15.22). Although authors^2,9 and others²⁴ have assumed that editors preferentially publish manuscripts with positive results, researchers are more likely to write and submit manuscripts for studies with positive results.^2,4,10

Editors may interpret our results as showing no evidence for publication bias. The Declaration of Helsinki²⁵ charges authors and publishers to make available negative and positive results of investigations. Registries of clinical trials²⁶ and online journals may facilitate the reporting of trials with negative results.²⁷ Editors must guard against basing publication on the significance of a study's results; they should also judge manuscripts on the clinical question addressed and quality of the research methods.^28,29

Assessing the influence of reviewers on editors' decisions is difficult. In one study,³⁰ reviewers were no more likely to recommend publication of manuscripts reporting positive results.

The strengths of our investigation include its large sample size, prospective design, consideration of consecutive manuscripts submitted to a large-circulation, high-impact, general medical journal, objective inclusion criteria, data abstraction by 2 independent investigators blinded to publication status, and analysis of confounding variables.

Our results may not be generalizable to manuscripts describing studies other than controlled trials. Publication bias may affect studies of various designs differently.^2,4,11,31 Our results apply specifically to the editorial process at JAMA, for a specific period and set of editors; our findings may not be generalizable to specialty journals or journals with fewer submissions, fewer editors, or lower circulation.³²

We accrued more manuscripts than required under the assumptions of our sample-size calculation. The difference in publication rates (5.4%) was smaller than we hypothesized (8%), decreasing power to detect a difference between groups. In addition, adjusting for covariates decreased the power to detect publication bias.³³ We used a conservative 2-sided test for significance, which allows for the possibility that studies with negative results are published at a higher rate than studies with positive results; using 1-sided tests for our analysis would have given statistically significant results.