US and Non-US Submissions: An Analysis of Reviewer Bias

Context.— Reviewers increasingly are asked to review manuscripts from outside their own country, but whether they are more likely to recommend acceptance of such manuscripts is not known.

Objective.— To assess whether US reviewers or non-US reviewers evaluate manuscripts differently, depending on whether the manuscripts are submitted from outside the United States or from the United States.

Design and Setting.— A retrospective analysis of all original submissions received by Gastroenterology in 1995 and 1996. Reviewers ranked manuscripts in 4 decision categories: accept, provisionally accept, reject with resubmission, or reject.

Main Outcome Measure.— Ranking of papers based on nationality of authors and reviewers.

Results.— The percentage of non-US manuscripts placed in each decision category by US (n=2355) and non-US reviewers (n=1297) was nearly identical (P=.31). However, US reviewers recommended acceptance of papers submitted by US authors more often than did non-US reviewers (P=.001). Non-US reviewers ranked US papers slightly more favorably than non-US papers (P=.09), while US reviewers ranked US papers much more favorably (P=.001).

Conclusions.— Reviewers from the United States and outside the United States evaluate non-US papers similarly and evaluate papers submitted by US authors more favorably, with US reviewers having a significant preference for US papers.

MANY US journals cite increased numbers of international submissions.^1-3 Seventy percent of submissions to Gastroenterology are from outside the United States. However, non-US submissions are less likely to be published based on peer review. These data prompted a study assessing whether US and non-US reviewers evaluate manuscripts submitted to Gastroenterology from outside the US differently than manuscripts submitted from US-based authors.

Using a manuscript tracking database, a retrospective study was conducted of original research articles submitted to Gastroenterology during 1995 and 1996. Manuscripts not sent out for peer review and revised manuscripts were excluded. The corresponding author's nationality, each reviewer's nationality, and each reviewer's ranking of the paper were obtained. Reviewers who did not indicate a decision ranking were excluded. Decision rankings of papers were obtained from the reviewer forms. These forms ask reviewers to recommend one of the following decisions on a paper: accept, provisionally accept, reject with resubmission, or reject. Two nonblinded reviewers are asked to review each paper.

Authors and reviewers were identified as either "US" or "non-US." Reviews were tabulated according to author nationality, reviewer nationality, and reviewer ranking combinations. Comparisons of reviewer recommendations and reviewers' nationality were assessed using the χ² test for independence (with 3 df). Reviewer recommendations and origin of the paper were analyzed in the same manner.

International papers were assessed to determine if they were ranked differently by US and non-US reviewers. There were 2355 US reviewers and 1297 non-US reviewers of non-US papers. The percentages of international papers placed in each decision category by US and non-US reviewers were similar, indicating no significant correlation between a reviewers' nationality and the ranking of international papers (accept: 3.6% vs 3.2%; provisionally accept: 24.9% vs 24.7%; reject with resubmission: 26.8% vs 24.6%; reject: 44.7% vs 47.6%) (P=.31).

Next, US papers were analyzed to determine if there were differences in US and non-US reviewers' rankings. There were 1174 US reviewers and 449 non-US reviewers of US papers. US reviewers placed papers in the 3 acceptance categories more often than non-US reviewers (64% vs 55%) and also ranked US papers significantly higher (accept: 7.0% vs 3.6%; provisionally accept: 31.3% vs 30.5%; reject with resubmission: 26.1% vs 21.6%; reject: 35.6% vs 44.3%) (P=.001).

Data were analyzed to determine whether these findings indicated bias. Reviewers' evaluations were examined to determine whether US reviewers evaluated non-US and US papers similarly. This process was also used to determine whether non-US reviewers evaluate the 2 sets of papers similarly. Non-US reviewers rank papers submitted from the United States more favorably (accept: 3.6% vs 3.2%; provisionally accept: 30.5% vs 24.7%; reject with resubmission: 21.6% vs 24.6%; reject: 44.3% vs 47.6%; P=.09). Moreover, US reviewers show a stronger preference for US papers than non-US reviewers (accept: 7.0% vs 3.6%; provisionally accept: 31.3% vs 24.9%; reject with resubmission: 26.1% vs 26.8%; reject: 35.6% vs 44.7%; P=.001).

Based on logistic regression looking simultaneously at reviewer nationality and manuscript source, the data show that the manuscript source was significant (P=.001), with domestic papers having an odds ratio of 1.49 for background of review and background of acceptance, while the reviewer's nationality was not significant (P=.22) (Table 1).

Several factors prevent conclusive findings on international bias in this study. First, non-US reviewers were pooled because there were not enough authors and reviewers from the same country to obtain statistically significant results. Second, nationality was based on the corresponding author's location rather than actual nationality. Third, associate editors may choose non-US reveiwers who have similiar training and viewpoints; however, data regarding location of training were not available. Fourth, reviewers were not blinded. Most important, bias cannot be unequivocally inferred without measuring the quality of individual papers. Still, this study does provide insights into potential variability in manuscript evaluations across national lines and lends itself to further investigation and more complex studies.