From Gastroenterology , Bethesda, Md. Ms Link is now with the Journal of Immunology , Bethesda, Md.
Context.— Reviewers increasingly are asked to review manuscripts from outside
their own country, but whether they are more likely to recommend acceptance
of such manuscripts is not known.
Objective.— To assess whether US reviewers or non-US reviewers evaluate manuscripts
differently, depending on whether the manuscripts are submitted from outside
the United States or from the United States.
Design and Setting.— A retrospective analysis of all original submissions received by Gastroenterology in 1995 and 1996. Reviewers ranked manuscripts
in 4 decision categories: accept, provisionally accept, reject with resubmission,
Main Outcome Measure.— Ranking of papers based on nationality of authors and reviewers.
Results.— The percentage of non-US manuscripts placed in each decision category
by US (n=2355) and non-US reviewers (n=1297) was nearly identical (P=.31). However, US reviewers recommended acceptance of papers submitted
by US authors more often than did non-US reviewers (P=.001).
Non-US reviewers ranked US papers slightly more favorably than non-US papers
(P=.09), while US reviewers ranked US papers much
more favorably (P=.001).
Conclusions.— Reviewers from the United States and outside the United States evaluate
non-US papers similarly and evaluate papers submitted by US authors more favorably,
with US reviewers having a significant preference for US papers.
MANY US journals cite increased numbers of international submissions.1- 3 Seventy percent of submissions
to Gastroenterology are from outside the United States.
However, non-US submissions are less likely to be published based on peer
review. These data prompted a study assessing whether US and non-US reviewers
evaluate manuscripts submitted to Gastroenterology
from outside the US differently than manuscripts submitted from US-based authors.
Using a manuscript tracking database, a retrospective study was conducted
of original research articles submitted to Gastroenterology during 1995 and 1996. Manuscripts not sent out for peer review and
revised manuscripts were excluded. The corresponding author's nationality,
each reviewer's nationality, and each reviewer's ranking of the paper were
obtained. Reviewers who did not indicate a decision ranking were excluded.
Decision rankings of papers were obtained from the reviewer forms. These forms
ask reviewers to recommend one of the following decisions on a paper: accept,
provisionally accept, reject with resubmission, or reject. Two nonblinded
reviewers are asked to review each paper.
Authors and reviewers were identified as either "US" or "non-US." Reviews
were tabulated according to author nationality, reviewer nationality, and
reviewer ranking combinations. Comparisons of reviewer recommendations and
reviewers' nationality were assessed using the χ2
test for independence (with 3 df). Reviewer recommendations
and origin of the paper were analyzed in the same manner.
International papers were assessed to determine if they were ranked
differently by US and non-US reviewers. There were 2355 US reviewers and 1297
non-US reviewers of non-US papers. The percentages of international papers
placed in each decision category by US and non-US reviewers were similar,
indicating no significant correlation between a reviewers' nationality and
the ranking of international papers (accept: 3.6% vs 3.2%; provisionally accept:
24.9% vs 24.7%; reject with resubmission: 26.8% vs 24.6%; reject: 44.7% vs
Next, US papers were analyzed to determine if there were differences
in US and non-US reviewers' rankings. There were 1174 US reviewers and 449
non-US reviewers of US papers. US reviewers placed papers in the 3 acceptance
categories more often than non-US reviewers (64% vs 55%) and also ranked US
papers significantly higher (accept: 7.0% vs 3.6%; provisionally accept: 31.3%
vs 30.5%; reject with resubmission: 26.1% vs 21.6%; reject: 35.6% vs 44.3%)
Data were analyzed to determine whether these findings indicated bias.
Reviewers' evaluations were examined to determine whether US reviewers evaluated
non-US and US papers similarly. This process was also used to determine whether
non-US reviewers evaluate the 2 sets of papers similarly. Non-US reviewers
rank papers submitted from the United States more favorably (accept: 3.6%
vs 3.2%; provisionally accept: 30.5% vs 24.7%; reject with resubmission: 21.6%
vs 24.6%; reject: 44.3% vs 47.6%; P=.09). Moreover,
US reviewers show a stronger preference for US papers than non-US reviewers
(accept: 7.0% vs 3.6%; provisionally accept: 31.3% vs 24.9%; reject with resubmission:
26.1% vs 26.8%; reject: 35.6% vs 44.7%; P=.001).
Based on logistic regression looking simultaneously at reviewer nationality
and manuscript source, the data show that the manuscript source was significant
(P=.001), with domestic papers having an odds ratio
of 1.49 for background of review and background of acceptance, while the reviewer's
nationality was not significant (P=.22) (Table 1).
Several factors prevent conclusive findings on international bias in
this study. First, non-US reviewers were pooled because there were not enough
authors and reviewers from the same country to obtain statistically significant
results. Second, nationality was based on the corresponding author's location
rather than actual nationality. Third, associate editors may choose non-US
reveiwers who have similiar training and viewpoints; however, data regarding
location of training were not available. Fourth, reviewers were not blinded.
Most important, bias cannot be unequivocally inferred without measuring the
quality of individual papers. Still, this study does provide insights into
potential variability in manuscript evaluations across national lines and
lends itself to further investigation and more complex studies.
Link AM. US and Non-US SubmissionsAn Analysis of Reviewer Bias. JAMA. 1998;280(3):246-247. doi:10.1001/jama.280.3.246