Context Editorial peer review is widely used to select submissions to journals
for publication and is presumed to improve their usefulness. Sufficient research
on peer review has been published to consider a synthesis of its effects.
Methods To examine the evidence of the effects of editorial peer-review processes
in biomedical journals, we conducted electronic and full-text searches of
private and public databases to June 2000 and corresponded with the World
Association of Medical Editors, European Association of Science Editors, Council
of Science Editors, and researchers in the field to locate comparative studies
assessing the effects of any stage of the peer-review process that made some
attempt to control for confounding. Nineteen of 135 identified studies fulfilled
our criteria. Because of the diversity of study questions, methods, and outcomes,
we did not pool results.
Results Nine studies considered the effects of concealing reviewer/author identity.
Four studies suggested that concealing reviewer or author identity affected
review quality (mostly positively); however, methodological limitations make
their findings ambiguous, and other studies' results were either negative
or inconclusive. One study suggested that a statistical checklist can improve
report quality, but another failed to find an effect of publishing another
checklist. One study found no evidence that training referees improves performance
and another showed increased interrater reliability; both used open designs,
making interpretation difficult. Two studies of how journals communicate with
reviewers did not demonstrate any effect on review quality. One study failed
to show reviewer bias, but the findings may not be generalizable. One nonrandomized
study compared the quality of articles published in peer-reviewed vs other
journals. Two studies showed that editorial processes make articles more readable
and improve the quality of reporting, but the findings may have limited generalizability
to other journals.
Conclusions Editorial peer review, although widely used, is largely untested and
its effects are uncertain.