Author Affiliations: Health Reviews Ltd, Rome, Italy (Dr Jefferson); UK Cochrane Centre, Oxford, England (Dr Alderson); Sideview, Princes Risborough, England (Ms Wager); and Annals of Internal Medicine, Philadelphia, Pa (Dr Davidoff).
Context Editorial peer review is widely used to select submissions to journals
for publication and is presumed to improve their usefulness. Sufficient research
on peer review has been published to consider a synthesis of its effects.
Methods To examine the evidence of the effects of editorial peer-review processes
in biomedical journals, we conducted electronic and full-text searches of
private and public databases to June 2000 and corresponded with the World
Association of Medical Editors, European Association of Science Editors, Council
of Science Editors, and researchers in the field to locate comparative studies
assessing the effects of any stage of the peer-review process that made some
attempt to control for confounding. Nineteen of 135 identified studies fulfilled
our criteria. Because of the diversity of study questions, methods, and outcomes,
we did not pool results.
Results Nine studies considered the effects of concealing reviewer/author identity.
Four studies suggested that concealing reviewer or author identity affected
review quality (mostly positively); however, methodological limitations make
their findings ambiguous, and other studies' results were either negative
or inconclusive. One study suggested that a statistical checklist can improve
report quality, but another failed to find an effect of publishing another
checklist. One study found no evidence that training referees improves performance
and another showed increased interrater reliability; both used open designs,
making interpretation difficult. Two studies of how journals communicate with
reviewers did not demonstrate any effect on review quality. One study failed
to show reviewer bias, but the findings may not be generalizable. One nonrandomized
study compared the quality of articles published in peer-reviewed vs other
journals. Two studies showed that editorial processes make articles more readable
and improve the quality of reporting, but the findings may have limited generalizability
to other journals.
Conclusions Editorial peer review, although widely used, is largely untested and
its effects are uncertain.
The use of peers to assess the work of fellow scientists goes back at
least 200 years.1 It is usually assumed to
raise the quality of the end product and to provide a mechanism for rational,
fair, and objective decision making. Despite the fact that peer review has
such a long history and is so well established, research into its effects
is a recent phenomenon. However, the body of original research on the effects
of peer review has been growing, and systematic review and synthesis may now
be possible.2 This review assesses the effects
of processes undertaken as part of editorial peer review of original research
studies submitted for paper or electronic publication in biomedical journals.
We used Cochrane methods to carry out our review, which will also be
published in the Cochrane Database of Methodology Reviews.3
A detailed description of the search strategy, sources, and terms used is
available in online Table A (available in PDF format).4- 6
We considered for inclusion all types of comparative studies in which
some attempt to control for confounding had been made. This was an attempt
to widen the criteria to collect as much evidence as possible, without collecting
studies in which obvious confounding might explain any observed effect. We
considered studies assessing the effects of any stage of the peer review process
except those solely concerned with technical editing, which was the subject
of another review.7 For example, we looked
for studies of the effects of different ways of assigning submissions to assessors
and eliciting opinions, decision-making procedures, and methods of feedback
Two reviewers (T.J. and P.A.) examined all retrieved citations. Studies
for possible inclusion were retrieved in full. The same 2 reviewers then examined
studies independently against the selection criteria. The same reviewers extracted
information on study design and outcomes. We collected descriptive information
on study quality, as reported herein. We identified 135 reports of studies
that could possibly fulfill our inclusion criteria. Nineteen of these fulfilled
our criteria and we excluded the remaining 116 studies from our review.
We did not pool the results of similar studies into a formal meta-analysis
because no 2 studies were alike and all asked a slightly different study question
or used different designs or outcome measures. We decided to group studies
by the broad issues they addressed.
A summary of the 19 included studies is shown online Table B
(available in PDF format ). A more
detailed description will be available in our Cochrane review.3
We included 9 studies assessing the effect of blinding/masking or revealing
authors' and/or reviewers' identities in some way (eg, exchanging reviews
between peer reviewers) on the quality of external reviews.8- 16
All of the studies in this group were randomized controlled trials. Five studies11- 15
reported no apparent effect in quality of reviews, time taken to review, or
tone of reviews. The 4 studies8- 10,16
that suggested that concealing reviewer or author identity affected review
quality had methodological limitations that make their outcomes ambiguous.
The most interesting methodological issue we identified in the design of the
studies was the consistent difficulty in ensuring robust blinding procedures.
This was probably a reflection of the relatively small world of editorial
peer review, especially when applied to specialist areas of knowledge. Overall,
most of the studies were inconclusive because blinding was ineffective and
confidence intervals were not sufficiently narrow to allow firm conclusions
to be made.
We included 2 studies that assessed the effects of submission checklists
on outcomes.17,18 The study by
Gardner and Bond 18 shows some benefit from
using a statistical checklist, although this finding may be of limited generalizability
given the setting of the study (BMJ) and its small
size. The other study found that publication of a checklist for economic studies
did not appear to improve submissions.17
Two studies assessed the effects of the media used by journals to communicate
with reviewers. One found that posting to the Internet may benefit authors
but does not appear to affect the quality of reviews.19
The randomized controlled trial by Neuhauser and Koran20
showed that warning reviewers of an impending review by telephone lengthened
total turnaround time, although it shortened review time.
Two studies assessed the effects of training reviewers.21,22
The results were ambiguous in the study by Callaham et al,21
showing no obvious effect, in contrast with the results of Strayhorn et al,22 which showed that interrater reliability increased.
The open design of both studies makes minimization of biases impossible, and
results are therefore difficult to interpret. Generalizability of results
beyond the context of the specific interventions tested is questionable.
A randomized controlled trial by Ernst and Resch23
assessed the presence and effects of reviewer bias. They found no evidence
of reviewer bias toward an unconventional treatment. Despite its randomized
design, bias may distort the results of this study (the response rate was
We included 1 study assessing the effects of peer review on study validity,24 which failed to show any difference, although the
small size and nonrandomized design of the study again makes generalizability
of its results uncertain.
Two studies assessed the effects of peer review on study report quality.25,26 Both studies showed a beneficial
effect, but results may again have limited generalizability because of atypical
settings (both journals studied are well resourced and keen on improving quality).
In summary, the quality of randomized studies was superior to that of
trials with open design. (Quality refers to the ability of a study to answer
its research question.) Overall, the small numbers of reviews and reviewers
involved, potentially atypical settings (with a few major journals being the
object of many studies), and many methodological weaknesses make both the
internal and external validity of these studies difficult to interpret.
Our review was limited to evidence about peer review as practiced by
biomedical journals. We discovered remarkably few well-designed studies assessing
the effects of this process. Of the studies identified, the majority were
focused on specific editorial processes, such as masking of reviewers and
authors or use of checklists. Very few examined the broader effects of peer
review. We assess the implications of this in a separate study.27
The only study that attempted to address broader issues24
is difficult to interpret because of methodological weaknesses and the likelihood
of limited generalizability. The results of our systematic review are consistent
with those of an earlier descriptive review.2
Given the widespread use of peer review and its importance, it is surprising
that so little is known of its effects. However, the research needed to address
these questions would require a well-funded and coordinated effort involving
several sectors of the scientific community as well as the cooperation of
large numbers of authors and editors, and the methodological issues in conducting
proper studies of the subject are daunting.
The 2 principal functions of peer review, filtering out incorrect or
inadequate work and improving the accuracy and clarity of published reports,
directly reflect the 2 virtues that Francis Bacon in 1605 attributed to doubt:
guarding "against errors" and causing issues that "would have been passed
by lightly without intervention" to be "attentively and carefully observed."
As such, the doubts contributed by peer review are an intrinsic and essential
part of science, layered over the critical reviews that take place during
the process of doing the work. Future research in biomedical peer review needs
to recognize these basic concepts and must consider a number of specific underlying
methodological issues. Among these are both the positive contributions of
peer review and its abuses, the complexity of biomedical science, and the
incremental nature of scientific progress. This is based on rare major discoveries
and very frequent minor ones, clarifying patterns of life and disease and
society's response to them. Moreover, most studies of biomedical journal peer
review have been concerned with manuscripts that were accepted for publication.
A number of important questions about peer review can only be answered, however,
by studying rejected manuscripts as well as those that are accepted.
Until such research is undertaken, peer review should be regarded as
an untested process with uncertain outcomes.
Jefferson T, Alderson P, Wager E, Davidoff F. Effects of Editorial Peer ReviewA Systematic Review. JAMA. 2002;287(21):2784-2786. doi:10.1001/jama.287.21.2784