Context Editorial peer review is widely used to select submissions to journals
for publication and is presumed to improve their usefulness. Sufficient research
on peer review has been published to consider a synthesis of its effects.
Methods To examine the evidence of the effects of editorial peer-review processes
in biomedical journals, we conducted electronic and full-text searches of
private and public databases to June 2000 and corresponded with the World
Association of Medical Editors, European Association of Science Editors, Council
of Science Editors, and researchers in the field to locate comparative studies
assessing the effects of any stage of the peer-review process that made some
attempt to control for confounding. Nineteen of 135 identified studies fulfilled
our criteria. Because of the diversity of study questions, methods, and outcomes,
we did not pool results.
Results Nine studies considered the effects of concealing reviewer/author identity.
Four studies suggested that concealing reviewer or author identity affected
review quality (mostly positively); however, methodological limitations make
their findings ambiguous, and other studies' results were either negative
or inconclusive. One study suggested that a statistical checklist can improve
report quality, but another failed to find an effect of publishing another
checklist. One study found no evidence that training referees improves performance
and another showed increased interrater reliability; both used open designs,
making interpretation difficult. Two studies of how journals communicate with
reviewers did not demonstrate any effect on review quality. One study failed
to show reviewer bias, but the findings may not be generalizable. One nonrandomized
study compared the quality of articles published in peer-reviewed vs other
journals. Two studies showed that editorial processes make articles more readable
and improve the quality of reporting, but the findings may have limited generalizability
to other journals.
Conclusions Editorial peer review, although widely used, is largely untested and
its effects are uncertain.
The use of peers to assess the work of fellow scientists goes back at
least 200 years.1 It is usually assumed to
raise the quality of the end product and to provide a mechanism for rational,
fair, and objective decision making. Despite the fact that peer review has
such a long history and is so well established, research into its effects
is a recent phenomenon. However, the body of original research on the effects
of peer review has been growing, and systematic review and synthesis may now
be possible.2 This review assesses the effects
of processes undertaken as part of editorial peer review of original research
studies submitted for paper or electronic publication in biomedical journals.
We used Cochrane methods to carry out our review, which will also be
published in the Cochrane Database of Methodology Reviews.3
A detailed description of the search strategy, sources, and terms used is
available in online Table A (available in PDF format).4-6
We considered for inclusion all types of comparative studies in which
some attempt to control for confounding had been made. This was an attempt
to widen the criteria to collect as much evidence as possible, without collecting
studies in which obvious confounding might explain any observed effect. We
considered studies assessing the effects of any stage of the peer review process
except those solely concerned with technical editing, which was the subject
of another review.7 For example, we looked
for studies of the effects of different ways of assigning submissions to assessors
and eliciting opinions, decision-making procedures, and methods of feedback
to authors.
Two reviewers (T.J. and P.A.) examined all retrieved citations. Studies
for possible inclusion were retrieved in full. The same 2 reviewers then examined
studies independently against the selection criteria. The same reviewers extracted
information on study design and outcomes. We collected descriptive information
on study quality, as reported herein. We identified 135 reports of studies
that could possibly fulfill our inclusion criteria. Nineteen of these fulfilled
our criteria and we excluded the remaining 116 studies from our review.
We did not pool the results of similar studies into a formal meta-analysis
because no 2 studies were alike and all asked a slightly different study question
or used different designs or outcome measures. We decided to group studies
by the broad issues they addressed.
A summary of the 19 included studies is shown online Table B
(available in PDF format ). A more
detailed description will be available in our Cochrane review.3
We included 9 studies assessing the effect of blinding/masking or revealing
authors' and/or reviewers' identities in some way (eg, exchanging reviews
between peer reviewers) on the quality of external reviews.8-16
All of the studies in this group were randomized controlled trials. Five studies11-15
reported no apparent effect in quality of reviews, time taken to review, or
tone of reviews. The 4 studies8-10,16
that suggested that concealing reviewer or author identity affected review
quality had methodological limitations that make their outcomes ambiguous.
The most interesting methodological issue we identified in the design of the
studies was the consistent difficulty in ensuring robust blinding procedures.
This was probably a reflection of the relatively small world of editorial
peer review, especially when applied to specialist areas of knowledge. Overall,
most of the studies were inconclusive because blinding was ineffective and
confidence intervals were not sufficiently narrow to allow firm conclusions
to be made.
We included 2 studies that assessed the effects of submission checklists
on outcomes.17,18 The study by
Gardner and Bond 18 shows some benefit from
using a statistical checklist, although this finding may be of limited generalizability
given the setting of the study (BMJ) and its small
size. The other study found that publication of a checklist for economic studies
did not appear to improve submissions.17
Two studies assessed the effects of the media used by journals to communicate
with reviewers. One found that posting to the Internet may benefit authors
but does not appear to affect the quality of reviews.19
The randomized controlled trial by Neuhauser and Koran20
showed that warning reviewers of an impending review by telephone lengthened
total turnaround time, although it shortened review time.
Two studies assessed the effects of training reviewers.21,22
The results were ambiguous in the study by Callaham et al,21
showing no obvious effect, in contrast with the results of Strayhorn et al,22 which showed that interrater reliability increased.
The open design of both studies makes minimization of biases impossible, and
results are therefore difficult to interpret. Generalizability of results
beyond the context of the specific interventions tested is questionable.
A randomized controlled trial by Ernst and Resch23
assessed the presence and effects of reviewer bias. They found no evidence
of reviewer bias toward an unconventional treatment. Despite its randomized
design, bias may distort the results of this study (the response rate was
61%).
We included 1 study assessing the effects of peer review on study validity,24 which failed to show any difference, although the
small size and nonrandomized design of the study again makes generalizability
of its results uncertain.
Two studies assessed the effects of peer review on study report quality.25,26 Both studies showed a beneficial
effect, but results may again have limited generalizability because of atypical
settings (both journals studied are well resourced and keen on improving quality).
In summary, the quality of randomized studies was superior to that of
trials with open design. (Quality refers to the ability of a study to answer
its research question.) Overall, the small numbers of reviews and reviewers
involved, potentially atypical settings (with a few major journals being the
object of many studies), and many methodological weaknesses make both the
internal and external validity of these studies difficult to interpret.
Our review was limited to evidence about peer review as practiced by
biomedical journals. We discovered remarkably few well-designed studies assessing
the effects of this process. Of the studies identified, the majority were
focused on specific editorial processes, such as masking of reviewers and
authors or use of checklists. Very few examined the broader effects of peer
review. We assess the implications of this in a separate study.27
The only study that attempted to address broader issues24
is difficult to interpret because of methodological weaknesses and the likelihood
of limited generalizability. The results of our systematic review are consistent
with those of an earlier descriptive review.2
Given the widespread use of peer review and its importance, it is surprising
that so little is known of its effects. However, the research needed to address
these questions would require a well-funded and coordinated effort involving
several sectors of the scientific community as well as the cooperation of
large numbers of authors and editors, and the methodological issues in conducting
proper studies of the subject are daunting.
The 2 principal functions of peer review, filtering out incorrect or
inadequate work and improving the accuracy and clarity of published reports,
directly reflect the 2 virtues that Francis Bacon in 1605 attributed to doubt:
guarding "against errors" and causing issues that "would have been passed
by lightly without intervention" to be "attentively and carefully observed."
As such, the doubts contributed by peer review are an intrinsic and essential
part of science, layered over the critical reviews that take place during
the process of doing the work. Future research in biomedical peer review needs
to recognize these basic concepts and must consider a number of specific underlying
methodological issues. Among these are both the positive contributions of
peer review and its abuses, the complexity of biomedical science, and the
incremental nature of scientific progress. This is based on rare major discoveries
and very frequent minor ones, clarifying patterns of life and disease and
society's response to them. Moreover, most studies of biomedical journal peer
review have been concerned with manuscripts that were accepted for publication.
A number of important questions about peer review can only be answered, however,
by studying rejected manuscripts as well as those that are accepted.
Until such research is undertaken, peer review should be regarded as
an untested process with uncertain outcomes.
1.Kronick DA. Peer-review in 18th-century scientific journalism.
JAMA.1990;263:1321-1322.Google Scholar 2.Overbeke J. The state of the evidence: what we know and what we don't know about
journal peer review. In: Godlee F, Jefferson T, eds. Peer Review in
Health Sciences. London, England: BMJ Books; 1999:32-45.
3.Alderson P, Davidoff F, Jefferson TO, Wager E. Editorial peer review for improving the quality of reports of biomedical
studies [Protocol for a Cochrane Methodology Review]. Oxford, England: Cochrane Library, Update Software; 2001; issue 3.
4.Rennie D. Editorial peer review in biomedical publication.
JAMA.1990;263:1317.Google Scholar 5.Rennie D, Flanagin A. The second International Congress on Peer Review in Biomedical Publication.
JAMA.1994;272:91.Google Scholar 6.Rennie D, Flanagin A. Congress on Biomedical Peer Review.
JAMA.1998;280:213.Google Scholar 7.Wager E, Middleton P. Effects of technical editing in biomedical journals: a systematic review.
JAMA.2002;287:2821-2824.Google Scholar 8.McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review.
JAMA.1990;263:1371-1376.Google Scholar 9.Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review.
JAMA.1994;272:143-146.Google Scholar 10.Jadad AR, Moore A, Carroll D.
et al. Assessing the quality of reports of randomized clinical trials.
Control Clin Trials.1996;17:1-12.Google Scholar 11.van Rooyen S, Godlee F, Evans S.
et al. Effect of blinding and unmasking on the quality of peer review.
JAMA.1998;280:234-237.Google Scholar 12.Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding peer reviewers and
asking them to sign their reports.
JAMA.1998;280:237-240.Google Scholar 13.Justice AC, Cho MK, Winker MA.
et al. Does masking author identity improve peer review quality?
JAMA.1998;280:240-242.Google Scholar 14.van Rooyen S, Godlee F, Evans S.
et al. Effect of open peer review on quality of reviews and on reviewers'
recommendations.
BMJ.1999;318:23-27.Google Scholar 15.Das Sinha S, Sahni P, Nundy S. Does exchanging comments of Indian and non-Indian reviewers improve
the quality of manuscript reviews?
Natl Med J India.1999;12:210-213.Google Scholar 16.Walsh E, Rooney M, Appleby L, Wilkinson G. Open peer review.
Br J Psychiatry.2000;176:47-51.Google Scholar 17.Jefferson T, Smith R, Yee Y.
et al. Evaluating the
BMJ guidelines for economic
submissions.
JAMA.1998;280:275-277.Google Scholar 18.Gardner MJ, Bond J. An exploratory study of statistical assessment of papers published
in the
British Medical Journal.
JAMA.1990;263:1355-1357.Google Scholar 19.Bingham CM, Higgins G, Coleman R, Van der Weyden MB. The
Medical Journal of Australia Internet
peer-review study.
Lancet.1998;352:441-445.Google Scholar 20.Neuhauser D, Koran CJ. Calling
Medical Care reviewers first.
Med Care.1989;27:664-666.Google Scholar 21.Callaham ML, Wears RL, Waeckerle JF. Effect of attendance at a training session on peer reviewer quality
and performance.
Ann Emerg Med.1998;32:318-322.Google Scholar 22.Strayhorn J, McDermott Jr JF, Tanguay P. An intervention to improve the reliability of manuscript reviews for
the
Journal of the American Academy of Child and Adolescent
Psychiatry.
Am J Psychiatry.1993;150:947-952.Google Scholar 23.Ernst E, Resch KL. Reviewer bias against the unconventional?
Complement Ther Med.1999;7:19-23.Google Scholar 24.Elvik R. Are road safety evaluation studies published in peer reviewed journals
more valid than similar studies not published in peer reviewed journals?
Accid Anal Prev.1998;30:101-118.Google Scholar 25.Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at
Annals of Internal Medicine.
Ann Intern Med.1994;121:11-21.Google Scholar 26.Pierie JP, Walvoort HC, Overbeke AJ. Readers' evaluation of effect of peer review and editing on quality
of articles in the
Nederlands Tijdschrift voor Geneeskunde.
Lancet.1996;348:1480-1483.Google Scholar 27.Jefferson T, Wager E, Davidoff F. Measuring the quality of editorial peer review.
JAMA.2002;287:2786-2790.Google Scholar