Context To determine author perception of peer review and association between
quality of review and author satisfaction.
Methods Survey between May 1999 and October 2000 of 897 corresponding authors
of manuscripts under consideration by the Annals of Emergency
Medicine and had received final editorial decisions during the study
period. A total of 576 authors (64%) returned the survey. Using a 5-point
Likert scale, the survey assessed differences in satisfaction between authors
whose manuscripts were accepted, reviewed and rejected, and rejected without
full review. The association of author satisfaction with editor's assessment
of review quality, publication decision, author sex, specialty, and publication
experience were also assessed.
Results Overall mean (SD) satisfaction score, indicated by agreement with "My
experience with the review process will make me more likely to submit to Annals in the future," was 3.1 (1.0) and was significantly
higher among authors of accepted papers (3.7 [0.9]) than among either group
of rejected papers (rejected/reviewed, 2.8 [1.0]; rejected/no review, 3.0
[0.9]; P.05). Authors whose manuscripts were reviewed
and rejected were the least satisfied with the time to decision (rejected/reviewed,
3.0 [1.2] vs accepted, 3.7 [1.0] and rejected/no review, 3.9 [0.9]; P<.05). Those whose papers were rejected without review
were the least satisfied with the letter explaining the editorial decision
(rejected/no review, 2.8 [1.2] vs accepted, 4.2 [0.7] and rejected/reviewed,
3.1 [1.2]; P<.05). Among respondents whose manuscripts
underwent full review (accepted and rejected/reviewed), overall satisfaction
was highly associated with acceptance of the manuscript for publication (odds
ratio [OR], 6.12; 95% confidence interval [CI], 3.43-10.91) but not with quality
rating of reviews (OR, 1.26; 95% CI, 0.84-1.90).
Conclusion Contributor satisfaction with peer review was modest. Authors of rejected
manuscripts were dissatisfied with the time to decision and communication
from the editor. Author satisfaction is associated with acceptance but not
with review quality.
Peer review is a resource-intensive process relying on considerable,
chiefly volunteer, effort to evaluate manuscripts for publication and craft
objective and constructive reviews. However, little is known about how author's
experience the peer-review process and, in particular, whether the quality
of the reviews affects their satisfaction. Previous studies suggest that prestige
and circulation are the factors frequently used by researchers in determining
the journal to which they submit their work, while other aspects of the process—quality
of the journal's peer-review panel, likelihood of acceptance, turnaround time,
and biostatistical review—have less influence on their choice.1,2
The Annals of Emergency Medicine has conducted
a number of studies and initiatives to monitor and improve the quality of
its review process.3,4 We conducted
a survey of authors who submitted manuscripts to Annals to understand their perceptions of the peer-review process and the
association between quality of review and author satisfaction.
Between May 1999 and October 2000, corresponding authors of manuscripts
under consideration by the Annals were sent surveys
assessing their satisfaction with the peer-review process. Eligible authors
were those whose manuscripts received a final editorial decision during this
period, regardless of when the paper had been submitted. Authors of invited
editorials, book reviews, and letters to the editor were excluded. Surveys
were mailed to contributors 1 month after the decision had been made and again
at 3 months to those who failed to respond to the initial request.
The survey was pilot tested with authors for comprehension and content
validity. Satisfaction questions used a 5-point Likert scale (1 = strongly
disagree and 5 = strongly agree) and covered the entire peer-review process,
including instructions for authors, timeliness of notification of receipt,
quality and impact of reviews, timeliness of decision, explanation for the
decision, copyediting, and timeliness of publication (if applicable). Overall
satisfaction with the peer-review experience was assessed by asking respondents
about the likelihood of submitting to this journal again. Respondents were
asked to disclose the number of previous publications for which they had been
first or corresponding author, number of different journals they had submitted
papers to, academic rank, sex, and specialty. Respondents also indicated reasons
for choosing Annals and whether it was the first
publication to which the manuscript had been submitted.
All Annals submissions sent out for peer review
receive a minimum of 2 reviews. As part of the ongoing oversight of the journal's
peer-review process, Annals' decision editors evaluate
each review and assign a single score on a previously validated 1 to 5 scale.5,6 For the purposes of this study, the
quality ratings of the reviews for each manuscript were averaged to create
a single Review Quality Rating score.
Mean satisfaction scores for each question were calculated. Analyses
of variance were used to compare mean satisfaction scores among 3 categories
of authors: those whose paper was accepted (accepted); those whose paper was
rejected after full review (rejected/reviewed); and those whose paper was
rejected by the decision editor without review (rejected/no review). If an
analysis of variance yielded a significant F test score, post hoc means comparisons
were used to examine differences between specific groups, while holding the
overall type 1 error rate at P = .05. For questions
applicable to only 2 groups, t tests were used for
comparisons. A P value of less than .05 was considered
significant.
The responses from authors whose papers were sent out for review (accepted
and rejected/reviewed) were further assessed using multiple logistic regression
to determine the association of satisfaction with review quality, publication
decision, and author characteristics (specialty, sex, and being first or corresponding
author of previous peer-review publications). For the purpose of this analysis,
responses on the 5-point Likert scale were dichotomized such that authors
were considered satisfied if they chose agree or strongly agree. All analyses
were conducted using SAS version 8.1 (SAS Institute Inc, Cary, NC).
Of 897 surveys mailed, 576 (64%) were returned. Rate of return was higher
for contributors with accepted papers (75%) than for rejected papers (61%).
Most manuscripts had been submitted in 1999 and 2000; 48 were originally submitted
in 1998, 4 in 1997, and 1 in 1996. Mean (SD) time from submission to decision
was 60.6 (29.8) days for accepted manuscripts; 65.7 (24.5) for manuscripts
reviewed and rejected; and 17.8 (16.6) for manuscripts rejected without review.
Eighty percent of respondents were men; men and women were similarly
distributed in each of the 3 author categories (Table 1). Approximately one third of contributors had been first
or corresponding author on 3 or fewer previous papers, while one third had
10 or more such papers. Although individuals with more experience as first
or corresponding author were more likely to have papers fully reviewed, the
2 groups had a similar rate of acceptance. Academic rank of authors was similarly
distributed among the 3 categories. For 84% of respondents, Annals was the first journal to which this manuscript was submitted.
The most common reasons for choosing Annals were
"most prestigious emergency medicine journal" (n = 114) and "content of paper
seemed appropriate for journal" (n = 105).
Table 1. Characteristics of Respondents*
Overall satisfaction, as indicated by agreement with "My experience
with the review process will make me more likely to submit to Annals in the future," yielded a mean (SD) score of 3.1 (1.0) on the
Likert scale (Table 2). Overall
satisfaction was higher among authors whose manuscripts were accepted than
among either group of rejected authors (post hoc means analysis, P<.05). There was no significant difference in overall satisfaction
between the 2 rejected groups.
Table 2. Satisfaction With Peer Review: Selected Questions*
Satisfaction with the timeliness of the initial editorial decision was
less for authors of manuscripts rejected after review compared with the other
decision groups (P<.05). Satisfaction with the
information provided about the reasons for rejection was less for authors
whose manuscripts were rejected without review (P<.05);
and both rejected groups were less satisfied with this information than authors
of accepted manuscripts (P<.05).
The mean (SD) Review Quality Rating score assigned by editors was 3.8
(0.7) for reviews of manuscripts that were ultimately accepted and 3.9 (0.7)
for reviews of manuscripts that were rejected. Authors of accepted papers
were more satisfied with all aspects of the reviews than rejected authors
(P<.001). As a group, authors whose manuscripts
underwent review were neutral regarding whether the reviews would affect future
papers or research (3.1 [1.0] and 2.9 [1.1], respectively). However, authors
of accepted papers were more likely to admit to a positive impact than the
reviewed and rejected authors (P<.001).
Of the 371 papers that underwent full review (accept and rejected/reviewed),
77% of these had review quality ratings. There was no association between
satisfaction with the peer-review process and the editor's quality rating
of the review (Table 3). Satisfaction
was highly associated with manuscript acceptance. Authors with more publication
experience expressed less satisfaction with the peer-review process. There
was no association between satisfaction and author's sex or academic rank.
Table 3. Association of Satisfaction With Decision on Manuscript and Quality Rating of Reviews*
Our study found that authors who submit research to a specialty journal
are only modestly satisfied with the peer-review process. Satisfaction had
a strong, positive association with acceptance of the manuscript for publication
and a smaller, inverse association with prior publication experience. Quality
of the review of the manuscript was not associated with author satisfaction.
The dramatic influence of the editorial decision on authors' perceptions
of the peer-review process is disappointing but not surprising. Authors of
manuscripts accepted by the Journal of Pediatrics
rated the editors' communications more positively than those whose manuscripts
were rejected, even though evaluations of the reviews were similar for accepted
or rejected manuscripts.7 In a study of blinded
vs unblinded review, authors gave higher ratings to reviews recommending publication
than those recommending rejection, regardless of blinding.8
In our study, satisfaction with reviews was significantly less among authors
of rejected manuscripts, despite the fact that editors rated the reviews of
rejected manuscripts as highly as reviews of accepted manuscripts.
Since only a small portion of manuscripts received by a journal will
be accepted, most hopeful authors will be unhappy with the publication decision.
Given this fact, our results suggest that rejected authors will inevitably
be critical of their reviews. Nevertheless, journal editors have a clear stake
in mitigating the sting of rejection so that authors will use the reviews
to improve their research and writing, and the journals can be ensured of
a continuous stream of quality future submissions.
Our results suggest that when the outcome is not fully satisfactory
to the author, service issues take on a significant role. Authors whose papers
were rejected after review were significantly less satisfied than others with
the time it took to reach that decision, while authors who were rejected without
review were the least satisfied with the explanation from the editor. The
parallel between our findings and the complaints of patients undergoing medical
care suggests the need for journals to focus additional energy on communication
and timeliness.9,10
Based on the modest satisfaction reported, and free text comments we
received, we doubt that authors were being cautious in relating their dissatisfaction
with the peer-review process. However, rejected authors may have hoped to
send a message to the editors by, for example, denying any impact of reviews
on their future research or papers. A survey conducted independently of Annals may have produced more tempered responses. Additionally,
the survey was conducted once at a single journal. We encourage others to
conduct similar surveys to determine if author satisfaction is different and
whether differences are associated with turnaround times, processes to ensure
quality reviews, blinding, or perhaps journal prestige.
Peer review is imperfect. The increasing number of biomedical journals,
the advent of online publication, and the increasingly interdisciplinary nature
and impact of research are forcing journals to compete for the best research.
Understanding the perceptions and concerns of authors will not only help journals
attract this research but also may provide additional insight into solutions
for the peer-review process.
1.Frank E. Authors' criteria for selecting journals.
JAMA.1994;272:163-164.Google Scholar 2.Menz HB. Publication patterns and perceptions of the Australian podiatric medical
faculty.
J Am Podiatr Med Assoc.2001;91:210-218.Google Scholar 3.Callaham ML, Wears RL, Waeckerle JF. Effect of attendance at a training session on peer reviewer quality
and performance.
Ann Emerg Med.1998;32:318-322.Google Scholar 4.Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? feasibility of using a fictitious manuscript
to evaluate peer reviewer performance.
Ann Emerg Med.1998;32:310-317.Google Scholar 5.Callaham ML, Baxt WG, Waeckerle JF, Wears RL. Reliability of editors' subjective quality ratings of peer reviews
of manuscripts.
JAMA.1998;280:229-231.Google Scholar 6.van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer
reviews of manuscripts.
J Clin Epidemiol.1999;52:625-629.Google Scholar 7.Garfunkel JM, Lawson EE, Hamrick HJ, Ulshen MH. Effect of acceptance or rejection on the author's evaluation of peer
review of medical manuscripts.
JAMA.1990;263:1376-1378.Google Scholar 8.van Rooyen S, Godlee F, Evans S.
et al. Effect of blinding and unmasking on the quality of peer review.
J Gen Intern Med.1999;14:622-624.Google Scholar 9.Larson CO, Nelson EC, Gustafson D, Batalden PB. The relationship between meeting patients' information needs and their
satisfaction with hospital care and general health status outcomes.
Int J Qual Health Care.1996;8:447-456.Google Scholar 10.Levinson W, Roter DL, Mullooly JP.
et al. Physician-patient communication: the relationship with malpractice
claims among primary care physicians and surgeons.
JAMA.1997;277:553-559.Google Scholar