Author Perception of Peer Review: Impact of Review Quality and Acceptance on Satisfaction

Context To determine author perception of peer review and association between quality of review and author satisfaction.

Methods Survey between May 1999 and October 2000 of 897 corresponding authors of manuscripts under consideration by the Annals of Emergency Medicine and had received final editorial decisions during the study period. A total of 576 authors (64%) returned the survey. Using a 5-point Likert scale, the survey assessed differences in satisfaction between authors whose manuscripts were accepted, reviewed and rejected, and rejected without full review. The association of author satisfaction with editor's assessment of review quality, publication decision, author sex, specialty, and publication experience were also assessed.

Results Overall mean (SD) satisfaction score, indicated by agreement with "My experience with the review process will make me more likely to submit to Annals in the future," was 3.1 (1.0) and was significantly higher among authors of accepted papers (3.7 [0.9]) than among either group of rejected papers (rejected/reviewed, 2.8 [1.0]; rejected/no review, 3.0 [0.9]; P.05). Authors whose manuscripts were reviewed and rejected were the least satisfied with the time to decision (rejected/reviewed, 3.0 [1.2] vs accepted, 3.7 [1.0] and rejected/no review, 3.9 [0.9]; P<.05). Those whose papers were rejected without review were the least satisfied with the letter explaining the editorial decision (rejected/no review, 2.8 [1.2] vs accepted, 4.2 [0.7] and rejected/reviewed, 3.1 [1.2]; P<.05). Among respondents whose manuscripts underwent full review (accepted and rejected/reviewed), overall satisfaction was highly associated with acceptance of the manuscript for publication (odds ratio [OR], 6.12; 95% confidence interval [CI], 3.43-10.91) but not with quality rating of reviews (OR, 1.26; 95% CI, 0.84-1.90).

Conclusion Contributor satisfaction with peer review was modest. Authors of rejected manuscripts were dissatisfied with the time to decision and communication from the editor. Author satisfaction is associated with acceptance but not with review quality.

Peer review is a resource-intensive process relying on considerable, chiefly volunteer, effort to evaluate manuscripts for publication and craft objective and constructive reviews. However, little is known about how author's experience the peer-review process and, in particular, whether the quality of the reviews affects their satisfaction. Previous studies suggest that prestige and circulation are the factors frequently used by researchers in determining the journal to which they submit their work, while other aspects of the process—quality of the journal's peer-review panel, likelihood of acceptance, turnaround time, and biostatistical review—have less influence on their choice.^1,2

The Annals of Emergency Medicine has conducted a number of studies and initiatives to monitor and improve the quality of its review process.^3,4 We conducted a survey of authors who submitted manuscripts to Annals to understand their perceptions of the peer-review process and the association between quality of review and author satisfaction.

Between May 1999 and October 2000, corresponding authors of manuscripts under consideration by the Annals were sent surveys assessing their satisfaction with the peer-review process. Eligible authors were those whose manuscripts received a final editorial decision during this period, regardless of when the paper had been submitted. Authors of invited editorials, book reviews, and letters to the editor were excluded. Surveys were mailed to contributors 1 month after the decision had been made and again at 3 months to those who failed to respond to the initial request.

The survey was pilot tested with authors for comprehension and content validity. Satisfaction questions used a 5-point Likert scale (1 = strongly disagree and 5 = strongly agree) and covered the entire peer-review process, including instructions for authors, timeliness of notification of receipt, quality and impact of reviews, timeliness of decision, explanation for the decision, copyediting, and timeliness of publication (if applicable). Overall satisfaction with the peer-review experience was assessed by asking respondents about the likelihood of submitting to this journal again. Respondents were asked to disclose the number of previous publications for which they had been first or corresponding author, number of different journals they had submitted papers to, academic rank, sex, and specialty. Respondents also indicated reasons for choosing Annals and whether it was the first publication to which the manuscript had been submitted.

All Annals submissions sent out for peer review receive a minimum of 2 reviews. As part of the ongoing oversight of the journal's peer-review process, Annals' decision editors evaluate each review and assign a single score on a previously validated 1 to 5 scale.^5,6 For the purposes of this study, the quality ratings of the reviews for each manuscript were averaged to create a single Review Quality Rating score.

Mean satisfaction scores for each question were calculated. Analyses of variance were used to compare mean satisfaction scores among 3 categories of authors: those whose paper was accepted (accepted); those whose paper was rejected after full review (rejected/reviewed); and those whose paper was rejected by the decision editor without review (rejected/no review). If an analysis of variance yielded a significant F test score, post hoc means comparisons were used to examine differences between specific groups, while holding the overall type 1 error rate at P = .05. For questions applicable to only 2 groups, t tests were used for comparisons. A P value of less than .05 was considered significant.

The responses from authors whose papers were sent out for review (accepted and rejected/reviewed) were further assessed using multiple logistic regression to determine the association of satisfaction with review quality, publication decision, and author characteristics (specialty, sex, and being first or corresponding author of previous peer-review publications). For the purpose of this analysis, responses on the 5-point Likert scale were dichotomized such that authors were considered satisfied if they chose agree or strongly agree. All analyses were conducted using SAS version 8.1 (SAS Institute Inc, Cary, NC).

Of 897 surveys mailed, 576 (64%) were returned. Rate of return was higher for contributors with accepted papers (75%) than for rejected papers (61%). Most manuscripts had been submitted in 1999 and 2000; 48 were originally submitted in 1998, 4 in 1997, and 1 in 1996. Mean (SD) time from submission to decision was 60.6 (29.8) days for accepted manuscripts; 65.7 (24.5) for manuscripts reviewed and rejected; and 17.8 (16.6) for manuscripts rejected without review.

Eighty percent of respondents were men; men and women were similarly distributed in each of the 3 author categories (Table 1). Approximately one third of contributors had been first or corresponding author on 3 or fewer previous papers, while one third had 10 or more such papers. Although individuals with more experience as first or corresponding author were more likely to have papers fully reviewed, the 2 groups had a similar rate of acceptance. Academic rank of authors was similarly distributed among the 3 categories. For 84% of respondents, Annals was the first journal to which this manuscript was submitted. The most common reasons for choosing Annals were "most prestigious emergency medicine journal" (n = 114) and "content of paper seemed appropriate for journal" (n = 105).

Overall satisfaction, as indicated by agreement with "My experience with the review process will make me more likely to submit to Annals in the future," yielded a mean (SD) score of 3.1 (1.0) on the Likert scale (Table 2). Overall satisfaction was higher among authors whose manuscripts were accepted than among either group of rejected authors (post hoc means analysis, P<.05). There was no significant difference in overall satisfaction between the 2 rejected groups.

Satisfaction with the timeliness of the initial editorial decision was less for authors of manuscripts rejected after review compared with the other decision groups (P<.05). Satisfaction with the information provided about the reasons for rejection was less for authors whose manuscripts were rejected without review (P<.05); and both rejected groups were less satisfied with this information than authors of accepted manuscripts (P<.05).

The mean (SD) Review Quality Rating score assigned by editors was 3.8 (0.7) for reviews of manuscripts that were ultimately accepted and 3.9 (0.7) for reviews of manuscripts that were rejected. Authors of accepted papers were more satisfied with all aspects of the reviews than rejected authors (P<.001). As a group, authors whose manuscripts underwent review were neutral regarding whether the reviews would affect future papers or research (3.1 [1.0] and 2.9 [1.1], respectively). However, authors of accepted papers were more likely to admit to a positive impact than the reviewed and rejected authors (P<.001).

Of the 371 papers that underwent full review (accept and rejected/reviewed), 77% of these had review quality ratings. There was no association between satisfaction with the peer-review process and the editor's quality rating of the review (Table 3). Satisfaction was highly associated with manuscript acceptance. Authors with more publication experience expressed less satisfaction with the peer-review process. There was no association between satisfaction and author's sex or academic rank.

Our study found that authors who submit research to a specialty journal are only modestly satisfied with the peer-review process. Satisfaction had a strong, positive association with acceptance of the manuscript for publication and a smaller, inverse association with prior publication experience. Quality of the review of the manuscript was not associated with author satisfaction.

The dramatic influence of the editorial decision on authors' perceptions of the peer-review process is disappointing but not surprising. Authors of manuscripts accepted by the Journal of Pediatrics rated the editors' communications more positively than those whose manuscripts were rejected, even though evaluations of the reviews were similar for accepted or rejected manuscripts.⁷ In a study of blinded vs unblinded review, authors gave higher ratings to reviews recommending publication than those recommending rejection, regardless of blinding.⁸ In our study, satisfaction with reviews was significantly less among authors of rejected manuscripts, despite the fact that editors rated the reviews of rejected manuscripts as highly as reviews of accepted manuscripts.

Since only a small portion of manuscripts received by a journal will be accepted, most hopeful authors will be unhappy with the publication decision. Given this fact, our results suggest that rejected authors will inevitably be critical of their reviews. Nevertheless, journal editors have a clear stake in mitigating the sting of rejection so that authors will use the reviews to improve their research and writing, and the journals can be ensured of a continuous stream of quality future submissions.

Our results suggest that when the outcome is not fully satisfactory to the author, service issues take on a significant role. Authors whose papers were rejected after review were significantly less satisfied than others with the time it took to reach that decision, while authors who were rejected without review were the least satisfied with the explanation from the editor. The parallel between our findings and the complaints of patients undergoing medical care suggests the need for journals to focus additional energy on communication and timeliness.^9,10

Based on the modest satisfaction reported, and free text comments we received, we doubt that authors were being cautious in relating their dissatisfaction with the peer-review process. However, rejected authors may have hoped to send a message to the editors by, for example, denying any impact of reviews on their future research or papers. A survey conducted independently of Annals may have produced more tempered responses. Additionally, the survey was conducted once at a single journal. We encourage others to conduct similar surveys to determine if author satisfaction is different and whether differences are associated with turnaround times, processes to ensure quality reviews, blinding, or perhaps journal prestige.

Peer review is imperfect. The increasing number of biomedical journals, the advent of online publication, and the increasingly interdisciplinary nature and impact of research are forcing journals to compete for the best research. Understanding the perceptions and concerns of authors will not only help journals attract this research but also may provide additional insight into solutions for the peer-review process.