Perceived Value of Providing Peer Reviewers With Abstracts and Preprints of Related Published and Unpublished Papers

Context.— Many journals provide peer reviewers with written instructions regarding review criteria, such as the originality of results, but little research has been done to investigate ways to improve or facilitate the peer review task.

Objective.— To assess the value that peer reviewers place on receipt of supplemental materials (eg, abstracts of related papers and preprints of related unpublished manuscripts).

Design.— Questionnaire survey sent to all 733 peer reviewers recruited by the Journal of the National Cancer Institute to review 356 manuscripts consecutively sent out for review from February 24, 1997, through January 16, 1998. The inclusion of supplemental materials with manuscript review packages was optional.

Main Outcome Measure.— The peer reviewers' assessment of the actual or potential usefulness of supplemental materials on the performance of peer review.

Results.— A total of 481 (66%) of 733 questionnaires were returned. Of the 471 respondents' questionnaires that could be used, 217 (46%) indicated that they received abstracts, and 44 (10%) of 458 respondents indicated that they received preprints. Higher proportions of peer reviewers who received supplemental materials than those who had not received them felt that they were (or would be) useful to them when reviewing the manuscript (63% [95% confidence interval (CI), 57%-69%] vs 45% [95% CI, 38%-52%]; P<.001) and to the peer review process in general (80% [95% CI, 75%-85%] vs 64% [95% CI, 58%-70%]; P<.001).

Conclusion.— The majority of respondents indicated that supplemental materials helped (or would have helped) them evaluate manuscripts and valued them more highly when they actually received them.

MANUSCRIPTS SUBMITTED to a journal for review and possible publication may contain results that are not novel or original, either because very similar results have been published by other investigators or because the authors themselves have already published the results. Following the lead of Ingelfinger,¹ most journals will not knowingly publish results that have already been reported. Manuscripts may also contain results that should be published as part of a larger, more complete study. Angell and Relman² have eloquently discussed what they termed "redundant publication" and its costs to the scientific community. Frank³ conducted a survey of the top 100 journals and found that only 72% asked peer reviewers to assess the originality and novelty of submitted manuscripts. Peer reviewers may not spend much time reviewing a manuscript.⁴ Little research appears to have been done to design and study interventions that might aid peer reviewers in their performance of peer review.

Since 1993, senior editors at the Journal of the National Cancer Institute have searched the MEDLINE biomedical literature database for published papers related to the topics of many of the manuscripts being sent out for peer review. When such a literature search was done, it may have been done to educate the editor on the topic, to identify related papers and/or appropriate peer reviewers, or to provide a service to the editorial board members and peer reviewers. The current study was conducted to obtain an objective assessment of the usefulness of these supplemental materials to the peer review process from the peer reviewers themselves.

A questionnaire was designed to prospectively measure peer reviewers' assessment of the actual or potential usefulness of abstracts and preprints of related papers to their individual review of a manuscript and to the peer review process. The questionnaires were included as part of the normal peer reviewer packages for consecutive manuscripts sent for peer review; recipients were informed that the results of this study might be published.

During the period covered by this study, the review of related literature by use of MEDLINE or other means (eg, other literature databases or library resources) by senior editors before peer reviewers were selected was optional; the completion of this task by senior editors was typically dependent on their workloads. The performance of the literature search and printing of abstracts typically required about 1 hour per manuscript. Senior editors also checked whether related unpublished (ie, labeled "in press" or "submitted") manuscripts had been provided (as specified in the Information for Authors section of the Journal of the National Cancer Institute and other journals²) or cited; when deemed relevant to the review of a submitted manuscript, copies of such manuscripts (provided by the authors or obtained by request) were sent as confidential supplemental documents to the peer reviewers.

The first 130 (18%) of 733 questionnaires sent out had 16 questions; the 603 questionnaires sent out subsequently had 1 additional question, which asked the peer reviewer if he or she typically searches MEDLINE or another literature database when reviewing a manuscript (questionnaires are available on request). There were no statistically significant differences in the responses before or after addition of the 17th question (data not shown). Eight questions assessed specific issues related to the reviewers' perception of the usefulness of abstracts, 6 questions assessed specific issues related to their perception of the usefulness of preprints, and 2 questions assessed their perception of the overall usefulness of abstracts and preprints to their individual peer review and to the process in general. Reviewers were invited to write comments at the end of the questionnaire.

The reviewers were asked to mark their responses to each question on a Likert-type scale⁵ that ranged from 1 to 5, with 1 defined as definitely yes, 3 defined as not sure or no opinion, and 5 defined as definitely no. Any response of 1.0 through 2.0 was considered an affirmative or positive assessment of value in the analyses that follow.

The numbers of responses with values from 1.0 through 2.0 were divided by the total number of responses to a given question; percentages and 95% confidence intervals for those percentages were calculated. The statistical significance between groups was calculated by use of the 2-sided χ² test; P<.05 was considered to indicate a statistically significant difference.

From February 24, 1997, through January 16, 1998, 733 questionnaires were sent to peer reviewers; 481 (66%) were returned by February 16, 1998, and 471 (64%) were usable (10 questionnaires were returned blank). These questionnaires were included as part of the normal peer review packages for 356 consecutive manuscripts sent for peer review. Of the 471 respondents, 217 (46%) indicated that they received abstracts, and 44 (10%) of 458 respondents indicated that they received supplemental "in press" or "submitted" manuscripts. Of 388 respondents, 199 (51%) indicated that they perform their own literature search for papers relevant to a manuscript.

The peer reviewers' perceived assessments of 5 specific aspects of the usefulness of the supplemental materials (ie, abstracts and/or preprints) are presented in Table 1. The majority of respondents indicated that abstracts of related papers did (or would) help them judge aspects of the originality of the submitted manuscript (examination, duplication, and fragmentation; defined in the table footnote); the proportions of respondents who indicated that abstracts would be of positive value were similar whether they indicated that they received supplemental materials or not. Data from responses to the 2 questions regarding the overall usefulness of supplemental materials are also presented. Significantly higher proportions of respondents who indicated that they received supplemental materials than those who indicated that they had not received them felt that such materials would be useful to them and to the peer review process in general.

Seventy-six (35%) of 215 respondents who received abstracts and 16 (32%) of 50 respondents who received preprints indicated that at least 1 of their peer reviewer comments on the manuscript or study was directly affected by the material received.

The majority of respondents felt that the provision of abstracts and preprints of related papers was or could be helpful to them in their performance of peer review and to the peer review process. In fact, it appears that this need was particularly recognized when they actually received supplemental materials. Furthermore, approximately 1 in 3 respondents reported that a specific comment in their peer review was directly affected by receipt of these materials.

The strengths of this study include its large size and prospective design. The limitations of the study include the nonrandom provision of supplemental materials and the nonindependence of answers because manuscripts had several reviewers. Additionally, reviewers prone to respond with positive assessments may have been more likely to return their completed questionnaires. However, we think that the qualitative conclusion of this study, namely, that an appreciable proportion of reviewers think that supplemental materials could (and do) improve their reviews, is unlikely to have been materially affected by those factors. Whether this perceived improvement is apparent from the written peer review and whether it ultimately improves judgments about manuscripts should be the subject of future research.

The lack of originality as well as deficiencies in presentation and interpretation, ie, that previously published work has not been cited or adequately discussed, are important criteria that often form the basis for a peer reviewer's recommendation to reject a manuscript.⁶ The aspects of an accepted manuscript that are most likely to change during the peer review and editing processes are the presentation of limitations, generalizations, and the tone of the conclusions.⁷ Improved knowledge of related literature could affect the sophistication of the peer reviewers' judgment in all of these areas, and it is clear from this study that many peer reviewers felt that provision of related literature did help them. As an outcome of this study, senior editors at the Journal of the National Cancer Institute now perform a literature search for every manuscript that is sent for peer review.