From the Journal of the National Cancer Institute , Bethesda, Md (Dr Hatch); and the Johns Hopkins Oncology Center, Division of Biostatistics, Baltimore, Md (Dr Goodman). Dr Hatch is now with the Division of Extramural Activities, National Cancer Institute, Rockville, Md.
Context.— Many journals provide peer reviewers with written instructions regarding
review criteria, such as the originality of results, but little research has
been done to investigate ways to improve or facilitate the peer review task.
Objective.— To assess the value that peer reviewers place on receipt of supplemental
materials (eg, abstracts of related papers and preprints of related unpublished
Design.— Questionnaire survey sent to all 733 peer reviewers recruited by the Journal of the National Cancer Institute to review 356
manuscripts consecutively sent out for review from February 24, 1997, through
January 16, 1998. The inclusion of supplemental materials with manuscript
review packages was optional.
Main Outcome Measure.— The peer reviewers' assessment of the actual or potential usefulness
of supplemental materials on the performance of peer review.
Results.— A total of 481 (66%) of 733 questionnaires were returned. Of the 471
respondents' questionnaires that could be used, 217 (46%) indicated that they
received abstracts, and 44 (10%) of 458 respondents indicated that they received
preprints. Higher proportions of peer reviewers who received supplemental
materials than those who had not received them felt that they were (or would
be) useful to them when reviewing the manuscript (63% [95% confidence interval
(CI), 57%-69%] vs 45% [95% CI, 38%-52%]; P<.001)
and to the peer review process in general (80% [95% CI, 75%-85%] vs 64% [95%
CI, 58%-70%]; P<.001).
Conclusion.— The majority of respondents indicated that supplemental materials helped
(or would have helped) them evaluate manuscripts and valued them more highly
when they actually received them.
MANUSCRIPTS SUBMITTED to a journal for review and possible publication
may contain results that are not novel or original, either because very similar
results have been published by other investigators or because the authors
themselves have already published the results. Following the lead of Ingelfinger,1 most journals will not knowingly publish results that
have already been reported. Manuscripts may also contain results that should
be published as part of a larger, more complete study. Angell and Relman2 have eloquently discussed what they termed "redundant
publication" and its costs to the scientific community. Frank3
conducted a survey of the top 100 journals and found that only 72% asked peer
reviewers to assess the originality and novelty of submitted manuscripts.
Peer reviewers may not spend much time reviewing a manuscript.4
Little research appears to have been done to design and study interventions
that might aid peer reviewers in their performance of peer review.
Since 1993, senior editors at the Journal of the National
Cancer Institute have searched the MEDLINE biomedical literature database
for published papers related to the topics of many of the manuscripts being
sent out for peer review. When such a literature search was done, it may have
been done to educate the editor on the topic, to identify related papers and/or
appropriate peer reviewers, or to provide a service to the editorial board
members and peer reviewers. The current study was conducted to obtain an objective
assessment of the usefulness of these supplemental materials to the peer review
process from the peer reviewers themselves.
A questionnaire was designed to prospectively measure peer reviewers'
assessment of the actual or potential usefulness of abstracts and preprints
of related papers to their individual review of a manuscript and to the peer
review process. The questionnaires were included as part of the normal peer
reviewer packages for consecutive manuscripts sent for peer review; recipients
were informed that the results of this study might be published.
During the period covered by this study, the review of related literature
by use of MEDLINE or other means (eg, other literature databases or library
resources) by senior editors before peer reviewers were selected was optional;
the completion of this task by senior editors was typically dependent on their
workloads. The performance of the literature search and printing of abstracts
typically required about 1 hour per manuscript. Senior editors also checked
whether related unpublished (ie, labeled "in press" or "submitted") manuscripts
had been provided (as specified in the Information for Authors section of
the Journal of the National Cancer Institute and
other journals2) or cited; when deemed relevant
to the review of a submitted manuscript, copies of such manuscripts (provided
by the authors or obtained by request) were sent as confidential supplemental
documents to the peer reviewers.
The first 130 (18%) of 733 questionnaires sent out had 16 questions;
the 603 questionnaires sent out subsequently had 1 additional question, which
asked the peer reviewer if he or she typically searches MEDLINE or another
literature database when reviewing a manuscript (questionnaires are available
on request). There were no statistically significant differences in the responses
before or after addition of the 17th question (data not shown). Eight questions
assessed specific issues related to the reviewers' perception of the usefulness
of abstracts, 6 questions assessed specific issues related to their perception
of the usefulness of preprints, and 2 questions assessed their perception
of the overall usefulness of abstracts and preprints to their individual peer
review and to the process in general. Reviewers were invited to write comments
at the end of the questionnaire.
The reviewers were asked to mark their responses to each question on
a Likert-type scale5 that ranged from 1 to
5, with 1 defined as definitely yes, 3 defined as not sure or no opinion, and 5 defined as definitely no. Any response of 1.0 through 2.0 was considered an affirmative
or positive assessment of value in the analyses that follow.
The numbers of responses with values from 1.0 through 2.0 were divided
by the total number of responses to a given question; percentages and 95%
confidence intervals for those percentages were calculated. The statistical
significance between groups was calculated by use of the 2-sided χ2 test; P<.05 was considered to indicate
a statistically significant difference.
From February 24, 1997, through January 16, 1998, 733 questionnaires
were sent to peer reviewers; 481 (66%) were returned by February 16, 1998,
and 471 (64%) were usable (10 questionnaires were returned blank). These questionnaires
were included as part of the normal peer review packages for 356 consecutive
manuscripts sent for peer review. Of the 471 respondents, 217 (46%) indicated
that they received abstracts, and 44 (10%) of 458 respondents indicated that
they received supplemental "in press" or "submitted" manuscripts. Of 388 respondents,
199 (51%) indicated that they perform their own literature search for papers
relevant to a manuscript.
The peer reviewers' perceived assessments of 5 specific aspects of the
usefulness of the supplemental materials (ie, abstracts and/or preprints)
are presented in Table 1. The
majority of respondents indicated that abstracts of related papers did (or
would) help them judge aspects of the originality of the submitted manuscript
(examination, duplication, and fragmentation; defined in the table footnote);
the proportions of respondents who indicated that abstracts would be of positive
value were similar whether they indicated that they received supplemental
materials or not. Data from responses to the 2 questions regarding the overall
usefulness of supplemental materials are also presented. Significantly higher
proportions of respondents who indicated that they received supplemental materials
than those who indicated that they had not received them felt that such materials
would be useful to them and to the peer review process in general.
Seventy-six (35%) of 215 respondents who received abstracts and 16 (32%)
of 50 respondents who received preprints indicated that at least 1 of their
peer reviewer comments on the manuscript or study was directly affected by
the material received.
The majority of respondents felt that the provision of abstracts and
preprints of related papers was or could be helpful to them in their performance
of peer review and to the peer review process. In fact, it appears that this
need was particularly recognized when they actually received supplemental
materials. Furthermore, approximately 1 in 3 respondents reported that a specific
comment in their peer review was directly affected by receipt of these materials.
The strengths of this study include its large size and prospective design.
The limitations of the study include the nonrandom provision of supplemental
materials and the nonindependence of answers because manuscripts had several
reviewers. Additionally, reviewers prone to respond with positive assessments
may have been more likely to return their completed questionnaires. However,
we think that the qualitative conclusion of this study, namely, that an appreciable
proportion of reviewers think that supplemental materials could (and do) improve
their reviews, is unlikely to have been materially affected by those factors.
Whether this perceived improvement is apparent from the written peer review
and whether it ultimately improves judgments about manuscripts should be the
subject of future research.
The lack of originality as well as deficiencies in presentation and
interpretation, ie, that previously published work has not been cited or adequately
discussed, are important criteria that often form the basis for a peer reviewer's
recommendation to reject a manuscript.6 The
aspects of an accepted manuscript that are most likely to change during the
peer review and editing processes are the presentation of limitations, generalizations,
and the tone of the conclusions.7 Improved
knowledge of related literature could affect the sophistication of the peer
reviewers' judgment in all of these areas, and it is clear from this study
that many peer reviewers felt that provision of related literature did help
them. As an outcome of this study, senior editors at the Journal of the National Cancer Institute now perform a literature search
for every manuscript that is sent for peer review.
Hatch CL, Goodman SN. Perceived Value of Providing Peer Reviewers With Abstracts and Preprints
of Related Published and Unpublished Papers. JAMA. 1998;280(3):273–274. doi:10.1001/jama.280.3.273