From Department of Surgery, Duke University Medical Center, Durham, NC (Dr Purcell), and Editorial Offices, Annals of Internal Medicine, American College of Physicians, Philadelphia, Pa (Ms Donovan and Dr Davidoff).
Context.— Biomedical manuscripts undergo substantive change as a result of the
peer review and editorial revision processes.
Objective.— To characterize quantitatively problems in manuscripts identified during
peer review and changes made to address these problems.
Design and Setting.— Descriptive analysis of manuscripts submitted to and articles published
by the Annals of Internal Medicine. A taxonomy of
problems that occur in reporting clinical research was developed from analysis
of changes made to 7 manuscripts between submission and publication (published
October 15, 1996, and November 1, 1996). The taxonomy was used to characterize
changes to 12 additional manuscripts (published January 15, 1997, to April
Main Outcome Measure.— Types of problems necessitating changes to manuscripts during peer review
Results.— Changes occurred because of 5 types of problems: too much information,
too little information, inaccurate information, misplaced information, and
structural problems. Changes most often occurred because information was missing
or extraneous. The distribution of changes seemed to be influenced by the
type of information involved (such as background or conclusions).
Conclusion.— The proposed framework may be useful for characterizing quantitatively
the effects of peer review and for comparing those effects across editors,
journals, and specialties.
THE EDITORIAL process, including peer review, serves 2 principal functions
in medical journal publishing: (1) to select reports for publication that
meet scientific quality standards and (2) to improve presentation of research
during revision. Several investigators have studied the effectiveness of manuscript
selection, but little information is available on the changes to manuscripts
that occur during the revision process. Roberts and colleagues1
studied changes in readability, and Gardner and Bond2
reported on changes in the adequacy of statistical reporting in manuscripts
between submission and publication. Goodman et al3
described changes that occurred in subjective quality ratings of 34 manuscript
elements during revision. However, none of these authors explored the specific
objective changes made to manuscripts that enhanced their acceptability.
We developed a taxonomy to characterize quantitatively the problems
that prompt the diverse changes that medical manuscripts undergo as part of
the editorial and peer review processes. We also investigated the relationships
between changes to manuscripts and the types of information involved, such
as background, limitations, or conclusions; the type of paper; and the senior
editor who directed the revision.
Using 7 research articles from 2 issues of the Annals
of Internal Medicine (October 15, 1996, and November 1, 1996), 2 of
the authors (G.P.P. and F.D.) compared submitted versions with published papers
and identified all substantive changes (defined as revisions that altered
meaning) in the text. Changes resulting from spelling, grammar, and usage
errors were excluded.
On the basis of findings from these 7 papers, we constructed a taxonomy
of problems that prompted changes to manuscripts (Table 1). This taxonomy incorporated elements of a scheme developed
for classifying errors of clinical problem formulation.4
To test the completeness and applicability of the taxonomy, we then used it
to characterize the changes made in response to problems identified in 12
additional scientific articles (6 interventional studies and 6 risk or prognostic
factor studies). These studies, which had not been analyzed for the purposes
of other experiments, were selected sequentially from recent issues (January
15, 1997, to April 1, 1997) of the Annals of Internal Medicine. Equal numbers of manuscripts processed by 2 senior editors were chosen.
We performed a descriptive study of the associations among occurrence
of problems and editors, types of papers, and types of information. Through
line-by-line comparisons of original manuscripts with published articles,
we identified the problem that led to every substantive change and the type
of information involved. Information types were selected from an indexing
vocabulary that defines the semantic structure of a medical manuscript and
facilitates electronic searching of the primary medical literature.5 Two of the authors (G.P.P. and S.L.D.) performed the
analysis for each paper. Discrepancies in classification of information were
resolved by discussion.
Changes to manuscripts occurred for 5 major reasons: the text contained
(1) too much information, (2) too little information, (3) inaccurate information,
(4) misplaced information, or (5) structural problems (problems that resulted
in a change in the format of information, such as a table that was converted
to a text paragraph). Each of these 5 categories contained several subcategories
that were consistently observed within manuscripts. For example, the predominate
problem in some manuscripts was too much information (subcategorized as text
that was too detailed, redundant, or extraneous); problems in other manuscripts
were largely due to incorrect information (subcategorized as inaccurate facts
or data that had been insufficiently or inappropriately processed).
Table 1 shows our taxonomy
and the distribution of problems across the full set of study manuscripts.
Overall, changes most often occurred because information was missing or extraneous.
In general, the distribution of changes did not differ by type of study, except
for the problem of redundancy, which occurred twice as often in interventional
studies than in risk factor studies. The results also did not differ substantially
by senior editor (data not shown).
Certain problems were often associated with specific types of information
(Table 2). Limitations, for example,
were frequently omitted. Overall, changes were most often made to experimental
data, background information, and references.
We developed an analytical framework to describe the type of changes
that manuscripts undergo during peer review. By identifying the types of problems
in manuscripts that lead to substantive textual changes, we provide a means
of quantitatively assessing and understanding the nature of the objective
changes brought about by peer review and the editorial process. The taxonomy
may also provide a foundation for comparing the effects of peer review across
editors, authors, journals, and types of study. Our work extends portions
of the framework established by Kassirer and Campion6
for characterizing the peer review process.
Our research had limitations. This descriptive study was intended to
develop and test a quantitative framework, not to establish a "gold standard"
for scientific reporting. The sample size was too small to draw statistically
meaningful conclusions. The distribution of problems that we observed may
be specific to the Annals of Internal Medicine, and
some changes may reflect the requirements of this journal. We plan to apply
the taxonomy to manuscripts from other journals to test its generalizability.
Finally, we did not assess interrater reliability of assignment of problems.
Rather, we concentrated on resolving disagreements because one of our goals
was to define clearly the appropriate use of each category (for example, the
difference between extraneous information and information that was too detailed).
Future work is needed to determine whether other variables, such as journal,
specialty, or blinding of the review process, affect the distribution of reasons
for changes to manuscripts.
Our taxonomy may help authors to understand better the underlying structure
of a scientific paper and thereby identify and present relevant data more
effectively. Analysis of the editorial process may lead to better instructions
for authors. Contributors to the peer review process, such as peer reviewers
and senior editors, may also benefit from understanding the nature of commonly
identified problems. Finally, knowledge of the distribution of problems across
types of information may serve as a basis for the development of electronic
tools to facilitate peer review.
Gretchen P. Purcell, Shannon L. Donovan, Frank Davidoff. Changes to Manuscripts During the Editorial ProcessCharacterizing the Evolution of a Clinical Paper. JAMA. 1998;280(3):227–228. doi:10.1001/jama.280.3.227