Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
Peer Review Congress
July 15, 1998

Changes to Manuscripts During the Editorial ProcessCharacterizing the Evolution of a Clinical Paper

Author Affiliations

From Department of Surgery, Duke University Medical Center, Durham, NC (Dr Purcell), and Editorial Offices, Annals of Internal Medicine, American College of Physicians, Philadelphia, Pa (Ms Donovan and Dr Davidoff).

JAMA. 1998;280(3):227-228. doi:10.1001/jama.280.3.227

Context.— Biomedical manuscripts undergo substantive change as a result of the peer review and editorial revision processes.

Objective.— To characterize quantitatively problems in manuscripts identified during peer review and changes made to address these problems.

Design and Setting.— Descriptive analysis of manuscripts submitted to and articles published by the Annals of Internal Medicine. A taxonomy of problems that occur in reporting clinical research was developed from analysis of changes made to 7 manuscripts between submission and publication (published October 15, 1996, and November 1, 1996). The taxonomy was used to characterize changes to 12 additional manuscripts (published January 15, 1997, to April 1, 1997).

Main Outcome Measure.— Types of problems necessitating changes to manuscripts during peer review and revision.

Results.— Changes occurred because of 5 types of problems: too much information, too little information, inaccurate information, misplaced information, and structural problems. Changes most often occurred because information was missing or extraneous. The distribution of changes seemed to be influenced by the type of information involved (such as background or conclusions).

Conclusion.— The proposed framework may be useful for characterizing quantitatively the effects of peer review and for comparing those effects across editors, journals, and specialties.

THE EDITORIAL process, including peer review, serves 2 principal functions in medical journal publishing: (1) to select reports for publication that meet scientific quality standards and (2) to improve presentation of research during revision. Several investigators have studied the effectiveness of manuscript selection, but little information is available on the changes to manuscripts that occur during the revision process. Roberts and colleagues1 studied changes in readability, and Gardner and Bond2 reported on changes in the adequacy of statistical reporting in manuscripts between submission and publication. Goodman et al3 described changes that occurred in subjective quality ratings of 34 manuscript elements during revision. However, none of these authors explored the specific objective changes made to manuscripts that enhanced their acceptability.

We developed a taxonomy to characterize quantitatively the problems that prompt the diverse changes that medical manuscripts undergo as part of the editorial and peer review processes. We also investigated the relationships between changes to manuscripts and the types of information involved, such as background, limitations, or conclusions; the type of paper; and the senior editor who directed the revision.


Using 7 research articles from 2 issues of the Annals of Internal Medicine (October 15, 1996, and November 1, 1996), 2 of the authors (G.P.P. and F.D.) compared submitted versions with published papers and identified all substantive changes (defined as revisions that altered meaning) in the text. Changes resulting from spelling, grammar, and usage errors were excluded.

On the basis of findings from these 7 papers, we constructed a taxonomy of problems that prompted changes to manuscripts (Table 1). This taxonomy incorporated elements of a scheme developed for classifying errors of clinical problem formulation.4 To test the completeness and applicability of the taxonomy, we then used it to characterize the changes made in response to problems identified in 12 additional scientific articles (6 interventional studies and 6 risk or prognostic factor studies). These studies, which had not been analyzed for the purposes of other experiments, were selected sequentially from recent issues (January 15, 1997, to April 1, 1997) of the Annals of Internal Medicine. Equal numbers of manuscripts processed by 2 senior editors were chosen.

Table 1.—Taxonomy of Problems and Frequency of Resultant Changes in 12 Papers
Table 1.—Taxonomy of Problems and Frequency of Resultant Changes in 12 Papers
Image not available

We performed a descriptive study of the associations among occurrence of problems and editors, types of papers, and types of information. Through line-by-line comparisons of original manuscripts with published articles, we identified the problem that led to every substantive change and the type of information involved. Information types were selected from an indexing vocabulary that defines the semantic structure of a medical manuscript and facilitates electronic searching of the primary medical literature.5 Two of the authors (G.P.P. and S.L.D.) performed the analysis for each paper. Discrepancies in classification of information were resolved by discussion.


Changes to manuscripts occurred for 5 major reasons: the text contained (1) too much information, (2) too little information, (3) inaccurate information, (4) misplaced information, or (5) structural problems (problems that resulted in a change in the format of information, such as a table that was converted to a text paragraph). Each of these 5 categories contained several subcategories that were consistently observed within manuscripts. For example, the predominate problem in some manuscripts was too much information (subcategorized as text that was too detailed, redundant, or extraneous); problems in other manuscripts were largely due to incorrect information (subcategorized as inaccurate facts or data that had been insufficiently or inappropriately processed).

Table 1 shows our taxonomy and the distribution of problems across the full set of study manuscripts. Overall, changes most often occurred because information was missing or extraneous. In general, the distribution of changes did not differ by type of study, except for the problem of redundancy, which occurred twice as often in interventional studies than in risk factor studies. The results also did not differ substantially by senior editor (data not shown).

Certain problems were often associated with specific types of information (Table 2). Limitations, for example, were frequently omitted. Overall, changes were most often made to experimental data, background information, and references.

Table 2.—Distribution of the 2 Most Frequent Problems by Information Type*
Table 2.—Distribution of the 2 Most Frequent Problems by Information Type*
Image not available

We developed an analytical framework to describe the type of changes that manuscripts undergo during peer review. By identifying the types of problems in manuscripts that lead to substantive textual changes, we provide a means of quantitatively assessing and understanding the nature of the objective changes brought about by peer review and the editorial process. The taxonomy may also provide a foundation for comparing the effects of peer review across editors, authors, journals, and types of study. Our work extends portions of the framework established by Kassirer and Campion6 for characterizing the peer review process.

Our research had limitations. This descriptive study was intended to develop and test a quantitative framework, not to establish a "gold standard" for scientific reporting. The sample size was too small to draw statistically meaningful conclusions. The distribution of problems that we observed may be specific to the Annals of Internal Medicine, and some changes may reflect the requirements of this journal. We plan to apply the taxonomy to manuscripts from other journals to test its generalizability. Finally, we did not assess interrater reliability of assignment of problems. Rather, we concentrated on resolving disagreements because one of our goals was to define clearly the appropriate use of each category (for example, the difference between extraneous information and information that was too detailed). Future work is needed to determine whether other variables, such as journal, specialty, or blinding of the review process, affect the distribution of reasons for changes to manuscripts.

Our taxonomy may help authors to understand better the underlying structure of a scientific paper and thereby identify and present relevant data more effectively. Analysis of the editorial process may lead to better instructions for authors. Contributors to the peer review process, such as peer reviewers and senior editors, may also benefit from understanding the nature of commonly identified problems. Finally, knowledge of the distribution of problems across types of information may serve as a basis for the development of electronic tools to facilitate peer review.

Roberts JC, Fletcher RH, Fletcher SW. Effects of peer review and editing on the readability of articles published in Annals of Internal Medicine JAMA.1994;272:119-121.
Gardner MJ, Bond J. An exploratory study of statistical assessment of papers published in the British Medical Journal JAMA.1990;263:1355-1357.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine Ann Intern Med.1994;121:11-21.
Voytovich AE, Rippey RM, Jue D. Diagnostic reasoning in the multiproblem patient: an interactive microcomputer audit.  Eval Health Professions.1986;9:91-102.
Purcell CP, Rennels GD, Shortliffe EH. Development and evaluation of a context-based document representation for searching the medical literature.  Int J Digital Libraries.1997;1:288-296.
Kassirer JP, Campion EW. Peer review: crude and understudied, but indispensable.  JAMA.1994;272:96-97.