Customize your JAMA Network experience by selecting one or more topics from the list below.
Author Affiliations: Monash Institute of Health Services Research, Monash Medical Centre, Clayton, Victoria, Australia (Dr Silagy); and UK Cochrane Centre, Oxford, England (Mss Middleton and Hopewell). Dr Silagy is deceased.
Context Publication of research protocols minimizes bias by explicitly stating
a priori hypotheses and methods without prior knowledge of results.
Methods We conducted a retrospective comparative study to assess the extent
to which the content of published Cochrane reviews had changed compared with
their previously published protocols and to assess any potential impact these
changes may have had in introducing bias to the study. We identified previously
published protocols for new Cochrane reviews appearing in The Cochrane Library;
2000, issue 3. The texts of published protocols and completed reviews were
compared. Two raters independently identified changes to the different sections
of the protocol and classified the changes as none, minor, or major.
Results Of the 66 new Cochrane reviews, we identified a previously published
protocol for 47 reviews. Of these, 43 reviews had at least 1 section that
had undergone a major change compared with the most recently published protocol.
The greatest variation between protocols and reviews was in the methods section,
in which 68% of reviews (n = 32) had undergone a major change. Changes made
in other sections that may have resulted in the introduction of bias included
narrowing of objectives, addition of comparisons or new outcome measures,
broadening of criteria for the types of study design included, and narrowing
of types of participants included.
Conclusions Research protocols, even if published, are likely to remain, at least
to some extent, iterative documents. We found that a large number of changes
were made to Cochrane reviews, some of which could be prone to influence by
prior knowledge of results. Even if many of the changes between protocol and
review improve the overall study, the reasons for making these should be clearly
identified and documented within the final review.
Research studies are prone to many different types of bias in which
a systematic, rather than random, error may occur in the results.1 One of the fundamental strategies to minimize bias
is to explicitly state the hypotheses that will be tested and the methodological
approaches that will be used in a research study prior to undertaking it,
without prior knowledge of the data. Research protocols fulfill this role
by providing a road map to the planned research.
Ideally, a research protocol should provide a clear background and justification
for the proposed study and state the aims, objectives, and methods used to
conduct the study and to collect and analyze the data. In the case of protocols
for Cochrane reviews, these methods include statements about any a priori
hypotheses; the types of studies, participants, interventions, and outcomes
that will be included in the review; data sources; search strategy; data extraction
methods (including assessment of the quality of included studies); and methods
of combining data (where appropriate).2 Provision
of all of this information in a clear and transparent manner allows readers
to understand exactly what was done in carrying out a review. On occasion,
a good reason may exist for modifying a protocol in the course of undertaking
a review (although not after the results are known), and such changes (and
the reasons for them) ought to be explicitly identified and documented. However,
subsequent changes to the protocol should be kept to a minimum. This avoids
a situation in which a researcher modifies the methods of the research as
the research is in progress to attempt to deal with any unexpected findings.
Such changes are of particular concern once data collection and analysis are
The Cochrane Collaboration has a policy of peer review of all protocols
for its systematic reviews, with encouragement to publish them electronically
in The Cochrane Library to obtain additional feedback prior to proceeding
to data collection and analysis.2 The primary
objective of our study was to assess the extent to which the content of published
Cochrane reviews had changed compared with their previously published protocols.
A secondary objective was to explore the potential impact of any changes made
to published protocols, particularly in terms of the risk of introducing bias
to the study.
We identified all new Cochrane reviews appearing for the first time
in the Cochrane Library; 2000, issue 3. For each of these, we used the unique
identifier attached to the review to identify whether a protocol had been
published for the review in the previous issue of the Cochrane Library (2000,
issue 2). When we identified a published protocol, we downloaded the text
of both the protocol and the completed review into separate Microsoft Word
(Microsoft Office 1998; Microsoft Inc, Redmond, Wash) files. We then compared
and highlighted changes made to the published protocol in the completed review
using the "document compare" function in Microsoft Word.
Each of these comparative documents were rated independently by 2 of
the 3 raters. The raters assessed whether the authorship of the completed
review had changed from that of the protocol and whether there had been changes
in any of the 9 sections of Cochrane review protocols: background, objectives,
comparisons, types of studies, types of participants, types of interventions,
types of outcomes, search strategy, and methods. For each section, the raters
were required to identify whether a change had occurred and, if so, whether
it was a minor or major change.
For the purpose of this study, we did not regard alterations in tense
or abbreviations as a change. Changes in style or wording (other than changes
in tense or abbreviations) that did not alter the substance or meaning of
a section and would not have altered the interpretation by someone else replicating
the review were considered minor changes. Any other change that altered the
substance or meaning of a section or would have altered the interpretation
by someone else replicating the review was regarded as a major change. When
the raters disagreed on the categorization of any changes, disagreements were
resolved by discussion.
We examined the specific sections in which a major change had occurred
and sought to describe the nature of the changes and their potential for introducing
bias into the review. We also approached the authors of reviews in which a
major change had occurred and asked them to explain the reasons for making
We identified 66 completed reviews appearing for the first time in the
Cochrane Library; 2000, issue 3. Of these, we found a protocol for 71% (n
= 47) in 2000, issue 2, of the Cochrane Library. None of these published protocols
had any external comments or criticisms attached. The remaining 19 reviews
had apparently been carried out without prior publication of a protocol.
Only 1 of the 47 reviews with a prior protocol remained unchanged compared
with the published protocol. A further 3 reviews had only minor changes, eg,
grammatical changes, from the published protocol. The remaining 91% of reviews
(n = 43) had at least 1 section that had undergone a major change compared
with the previously published protocol. There were changes to all 9 sections
examined in 1 review. The median number of major changes per review was 2.
A summary of changes identified in the completed reviews compared with their
published protocols is shown in Table 1. Interrater reliability was high, with 80% agreement between raters.
The authors were unchanged in 68% of the reviews (n = 32) compared with
previously published protocols. In 28% (n = 13), additional authors were included
in the final review and in 4% (n = 2), authors who had been listed in the
published protocol did not appear in the completed review.
Sections of the Protocol
Major changes were made to the background section in 53% of reviews
(n = 25). Most of these involved inclusion of additional descriptive epidemiological
data or information to enhance the rationale for the proposed review.
In all but 1 of 16 reviews, there had been major changes to the stated
objectives of the review involving either a clarification or broadening of
the original objectives. Studies for which the objectives were clarified usually
involved inclusion of more-specific definitions or caveats regarding a particular
term (eg, from "pain" to "late postoperative pain"). In studies in which the
objectives were broadened, this usually involved consideration of safety in
addition to efficacy.
Only 25 of the 47 protocols had prestated comparisons. Of these, 48%
made major changes in the completed review. Three of the reviews removed all
mention of specific comparisons that had been identified in the protocol.
A further 9 reviews added or modified comparisons. When existing comparisons
were modified, this invariably involved clarification that the intervention
treatment was being compared with no treatment as well as with some form of
Types of Studies
Among the 15 reviews in which there were major changes to the types
of studies included, 1 involved a change to more-restrictive study design
criteria, whereas 4 involved a broadening of the types of studies included
beyond randomized controlled trials alone to other potentially less-robust
forms of comparison. In a further 5 reviews, the changes involved inclusion
of additional methodological criteria (eg, deciding to accept unpublished
data, requirement for trials to be double blind, requirement for nonaggregated
data from individual participants to be available, or need for a minimum proportion
of patients to have been followed up). In the remaining 5 reviews, the changes
were clarifications of a descriptive feature of the studies eligible for inclusion.
Types of Participants
Sixteen reviews made major changes to the types of participants included
in the review. The nature of changes to the section describing types of participants
was essentially to refine or add criteria for prestated diagnoses of inclusion.
Types of Interventions
Twenty-three reviews made major changes to this section of the protocol.
Changes to types of interventions fell into 3 broad categories: clarification
of terms used to describe or define the intervention, simplification of multiple
interventions to a broad all-encompassing heading, and more detailed specification
under a broad heading. When changes had been made to the objectives or comparisons,
the changes to types of interventions were generally to ensure consistency
with the changes to these other sections of the review.
Types of Outcomes
Of the 22 studies with major changes to the types of outcomes, two thirds
involved the addition of new outcomes or new measurement methods for existing
outcomes. Nine studies added the measurement of adverse effects and quality-of-life
measures as specific outcomes and 7 specified new measurement instruments
or criteria for existing outcomes. In the remaining 6 studies, outcome measures
present in the protocol were removed from the final review.
Changes to the search strategy occurred in 23 reviews. These involved
either addition or deletion of specific bibliographic databases and terms
(or simplifying the presentation of a comprehensive strategy by reference
to the relevant Cochrane Review Group's search strategy), modifying the dates
for which the search was conducted, or reversing a prior decision to seek
Methods of the Review
The methods section had the greatest variation between protocol and
review; 32 reviews (68%) had changes in the type of information presented.
Some changes simply incorporated greater detail of the methods of analyses,
whereas others involved more substantive additions or deletions of actual
or proposed subgroup analyses. Other major amendments within this section
were changes to the lists of information to be extracted from the component
studies and changes to the criteria for decision making relating to the process
of the review (eg, changes to the number of data extractors for a review and
the method for resolving any differences between them).
We wrote to the contact reviewer of the 43 reviews in which we had found
a major change to 1 or more sections compared with the published protocol,
of whom 65% (n = 28) responded. All of the reviewers gave pragmatic reasons
for making the changes that were unrelated to knowledge of the results or
a perception that bias was being deliberately introduced into the study. The
reviewers' reasons for the changes included: as a response to editorial suggestions
(n = 10); to improve clarity (n = 7); because the review changed in scope
as it progressed (n = 6); to reflect what was actually done (n = 3); because
the reviewer learned "on the job" (eg, learning how to analyze survival curves)
(n = 4); and to fit other allied protocols (n = 3). Reviewers may have given
more than 1 reason for making a change to their review.
This study shows that of the 71% of Cochrane reviews with a published
protocol, all but 1 underwent at least some changes in the completed review.
These changes were minor in only 3 reviews. All of the remaining 43 reviews
had at least 1 major change, and many reviews had changes in 5 or more sections
compared with the published protocol. Two questions arise from this situation:
(1) Do these changes matter? and (2) What value should be placed on the published
Many of the changes to the protocol, even when defined as major, may
have improved the overall quality of the review without introducing any significant
bias. For example, the inclusion of more detail in the background section
of the review is unlikely to cause bias. Similarly, when the authors of the
review increased the comprehensiveness of the search strategy or the security
of the data extraction methods or clarified the statistical methods to be
used, this was likely to enhance the overall quality of the review.
However, changes made to the protocol that could have been based on
prior knowledge of the results are of greater concern. These changes may have
introduced the possibility of bias, either intentionally or unintentionally,
into the review. Narrowing of the scope of the review in the form of more
restricted objectives or omission of types of comparisons may be a source
of bias, given that the omitted comparisons may show an intervention to be
more or less effective than the retained comparisons. The additions and deletions
of types of studies and deletions to search strategies could also potentially
cause bias. Addition or deletion of outcome measures has a potential for bias,
especially when the reviewers are aware of study results. For example, an
outcome may be deleted because its results do not "fit"' the other results.
A change from more rigorous methods to a looser set of methods in the review
could introduce bias. However, this study suggests that the authors were more
likely to add greater detail to their methods. Some of these additions, such
as changing from a fixed-effects to a random-effects model, use of a different
effect measure, and addition or deletion of subgroup analyses, could be sources
of bias, especially without sensitivity analyses to test the effect of such
Based on the feedback received from the authors of reviews that had
undergone a major change, it is clear that some of the changes were made because
of knowledge of studies that existed but not necessarily because of knowledge
of their results. While these responses do not exclude the possibility that
bias is unintentionally introduced in this way, it appears that most of the
changes to protocols are made for pragmatic reasons and as a result of feedback
from the peer review process, to improve the final review.
Research protocols, even if published, are likely to remain iterative
documents, at least to some extent. It would have been surprising, therefore,
if there had not been some changes to the published Cochrane protocols before
the publication of the reviews. The Cochrane Library provides a facility for
external comments and criticisms to be attached to both protocols and completed
reviews. However, at present, these comments seem to be almost entirely confined
to completed reviews. None of the protocols included in this study had received
external comments through this system. Of the 1013 protocols in the current
issue of the Cochrane Library; 2002, issue 1, only 2 have comments attached.
This lack of external comments or criticisms on published protocols is disappointing,
particularly since 31 (44%) of the 71 comments attached to completed reviews
in the same issue of the Cochrane Library were related to aspects of the protocol
but did not appear to have been made until the full review had been published.
Although few comments are made about published protocols in the form
of published criticisms, the process is not without value. Protocols still
serve as a detailed guide for a wide audience to research in progress and
may help contribute to reduced duplication of effort by others who are planning
to do similar research. The process of publishing protocols may also minimize
the likelihood of reviewers introducing bias into their review by making major
changes that might otherwise remain undisclosed and undetectable. The publication
of both protocols and completed reviews allows the tracking of any changes
that have taken place and an examination of what effect this may have had
on the results of the review. One strategy that might be used in the future
would be to indicate any changes made (and reasons, where appropriate) from
the original study protocol. This would allow readers to formulate their own
judgment as to the effect such changes may have had on the final review.
Silagy CA, Middleton P, Hopewell S. Publishing Protocols of Systematic Reviews: Comparing What Was Done to What Was Planned. JAMA. 2002;287(21):2831–2834. doi:10.1001/jama.287.21.2831
Create a personal account or sign in to: