Publishing Protocols of Systematic Reviews: Comparing What Was Done to What Was Planned

Context Publication of research protocols minimizes bias by explicitly stating a priori hypotheses and methods without prior knowledge of results.

Methods We conducted a retrospective comparative study to assess the extent to which the content of published Cochrane reviews had changed compared with their previously published protocols and to assess any potential impact these changes may have had in introducing bias to the study. We identified previously published protocols for new Cochrane reviews appearing in The Cochrane Library; 2000, issue 3. The texts of published protocols and completed reviews were compared. Two raters independently identified changes to the different sections of the protocol and classified the changes as none, minor, or major.

Results Of the 66 new Cochrane reviews, we identified a previously published protocol for 47 reviews. Of these, 43 reviews had at least 1 section that had undergone a major change compared with the most recently published protocol. The greatest variation between protocols and reviews was in the methods section, in which 68% of reviews (n = 32) had undergone a major change. Changes made in other sections that may have resulted in the introduction of bias included narrowing of objectives, addition of comparisons or new outcome measures, broadening of criteria for the types of study design included, and narrowing of types of participants included.

Conclusions Research protocols, even if published, are likely to remain, at least to some extent, iterative documents. We found that a large number of changes were made to Cochrane reviews, some of which could be prone to influence by prior knowledge of results. Even if many of the changes between protocol and review improve the overall study, the reasons for making these should be clearly identified and documented within the final review.

Research studies are prone to many different types of bias in which a systematic, rather than random, error may occur in the results.¹ One of the fundamental strategies to minimize bias is to explicitly state the hypotheses that will be tested and the methodological approaches that will be used in a research study prior to undertaking it, without prior knowledge of the data. Research protocols fulfill this role by providing a road map to the planned research.

Ideally, a research protocol should provide a clear background and justification for the proposed study and state the aims, objectives, and methods used to conduct the study and to collect and analyze the data. In the case of protocols for Cochrane reviews, these methods include statements about any a priori hypotheses; the types of studies, participants, interventions, and outcomes that will be included in the review; data sources; search strategy; data extraction methods (including assessment of the quality of included studies); and methods of combining data (where appropriate).² Provision of all of this information in a clear and transparent manner allows readers to understand exactly what was done in carrying out a review. On occasion, a good reason may exist for modifying a protocol in the course of undertaking a review (although not after the results are known), and such changes (and the reasons for them) ought to be explicitly identified and documented. However, subsequent changes to the protocol should be kept to a minimum. This avoids a situation in which a researcher modifies the methods of the research as the research is in progress to attempt to deal with any unexpected findings. Such changes are of particular concern once data collection and analysis are under way.

The Cochrane Collaboration has a policy of peer review of all protocols for its systematic reviews, with encouragement to publish them electronically in The Cochrane Library to obtain additional feedback prior to proceeding to data collection and analysis.² The primary objective of our study was to assess the extent to which the content of published Cochrane reviews had changed compared with their previously published protocols. A secondary objective was to explore the potential impact of any changes made to published protocols, particularly in terms of the risk of introducing bias to the study.

We identified all new Cochrane reviews appearing for the first time in the Cochrane Library; 2000, issue 3. For each of these, we used the unique identifier attached to the review to identify whether a protocol had been published for the review in the previous issue of the Cochrane Library (2000, issue 2). When we identified a published protocol, we downloaded the text of both the protocol and the completed review into separate Microsoft Word (Microsoft Office 1998; Microsoft Inc, Redmond, Wash) files. We then compared and highlighted changes made to the published protocol in the completed review using the "document compare" function in Microsoft Word.

Each of these comparative documents were rated independently by 2 of the 3 raters. The raters assessed whether the authorship of the completed review had changed from that of the protocol and whether there had been changes in any of the 9 sections of Cochrane review protocols: background, objectives, comparisons, types of studies, types of participants, types of interventions, types of outcomes, search strategy, and methods. For each section, the raters were required to identify whether a change had occurred and, if so, whether it was a minor or major change.

For the purpose of this study, we did not regard alterations in tense or abbreviations as a change. Changes in style or wording (other than changes in tense or abbreviations) that did not alter the substance or meaning of a section and would not have altered the interpretation by someone else replicating the review were considered minor changes. Any other change that altered the substance or meaning of a section or would have altered the interpretation by someone else replicating the review was regarded as a major change. When the raters disagreed on the categorization of any changes, disagreements were resolved by discussion.

We examined the specific sections in which a major change had occurred and sought to describe the nature of the changes and their potential for introducing bias into the review. We also approached the authors of reviews in which a major change had occurred and asked them to explain the reasons for making the change.

We identified 66 completed reviews appearing for the first time in the Cochrane Library; 2000, issue 3. Of these, we found a protocol for 71% (n = 47) in 2000, issue 2, of the Cochrane Library. None of these published protocols had any external comments or criticisms attached. The remaining 19 reviews had apparently been carried out without prior publication of a protocol.

Only 1 of the 47 reviews with a prior protocol remained unchanged compared with the published protocol. A further 3 reviews had only minor changes, eg, grammatical changes, from the published protocol. The remaining 91% of reviews (n = 43) had at least 1 section that had undergone a major change compared with the previously published protocol. There were changes to all 9 sections examined in 1 review. The median number of major changes per review was 2. A summary of changes identified in the completed reviews compared with their published protocols is shown in Table 1. Interrater reliability was high, with 80% agreement between raters.

The authors were unchanged in 68% of the reviews (n = 32) compared with previously published protocols. In 28% (n = 13), additional authors were included in the final review and in 4% (n = 2), authors who had been listed in the published protocol did not appear in the completed review.

Major changes were made to the background section in 53% of reviews (n = 25). Most of these involved inclusion of additional descriptive epidemiological data or information to enhance the rationale for the proposed review.

In all but 1 of 16 reviews, there had been major changes to the stated objectives of the review involving either a clarification or broadening of the original objectives. Studies for which the objectives were clarified usually involved inclusion of more-specific definitions or caveats regarding a particular term (eg, from "pain" to "late postoperative pain"). In studies in which the objectives were broadened, this usually involved consideration of safety in addition to efficacy.

Only 25 of the 47 protocols had prestated comparisons. Of these, 48% made major changes in the completed review. Three of the reviews removed all mention of specific comparisons that had been identified in the protocol. A further 9 reviews added or modified comparisons. When existing comparisons were modified, this invariably involved clarification that the intervention treatment was being compared with no treatment as well as with some form of new treatment.

Among the 15 reviews in which there were major changes to the types of studies included, 1 involved a change to more-restrictive study design criteria, whereas 4 involved a broadening of the types of studies included beyond randomized controlled trials alone to other potentially less-robust forms of comparison. In a further 5 reviews, the changes involved inclusion of additional methodological criteria (eg, deciding to accept unpublished data, requirement for trials to be double blind, requirement for nonaggregated data from individual participants to be available, or need for a minimum proportion of patients to have been followed up). In the remaining 5 reviews, the changes were clarifications of a descriptive feature of the studies eligible for inclusion.

Sixteen reviews made major changes to the types of participants included in the review. The nature of changes to the section describing types of participants was essentially to refine or add criteria for prestated diagnoses of inclusion.

Twenty-three reviews made major changes to this section of the protocol. Changes to types of interventions fell into 3 broad categories: clarification of terms used to describe or define the intervention, simplification of multiple interventions to a broad all-encompassing heading, and more detailed specification under a broad heading. When changes had been made to the objectives or comparisons, the changes to types of interventions were generally to ensure consistency with the changes to these other sections of the review.

Of the 22 studies with major changes to the types of outcomes, two thirds involved the addition of new outcomes or new measurement methods for existing outcomes. Nine studies added the measurement of adverse effects and quality-of-life measures as specific outcomes and 7 specified new measurement instruments or criteria for existing outcomes. In the remaining 6 studies, outcome measures present in the protocol were removed from the final review.

Changes to the search strategy occurred in 23 reviews. These involved either addition or deletion of specific bibliographic databases and terms (or simplifying the presentation of a comprehensive strategy by reference to the relevant Cochrane Review Group's search strategy), modifying the dates for which the search was conducted, or reversing a prior decision to seek unpublished trials.

The methods section had the greatest variation between protocol and review; 32 reviews (68%) had changes in the type of information presented. Some changes simply incorporated greater detail of the methods of analyses, whereas others involved more substantive additions or deletions of actual or proposed subgroup analyses. Other major amendments within this section were changes to the lists of information to be extracted from the component studies and changes to the criteria for decision making relating to the process of the review (eg, changes to the number of data extractors for a review and the method for resolving any differences between them).

We wrote to the contact reviewer of the 43 reviews in which we had found a major change to 1 or more sections compared with the published protocol, of whom 65% (n = 28) responded. All of the reviewers gave pragmatic reasons for making the changes that were unrelated to knowledge of the results or a perception that bias was being deliberately introduced into the study. The reviewers' reasons for the changes included: as a response to editorial suggestions (n = 10); to improve clarity (n = 7); because the review changed in scope as it progressed (n = 6); to reflect what was actually done (n = 3); because the reviewer learned "on the job" (eg, learning how to analyze survival curves) (n = 4); and to fit other allied protocols (n = 3). Reviewers may have given more than 1 reason for making a change to their review.

This study shows that of the 71% of Cochrane reviews with a published protocol, all but 1 underwent at least some changes in the completed review. These changes were minor in only 3 reviews. All of the remaining 43 reviews had at least 1 major change, and many reviews had changes in 5 or more sections compared with the published protocol. Two questions arise from this situation: (1) Do these changes matter? and (2) What value should be placed on the published protocol?

Many of the changes to the protocol, even when defined as major, may have improved the overall quality of the review without introducing any significant bias. For example, the inclusion of more detail in the background section of the review is unlikely to cause bias. Similarly, when the authors of the review increased the comprehensiveness of the search strategy or the security of the data extraction methods or clarified the statistical methods to be used, this was likely to enhance the overall quality of the review.

However, changes made to the protocol that could have been based on prior knowledge of the results are of greater concern. These changes may have introduced the possibility of bias, either intentionally or unintentionally, into the review. Narrowing of the scope of the review in the form of more restricted objectives or omission of types of comparisons may be a source of bias, given that the omitted comparisons may show an intervention to be more or less effective than the retained comparisons. The additions and deletions of types of studies and deletions to search strategies could also potentially cause bias. Addition or deletion of outcome measures has a potential for bias, especially when the reviewers are aware of study results. For example, an outcome may be deleted because its results do not "fit"' the other results. A change from more rigorous methods to a looser set of methods in the review could introduce bias. However, this study suggests that the authors were more likely to add greater detail to their methods. Some of these additions, such as changing from a fixed-effects to a random-effects model, use of a different effect measure, and addition or deletion of subgroup analyses, could be sources of bias, especially without sensitivity analyses to test the effect of such changes.

Based on the feedback received from the authors of reviews that had undergone a major change, it is clear that some of the changes were made because of knowledge of studies that existed but not necessarily because of knowledge of their results. While these responses do not exclude the possibility that bias is unintentionally introduced in this way, it appears that most of the changes to protocols are made for pragmatic reasons and as a result of feedback from the peer review process, to improve the final review.

Research protocols, even if published, are likely to remain iterative documents, at least to some extent. It would have been surprising, therefore, if there had not been some changes to the published Cochrane protocols before the publication of the reviews. The Cochrane Library provides a facility for external comments and criticisms to be attached to both protocols and completed reviews. However, at present, these comments seem to be almost entirely confined to completed reviews. None of the protocols included in this study had received external comments through this system. Of the 1013 protocols in the current issue of the Cochrane Library; 2002, issue 1, only 2 have comments attached. This lack of external comments or criticisms on published protocols is disappointing, particularly since 31 (44%) of the 71 comments attached to completed reviews in the same issue of the Cochrane Library were related to aspects of the protocol but did not appear to have been made until the full review had been published.

Although few comments are made about published protocols in the form of published criticisms, the process is not without value. Protocols still serve as a detailed guide for a wide audience to research in progress and may help contribute to reduced duplication of effort by others who are planning to do similar research. The process of publishing protocols may also minimize the likelihood of reviewers introducing bias into their review by making major changes that might otherwise remain undisclosed and undetectable. The publication of both protocols and completed reviews allows the tracking of any changes that have taken place and an examination of what effect this may have had on the results of the review. One strategy that might be used in the future would be to indicate any changes made (and reasons, where appropriate) from the original study protocol. This would allow readers to formulate their own judgment as to the effect such changes may have had on the final review.