Methodology and Reports of Systematic Reviews and Meta-analyses: A Comparison of Cochrane Reviews With Articles Published in Paper-Based Journals

Context.— Review articles are important sources of information to help guide decisions by clinicians, patients, and other decision makers. Ideally, reviews should include strategies to minimize bias and to maximize precision and be reported so explicitly that any interested reader would be able to replicate them.

Objective.— To compare the methodological and reporting aspects of systematic reviews and meta-analyses published by the Cochrane Collaboration with those published in paper-based journals indexed in MEDLINE.

Data Sources.— The Cochrane Library, issue 2 of 1995, and a search of MEDLINE restricted to 1995.

Study Selection.— All 36 completed reviews published in the Cochrane Database of Systematic Reviews and a randomly selected sample of 39 meta-analyses or systematic reviews published in journals indexed by MEDLINE in 1995.

Data Extraction.— Number of authors, trials, and patients; trial sources; inclusion and exclusion criteria; language restrictions; primary outcome; trial quality assessment; heterogeneity testing; and effect estimates. Updating by 1997 was evaluated.

Results.— Reviews found in MEDLINE included more authors (median, 3 vs 2; P<.001), more trials (median, 13.5 vs 5; P<.001), and more patients (median, 1280 vs 528; P <.001) than Cochrane reviews. More Cochrane reviews, however, included a description of the inclusion and exclusion criteria (35/36 vs 18/39; P<.001) and assessed trial quality (36/36 vs 12/39; P<.001). No Cochrane reviews had language restrictions (0/36 vs 7/39; P<.01). There were no differences in sources of trials, heterogeneity testing, or description of effect estimates. By June 1997, 18 of 36 Cochrane reviews had been updated vs 1 of 39 reviews listed in MEDLINE.

Conclusions.— Cochrane reviews appear to have greater methodological rigor and are more frequently updated than systematic reviews or meta-analyses published in paper-based journals.

THE COCHRANE Collaboration (CC) is an international organization that aims to help people make well-informed decisions about health care by preparing, maintaining, and promoting the accessibility of systematic reviews on the effects of health care interventions.^1,2 The reviews produced within the CC are prepared following standardized instructions published in the Cochrane Handbook, which is widely available to reviewers in paper, CD-ROM, or Internet format. These reviews are included in the Cochrane Database of Systematic Reviews, a quarterly electronic publication.^1,2 This electronic format allows reviewers to update or modify the Cochrane reviews in response to new evidence or comments and criticisms from readers. To safeguard the rigor and relevance of the Cochrane reviews, the CC has several levels of peer review, including assessment of protocols by editors and reviewers and evaluation of the reviews by methodology and content experts and by potential users.^1,2

There are major differences between the peer review process followed by most paper-based journals and the system used by the CC. Few paper-based journals provide authors with explicit instructions to report systematic reviews and meta-analyses; their editors and peer reviewers only provide input after the reviews have been completed. Once the reviews are published there is limited opportunity for corrections of omissions or mistakes and few incentives to produce and publish updated versions.

It has been suggested recently that Cochrane reviews are less prone to bias than systematic reviews or meta-analyses published in paper-based journals.³ To date, however, we have been unable to identify other studies assessing the impact of the CC on the methodology and reports of systematic reviews and meta-analyses.

Against this background, we designed a study to (1) compare the methodological and reporting aspects of Cochrane reviews with those published in paper-based journals and (2) evaluate the frequency with which systematic reviews and meta-analyses, published by the CC or in paper-based journals, are updated.

Reviews were included in the study if they included only randomized controlled trials, and if they were published in the Cochrane Database of Systematic Reviews or in paper-based journals indexed by MEDLINE. For the latter, they had to be coded by indexers as meta-analyses, or described by the authors as systematic reviews, meta-analyses, integrative research reviews, overviews, quantitative syntheses, and pooling or combining studies.

For the purposes of this study, all 36 Cochrane reviews published in the second issue of 1995 of the Cochrane Library (Update Software, Oxford, England) were included. They were compared with 39 randomly selected paper-based systematic reviews and meta-analyses, published in 32 different journals, identified using a refined search strategy of MEDLINE restricted to 1995 (Table 1). To ascertain the rate with which reviews are updated, the reviews published in 1995 and included in this study were followed up until June 1997 by searching the second issue of 1997 of the Cochrane Library and using the same MEDLINE strategy restricted to the period from January 1996 to June 1997.

Hard copies of all the reviews were obtained, but the identity of the authors and their affiliations were masked. The journal name was masked in the paper-based reviews. We could not mask the source of the documents as the reviews had different formats and we did not scan them. Using criteria defined a priori, 2 of us (A.R.J. and D.M.), independently, extracted the following information: number of authors, trials, and patients included; sources of trials; description of primary outcomes, inclusion and exclusion criteria, and pooled-effect estimates; and use of language restrictions, formal assessment of trial quality, and heterogeneity testing. Once data extraction was completed, the extractors met, compared their findings, and reached consensus. The number of those reviews that had been updated by 1997 was also evaluated.

The Mann-Whitney U test was used to compare the number of sources used and the number of trials and patients included in the reviews; χ² tests were used to compare the proportion of reviews that described the inclusion and exclusion criteria, heterogenity testing, and primary outcomes, and the Fisher exact test was used for the number of reviews with language restrictions and with descriptions of the quantitative effect estimates. Values of P <.05 were regarded as statistically significant.

Reviews published in paper-based journals included more authors (median, 3 vs 2; P<.001), more trials (median, 13.5 vs 5; P <.001), and more patients (median, 1280 vs 528; P <.001) than Cochrane reviews.

More Cochrane reviews, however, included a description of the inclusion and exclusion criteria (35/36 vs 18/39; P<.001) and assessed trial quality (36/36 vs 12/39; P<.001). No Cochrane reviews had language restrictions (0/36 vs 7/39; P <.01). There were no statistically significant differences between the Cochrane reviews and those published in paper-based journals in the number of sources of trials (median, 3 vs 2; P =.12), the frequency of heterogeneity testing (47% vs 54%; P =.56), or the description of quantitative effect estimates (92% vs 90%; P =.55). None of the Cochrane reviews and only 3 of the reviews published in paper-based journals described the primary outcome(s) of interest.

By June 1997, more Cochrane reviews had been updated (18/36 vs 1/39; P<.001).

Compared with reviews published in paper-based journals, Cochrane reviews include elements that make them less prone to bias, such as the description of inclusion and exclusion criteria and formal assessment of trial quality.⁴ The methodological deficiencies that we found in reviews published in paper-based journals are similar to those reported by others.^4-6 Perhaps paper-based journals could improve the methodology and reporting of the systematic reviews they publish by influencing the review process at a much earlier stage and by encouraging more frequent updates or correction of published material using other media such as the Internet.

Almost all reviews included in this study used meta-analyses (quantitative data synthesis), but only half of them incorporated formal assessments of heterogeneity testing. This may indicate that most reviewers use meta-analysis as the "default action" to synthesize the information provided by the individual studies. The proportion of reviews that should include meta-analysis is unknown, but is likely to vary depending on unique features of the topics and studies being summarized.

The number of recorded sources of trials used by Cochrane reviews is likely to be an underestimate, as most Cochrane reviews use specialized registers that typically include extensive searches of bibliographic databases (ie, MEDLINE) and hand searches of journals.

Most reviews did not specify the primary outcome(s). This could have been due to oversight (unlikely), to the lack of instructions for authors, to the lack of information on primary outcomes in the individual studies being reviewed, or to the perception that reporting the primary outcome is not as important in a review as it may be in an individual trial. The latter could be acceptable from the users' perspectives, as they would be given the opportunity to select freely the outcomes of greatest interest to them in different circumstances. However, from the researchers' perspective, the lack of clearly identified primary outcomes hinders empirical methodological studies designed to assess the effect of different characteristics of the reviews (ie, language restrictions, quality assessment, etc) on the direction of the results.

Cochrane reviews were updated more frequently than their paper-based counterparts. It is unclear, however, whether 50% is below or above the optimal update rate for systematic reviews. The comparatively low update rate among paper-based reviews suggests that editors of such journals are not sufficiently interested in publishing updated versions of previously published systematic reviews and meta-analyses or, if they are interested, that authors are not aware of such interest. Alternatively, authors of reviews published in paper-based journals may lack the interest or the resources to update them. We did not contact and ask the authors of these nonupdated reviews because of our intention to follow up the reviews in time. Regardless of the causes for not updating reviews, these reviews could be updated within the framework and with the support of the CC. Alternatively, it would be worthwhile for authors and editors to consider the publication of any updated review as research letters.

Our study also had limitations. The inclusion of reviews published in journals indexed only by MEDLINE limits the generalization of the results to reviews indexed in other databases. The lack of masking as to the origin of the reviews (Cochrane or MEDLINE) and the close links of most of us with the CC could have also introduced bias. Data extraction under masked conditions, however, has been shown to have little effect on systematic reviews.⁷

We will follow up these 2 groups of reviews to judge the impact of a more mature set of tools produced by the CC, particularly the Cochrane Criticisms Editor, as well as any set of standardized instructions for authors of systematic reviews and meta-analyses produced by paper-based journals. The Cochrane Criticisms Editor is an electronic facility included in the Cochrane Library for the delivery of comments on Cochrane reviews to editors and authors.² This system is expected to further improve the validity and relevance of the reviews.