From the Health Information Research Unit (Dr Jadad), Department of Clinical Epidemiology and Biostatistics (Drs Jadad and Cook), and the Department of Medicine (Dr Cook), McMaster University, Hamilton, Ontario; Thomas C. Chalmers Center for Systematic Reviews, Children's Hospital of Eastern Ontario Research Institute, Ottawa, Ontario (Ms Jones, Drs Klassen and Tugwell, and Mr D. Moher); the Departments of Pediatrics (Mr D. Moher) and Medicine (Dr Tugwell and Mr D. Moher), University of Ottawa, Ottawa, Ontario; and Department of Primary Health Care, University of Oxford, Oxford, England (Dr M. Moher).
Context.— Review articles are important sources of information to help guide decisions
by clinicians, patients, and other decision makers. Ideally, reviews should
include strategies to minimize bias and to maximize precision and be reported
so explicitly that any interested reader would be able to replicate them.
Objective.— To compare the methodological and reporting aspects of systematic reviews
and meta-analyses published by the Cochrane Collaboration with those published
in paper-based journals indexed in MEDLINE.
Data Sources.— The Cochrane Library, issue 2 of 1995, and a search of MEDLINE restricted
Study Selection.— All 36 completed reviews published in the Cochrane Database of Systematic
Reviews and a randomly selected sample of 39 meta-analyses or systematic reviews
published in journals indexed by MEDLINE in 1995.
Data Extraction.— Number of authors, trials, and patients; trial sources; inclusion and
exclusion criteria; language restrictions; primary outcome; trial quality
assessment; heterogeneity testing; and effect estimates. Updating by 1997
Results.— Reviews found in MEDLINE included more authors (median, 3 vs 2; P<.001), more trials (median, 13.5 vs 5; P<.001), and more patients (median, 1280 vs 528; P <.001) than Cochrane reviews. More Cochrane reviews, however,
included a description of the inclusion and exclusion criteria (35/36 vs 18/39; P<.001) and assessed trial quality (36/36 vs 12/39; P<.001). No Cochrane reviews had language restrictions
(0/36 vs 7/39; P<.01). There were no differences
in sources of trials, heterogeneity testing, or description of effect estimates.
By June 1997, 18 of 36 Cochrane reviews had been updated vs 1 of 39 reviews
listed in MEDLINE.
Conclusions.— Cochrane reviews appear to have greater methodological rigor and are
more frequently updated than systematic reviews or meta-analyses published
in paper-based journals.
THE COCHRANE Collaboration (CC) is an international organization that
aims to help people make well-informed decisions about health care by preparing,
maintaining, and promoting the accessibility of systematic reviews on the
effects of health care interventions.1,2
The reviews produced within the CC are prepared following standardized instructions
published in the Cochrane Handbook, which is widely
available to reviewers in paper, CD-ROM, or Internet format. These reviews
are included in the Cochrane Database of Systematic Reviews, a quarterly electronic publication.1,2
This electronic format allows reviewers to update or modify the Cochrane reviews
in response to new evidence or comments and criticisms from readers. To safeguard
the rigor and relevance of the Cochrane reviews, the CC has several levels
of peer review, including assessment of protocols by editors and reviewers
and evaluation of the reviews by methodology and content experts and by potential
There are major differences between the peer review process followed
by most paper-based journals and the system used by the CC. Few paper-based
journals provide authors with explicit instructions to report systematic reviews
and meta-analyses; their editors and peer reviewers only provide input after
the reviews have been completed. Once the reviews are published there is limited
opportunity for corrections of omissions or mistakes and few incentives to
produce and publish updated versions.
It has been suggested recently that Cochrane reviews are less prone
to bias than systematic reviews or meta-analyses published in paper-based
journals.3 To date, however, we have been unable
to identify other studies assessing the impact of the CC on the methodology
and reports of systematic reviews and meta-analyses.
Against this background, we designed a study to (1) compare the methodological
and reporting aspects of Cochrane reviews with those published in paper-based
journals and (2) evaluate the frequency with which systematic reviews and
meta-analyses, published by the CC or in paper-based journals, are updated.
Reviews were included in the study if they included only randomized
controlled trials, and if they were published in the Cochrane Database of
Systematic Reviews or in paper-based journals indexed by MEDLINE. For the
latter, they had to be coded by indexers as meta-analyses, or described by
the authors as systematic reviews, meta-analyses, integrative research reviews,
overviews, quantitative syntheses, and pooling or combining studies.
For the purposes of this study, all 36 Cochrane reviews published in
the second issue of 1995 of the Cochrane Library
(Update Software, Oxford, England) were included. They were compared with
39 randomly selected paper-based systematic reviews and meta-analyses, published
in 32 different journals, identified using a refined search strategy of MEDLINE
restricted to 1995 (Table 1).
To ascertain the rate with which reviews are updated, the reviews published
in 1995 and included in this study were followed up until June 1997 by searching
the second issue of 1997 of the Cochrane Library
and using the same MEDLINE strategy restricted to the period from January
1996 to June 1997.
Hard copies of all the reviews were obtained, but the identity of the
authors and their affiliations were masked. The journal name was masked in
the paper-based reviews. We could not mask the source of the documents as
the reviews had different formats and we did not scan them. Using criteria
defined a priori, 2 of us (A.R.J. and D.M.), independently, extracted the
following information: number of authors, trials, and patients included; sources
of trials; description of primary outcomes, inclusion and exclusion criteria,
and pooled-effect estimates; and use of language restrictions, formal assessment
of trial quality, and heterogeneity testing. Once data extraction was completed,
the extractors met, compared their findings, and reached consensus. The number
of those reviews that had been updated by 1997 was also evaluated.
The Mann-Whitney U test was used to compare
the number of sources used and the number of trials and patients included
in the reviews; χ2 tests were used to compare the proportion
of reviews that described the inclusion and exclusion criteria, heterogenity
testing, and primary outcomes, and the Fisher exact test was used for the
number of reviews with language restrictions and with descriptions of the
quantitative effect estimates. Values of P <.05
were regarded as statistically significant.
Reviews published in paper-based journals included more authors (median,
3 vs 2; P<.001), more trials (median, 13.5 vs
5; P <.001), and more patients (median, 1280 vs
528; P <.001) than Cochrane reviews.
More Cochrane reviews, however, included a description of the inclusion
and exclusion criteria (35/36 vs 18/39; P<.001)
and assessed trial quality (36/36 vs 12/39; P<.001).
No Cochrane reviews had language restrictions (0/36 vs 7/39; P <.01). There were no statistically significant differences between
the Cochrane reviews and those published in paper-based journals in the number
of sources of trials (median, 3 vs 2; P =.12), the
frequency of heterogeneity testing (47% vs 54%; P
=.56), or the description of quantitative effect estimates (92% vs 90%; P =.55). None of the Cochrane reviews and only 3 of the
reviews published in paper-based journals described the primary outcome(s)
By June 1997, more Cochrane reviews had been updated (18/36 vs 1/39; P<.001).
Compared with reviews published in paper-based journals, Cochrane reviews
include elements that make them less prone to bias, such as the description
of inclusion and exclusion criteria and formal assessment of trial quality.4 The methodological deficiencies that we found in reviews
published in paper-based journals are similar to those reported by others.4- 6 Perhaps paper-based
journals could improve the methodology and reporting of the systematic reviews
they publish by influencing the review process at a much earlier stage and
by encouraging more frequent updates or correction of published material using
other media such as the Internet.
Almost all reviews included in this study used meta-analyses (quantitative
data synthesis), but only half of them incorporated formal assessments of
heterogeneity testing. This may indicate that most reviewers use meta-analysis
as the "default action" to synthesize the information provided by the individual
studies. The proportion of reviews that should include meta-analysis is unknown,
but is likely to vary depending on unique features of the topics and studies
The number of recorded sources of trials used by Cochrane reviews is
likely to be an underestimate, as most Cochrane reviews use specialized registers
that typically include extensive searches of bibliographic databases (ie,
MEDLINE) and hand searches of journals.
Most reviews did not specify the primary outcome(s). This could have
been due to oversight (unlikely), to the lack of instructions for authors,
to the lack of information on primary outcomes in the individual studies being
reviewed, or to the perception that reporting the primary outcome is not as
important in a review as it may be in an individual trial. The latter could
be acceptable from the users' perspectives, as they would be given the opportunity
to select freely the outcomes of greatest interest to them in different circumstances.
However, from the researchers' perspective, the lack of clearly identified
primary outcomes hinders empirical methodological studies designed to assess
the effect of different characteristics of the reviews (ie, language restrictions,
quality assessment, etc) on the direction of the results.
Cochrane reviews were updated more frequently than their paper-based
counterparts. It is unclear, however, whether 50% is below or above the optimal
update rate for systematic reviews. The comparatively low update rate among
paper-based reviews suggests that editors of such journals are not sufficiently
interested in publishing updated versions of previously published systematic
reviews and meta-analyses or, if they are interested, that authors are not
aware of such interest. Alternatively, authors of reviews published in paper-based
journals may lack the interest or the resources to update them. We did not
contact and ask the authors of these nonupdated reviews because of our intention
to follow up the reviews in time. Regardless of the causes for not updating
reviews, these reviews could be updated within the framework and with the
support of the CC. Alternatively, it would be worthwhile for authors and editors
to consider the publication of any updated review as research letters.
Our study also had limitations. The inclusion of reviews published in
journals indexed only by MEDLINE limits the generalization of the results
to reviews indexed in other databases. The lack of masking as to the origin
of the reviews (Cochrane or MEDLINE) and the close links of most of us with
the CC could have also introduced bias. Data extraction under masked conditions,
however, has been shown to have little effect on systematic reviews.7
We will follow up these 2 groups of reviews to judge the impact of a
more mature set of tools produced by the CC, particularly the Cochrane Criticisms
Editor, as well as any set of standardized instructions for authors of systematic
reviews and meta-analyses produced by paper-based journals. The Cochrane Criticisms
Editor is an electronic facility included in the Cochrane
Library for the delivery of comments on Cochrane reviews to editors
and authors.2 This system is expected to further
improve the validity and relevance of the reviews.
Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D. Methodology and Reports of Systematic Reviews and Meta-analysesA Comparison of Cochrane Reviews With Articles Published in Paper-Based
Journals. JAMA. 1998;280(3):278–280. doi:10.1001/jama.280.3.278