Assessment of Publication Trends of Systematic Reviews and Randomized Clinical Trials, 1995 to 2017

Systematic reviews and meta-analyses (SRMAs) and randomized clinical trials (RCTs) are considered the most robust and reliable forms of evidence to guide clinical practice. Previous research has demonstrated year-over-year increases in the number of published RCTs between 1950 and 2007¹ as well as increases in the number of published SRMAs through 2016.^2,3 The increase in SRMAs is needed to update cumulative evidence,² although some investigators speculate that SRMAs may also serve as “easily publishable units or marketing tools.”^2,3 Given this context, we sought to compare publication trends overall and across clinical topic areas among SRMAs and RCTs over the past 22 years.

We conducted a cross-sectional study of PubMed-indexed SRMAs and RCTs published from 1995 to 2017 using the UNIX terminal window Entrez Direct (EDirect). EDirect is the primary text search and retrieval system of the National Center for Biotechnology Information. The inclusion start period was set to 1995 to account for previous systematic errors in PubMed’s categorization of SRMAs prior to this time period.³ Systematic reviews and meta-analyses were searched as a single category because PubMed indexes meta-analyses within systematic reviews, and up to 60% of systematic reviews include meta-analyses (Figure 1).⁴

Medical subject headings (MeSH) were used to define clinical topic areas when the term was a major topic of an article using the following heuristic for MeSH categories: medical specialty, surgical specialty, surgical procedure, disease, and anatomic system where applicable. Searches for SRMAs used the terms Systematic Review[Ptyp] OR Meta-Analysis[Ptyp], whereas RCT searches used Randomized Controlled Trial[Ptyp]. The 18 medical and surgical topic areas included in this study are noted in Figure 2, with an example of a search strategy noted in the legend. Standard identifiers (PubMed identification numbers) indexed across more than 1 specialty were only counted once. The ratio of SRMAs to RCTs was calculated for each year. A ratio greater than 1 indicates that more SRMAs than RCTs were published, whereas a ratio less than 1 indicates that more RCTs than SRMAs were published. Data analysis was performed from February 1 to February 12, 2018, and Stata version 15 (StataCorp) was used for all analyses.

From 1995 to 2017, increases were observed in the absolute number of published SRMAs (435 in 1995 vs 20 774 in 2017) and RCTs (9486 in 1995 vs 22 560 in 2017); however, the rate of growth was significantly greater for SRMAs vs RCTs at 4676% and 138%, respectively (Figure 1). In 1995, the overall ratio (SD) of SRMAs to RCTs was 0.045 (0.02), whereas in 2017 it was 0.871 (0.26). Increases in published SRMAs and RCTs were observed for all 18 clinical topic areas (Figure 2). In 1995, the lowest ratio of SRMAs to RCTs was observed for anesthesiology (0.005) and the highest was observed for hematology/oncology (0.083); in 2017, the lowest ratio was observed for anesthesiology (0.317) and the highest was observed for hematology/oncology (1.443).

The number of published SRMAs and RCTs has substantially increased over the last 22 years, although the rate of growth was notably greater for SRMAs. These findings update those of previous studies and are consistent with earlier studies estimating an approximately 2700% increase of SRMA indexed in PubMed.^2,3 This increase may be secondary to the incorporation of the larger numbers of RCTs into SRMAs, incorporation of nonrandomized studies in SRMAs,⁵ and/or the proliferation of SRMAs conducted by researchers in China, who now account for production of more than one-third of all published meta-analyses.³

This study was limited by the use of PubMed, which may not be representative of overall trends in the literature.³ Additionally, our search criteria relied on the National Library of Medicine’s controlled vocabulary thesaurus, MeSH, instead of keywords to extract indexed papers.

Systematic reviews and meta-analyses help to synthesize and update the literature using valuable methods for evidence-based medicine. However, an estimated 3% of SRMAs are methodologically sound, nonredundant, and provide useful clinical information.³ Although the optimal SRMA/RCT ratio has yet to be determined, an ever increasing proportion of this literature may provide minimal value, which should precipitate a reappraisal of the foundations, production, and reporting of SRMAs.⁶

Accepted for Publication: June 1, 2019.

Corresponding Author: Joshua D. Niforatos, MD, MTS, Cleveland Clinic Lerner College of Medicine, 9500 Euclid Ave, Na/21, Cleveland, OH 44195 (jxn187@case.edu).

Published Online: July 29, 2019. doi:10.1001/jamainternmed.2019.3013

Author Contributions: All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Niforatos.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Niforatos, Johansen.

Administrative, technical, or material support: Weaver, Johansen.

Conflict of Interest Disclosures: None reported.