[Skip to Content]
[Skip to Content Landing]
Views 944
Citations 0
Original Investigation
September 5, 2019

Evaluation of Reproducible Research Practices in Oncology Systematic Reviews With Meta-analyses Referenced by National Comprehensive Cancer Network Guidelines

Author Affiliations
  • 1Department of Biomedical Sciences, Oklahoma State University Center for Health Sciences, Tulsa
  • 2School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
  • 3Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa
JAMA Oncol. Published online September 5, 2019. doi:10.1001/jamaoncol.2019.2564
Key Points

Question  To what extent do clinically relevant oncology systematic reviews cited by National Comprehensive Cancer Network guidelines use reproducible research practices?

Findings  In this cross-sectional study of 154 oncology meta-analyses comprising 3696 meta-analytic effect sizes, 2375 (64.3%), including subgroup and sensitivity analyses, were reproducible in theory, with the main driver of reproducibility being whether a meta-analysis was presented in a forest plot. Authors infrequently described how missing data were handled, and only 1 meta-analysis provided a link to a data set.

Meaning  An emphasis on the reporting of meta-analytic effects in forest plots and requirements for providing access to data sets would strengthen the reproducibility of oncology meta-analyses.

Abstract

Importance  Reproducible research practices are essential to biomedical research because these practices promote trustworthy evidence. In systematic reviews and meta-analyses, reproducible research practices ensure that summary effects used to guide patient care are stable and trustworthy.

Objective  To evaluate the reproducibility in theory of meta-analyses in oncology systematic reviews cited by the 49 National Comprehensive Cancer Network (NCCN) guidelines for the treatment of cancer by site and evaluate whether Cochrane reviews or systematic reviews that report adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines use more reproducible research practices.

Design, Setting, and Participants  A cross-sectional investigation of all systematic reviews with at least 1 meta-analysis and at least 1 included randomized clinical trial (RCT) that are cited by NCCN guidelines for treatment of cancer by site. We scanned the reference list of all NCCN guidelines (n = 49) for potential systematic reviews and meta-analyses. All retrieved studies were screened, and data were extracted, independently and in duplicate. The analysis was carried out between May 6, 2018, and January 28, 2019.

Main Outcomes and Measures  The frequency of reproducible research practices, defined as (1) effect estimate and measure of precision (eg, hazard ratio with 95% confidence interval); (2) clear list of studies included for each analysis; and (3) for subgroup and sensitivity analyses, it must be clear which studies were included in each group or level.

Results  We identified 1124 potential systematic reviews, and 154 meta-analyses comprising 3696 meta-analytic effect size estimates were included. Only 2375 of the 3696 meta-analytic estimates (64.3%), including subgroup and sensitivity analyses, were reproducible in theory. Forest plots appear to improve the reproducibility of meta-analyses. All meta-analytic estimates were reproducible in theory in 100 systematic reviews (64.9%), and in 15 systematic reviews (9.7%), no meta-analytic estimates could potentially be reproduced. Data were said to be imputed in 29 meta-analyses, but none specified which data. Only 1 meta-analysis included a link to an online data set.

Conclusions and Relevance  More reproducible research practices are needed in oncology meta-analyses, as suggested by those that are cited by the NCCN. Reporting meta-analyses in forest plots and requirements for full data sharing are recommended.

×