[Skip to Content]
[Skip to Content Landing]
Figure.
Estimated Overall Odds Ratios of 6 Treatment Comparisons in the Network Meta-analysis of Smoking Cessation Using Stata
Estimated Overall Odds Ratios of 6 Treatment Comparisons in the Network Meta-analysis of Smoking Cessation Using Stata

Centers of boxes indicate odds ratios, with edges representing 95% CIs. Whiskers represent 95% prediction intervals (PIs).

1.
Hutton  B, Salanti  G, Caldwell  DM,  et al.  The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations.  Ann Intern Med. 2015;162(11):777-784. doi:10.7326/M14-2385PubMedGoogle ScholarCrossref
2.
Tonin  FS, Borba  HH, Leonart  LP,  et al.  Methodological quality assessment of network meta-analysis of drug interventions: implications from a systematic review.  Int J Epidemiol. 2018;48(2):620-632. doi:10.1093/ije/dyy197PubMedGoogle ScholarCrossref
3.
Riley  RD, Higgins  JPT, Deeks  JJ.  Interpretation of random effects meta-analyses.  BMJ. 2011;342:d549. doi:10.1136/bmj.d549PubMedGoogle ScholarCrossref
4.
IntHout  J, Ioannidis  JP, Rovers  MM, Goeman  JJ.  Plea for routinely presenting prediction intervals in meta-analysis.  BMJ Open. 2016;6(7):e010247. doi:10.1136/bmjopen-2015-010247PubMedGoogle ScholarCrossref
5.
Borenstein  M, Higgins  JPT, Hedges  LV, Rothstein  HR.  Basics of meta-analysis: I2 is not an absolute measure of heterogeneity.  Res Synth Methods. 2017;8(1):5-18. doi:10.1002/jrsm.1230PubMedGoogle ScholarCrossref
6.
Hasselblad  V.  Meta-analysis of multitreatment studies.  Med Decis Making. 1998;18(1):37-43. doi:10.1177/0272989X9801800110PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Research Letter
    Statistics and Research Methods
    August 21, 2019

    Use of Prediction Intervals in Network Meta-analysis

    Author Affiliations
    • 1Department of Statistics, Florida State University, Tallahassee
    JAMA Netw Open. 2019;2(8):e199735. doi:10.1001/jamanetworkopen.2019.9735
    Introduction

    Network meta-analysis (NMA) is an increasingly popular tool to synthesize direct and indirect evidence for simultaneously comparing multiple treatments. Although guidelines have been developed to improve NMAs,1 the quality of methodology and reporting of many NMAs is inconsistent with the rapid growth of their publication rates.2 Within the context of conventional pairwise meta-analyses, it has been advocated to routinely report 95% prediction (or predictive) intervals (PIs) alongside 95% CIs.3,4 Prediction intervals give the range within which the results of a future study might lie. They can also be used to effectively appraise heterogeneity and avoid the problems of the popular I2 statistic.5 However, the practice of using PIs is uncommon in NMAs.

    As in conventional meta-analyses, heterogeneity frequently exists in NMAs owing to differences in study setting, participants, treatment definitions, and so on. This may affect the validity and interpretation of the NMA. To account for heterogeneity, random-effects models are usually applied, and evidence users often focus on the summarized overall results with 95% CIs.

    Confidence intervals and PIs are used for different purposes. Suppose an NMA compares multiple treatments for obesity, with its primary outcome as weight loss. Confidence intervals describe uncertainty in overall weight loss for various treatment comparisons among the populations in the NMA. As such, their widths typically shrink toward 0 as the number of studies increases. In addition to uncertainty in overall results, PIs’ widths incorporate the variation caused by heterogeneity between different studies. When a new patient comes to the clinic, PIs (rather than CIs) should be used to predict the new patient’s weight loss and thus recommend the optimal treatment.

    Methods

    To investigate the use of PIs in NMAs, relevant articles published in JAMA, The Lancet, and BMJ from January 1, 2010, to December 31, 2018, were identified. Full-length articles with original data were the focus. The search procedures are provided in eAppendix 1 in the Supplement. This study did not require institutional review board approval because it focuses on statistical methods and uses published data in the literature. Its reporting follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

    In addition, to help clinicians understand how PIs can be produced in NMAs, available software programs and their implementations were illustrated via a worked example. This example used a classic NMA data set initially reported by Hasselblad6 and based on 24 studies. This NMA compared 4 treatments for smoking cessation (a binary outcome) as follows: (1) no intervention, (2) self-help, (3) individual counseling, and (4) group counseling.6 The estimated overall odds ratios of all treatment comparisons with 95% CIs and 95% PIs are reported. Statistical significance was prespecified at α = .05.

    Results

    Fifty-eight full-length articles on NMAs with original data were identified (eTable 1 in the Supplement). JAMA, The Lancet, and BMJ published 12 (21%), 11 (19%), and 35 (60%) NMA articles, respectively. Only 5 articles (9%) reported PIs, all published in BMJ.

    Software programs for producing PIs in NMAs appear in eTable 2 in the Supplement. Detailed instructions and results using these software programs appear in eAppendix 2 and the eFigure in the Supplement. The Figure shows the results of the NMA of smoking cessation using Stata version 13 (StataCorp). The 95% CIs of individual counseling vs no intervention and group counseling vs no intervention were above the null value 1.0. The overall treatment effects of individual counseling and group counseling were significantly better than no intervention among the 24 studies; they led to higher smoking cessation rates. The remaining 4 comparisons were nonsignificant. Nevertheless, the 95% PIs of all comparisons were wider than the corresponding 95% CIs and contained 1.0; thus, they indicated substantial heterogeneity between studies. Owing to such heterogeneity, the overall effects of individual counseling and group counseling could be null or even in the opposite direction for patients in a new study.4

    Discussion

    A limitation of this study is that detailed statistical models used in different software programs were not explored. They may make different assumptions and thus have some impact on the NMA results (eFigure in the Supplement).

    In summary, the reporting rate of PIs was low among NMAs, but they are crucial for interpreting results within a future study setting and have been advocated for conventional meta-analyses. The low reporting rate was possibly because most NMA guidelines lack recommendations about PIs and performing NMAs is relatively complicated. Nevertheless, as shown in the worked example, several software programs are available to feasibly produce PIs in NMAs.

    Back to top
    Article Information

    Accepted for Publication: July 1, 2019.

    Published: August 21, 2019. doi:10.1001/jamanetworkopen.2019.9735

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Lin L. JAMA Network Open.

    Corresponding Author: Lifeng Lin, PhD, Department of Statistics, Florida State University, 201B OSB, 117 N Woodward Ave, Tallahassee, FL 32306 (linl@stat.fsu.edu).

    Author Contributions: Dr Lin had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Lin.

    Acquisition, analysis, or interpretation of data: Lin.

    Drafting of the manuscript: Lin.

    Critical revision of the manuscript for important intellectual content: Lin.

    Statistical analysis: Lin.

    Conflict of Interest Disclosures: None reported.

    Funding/Support: Dr Lin was supported in part by the Committee on Faculty Research Support program from the Florida State University Council on Research and Creativity.

    Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    References
    1.
    Hutton  B, Salanti  G, Caldwell  DM,  et al.  The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations.  Ann Intern Med. 2015;162(11):777-784. doi:10.7326/M14-2385PubMedGoogle ScholarCrossref
    2.
    Tonin  FS, Borba  HH, Leonart  LP,  et al.  Methodological quality assessment of network meta-analysis of drug interventions: implications from a systematic review.  Int J Epidemiol. 2018;48(2):620-632. doi:10.1093/ije/dyy197PubMedGoogle ScholarCrossref
    3.
    Riley  RD, Higgins  JPT, Deeks  JJ.  Interpretation of random effects meta-analyses.  BMJ. 2011;342:d549. doi:10.1136/bmj.d549PubMedGoogle ScholarCrossref
    4.
    IntHout  J, Ioannidis  JP, Rovers  MM, Goeman  JJ.  Plea for routinely presenting prediction intervals in meta-analysis.  BMJ Open. 2016;6(7):e010247. doi:10.1136/bmjopen-2015-010247PubMedGoogle ScholarCrossref
    5.
    Borenstein  M, Higgins  JPT, Hedges  LV, Rothstein  HR.  Basics of meta-analysis: I2 is not an absolute measure of heterogeneity.  Res Synth Methods. 2017;8(1):5-18. doi:10.1002/jrsm.1230PubMedGoogle ScholarCrossref
    6.
    Hasselblad  V.  Meta-analysis of multitreatment studies.  Med Decis Making. 1998;18(1):37-43. doi:10.1177/0272989X9801800110PubMedGoogle ScholarCrossref
    ×