[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.  Recruitment Periods Across Trials Comparing the Absorb Polymeric Everolimus-Eluting Bioresorbable Vascular Scaffold With the Metallic Everolimus-Eluting Stent
Recruitment Periods Across Trials Comparing the Absorb Polymeric Everolimus-Eluting Bioresorbable Vascular Scaffold With the Metallic Everolimus-Eluting Stent

The horizontal blue boxes indicate the recruitment period for each individual trial. Diamonds correspond to the publication of follow-up data for each study over time. BVS indicates bioresorbable vascular scaffolds; FDA, US Food and Drug Administration.

Figure 2.  Cumulative Meta-analysis of Trials Comparing the Absorb Polymeric Everolimus-Eluting Bioresorbable Vascular Scaffold With the Metallic Everolimus-Eluting Stent
Cumulative Meta-analysis of Trials Comparing the Absorb Polymeric Everolimus-Eluting Bioresorbable Vascular Scaffold With the Metallic Everolimus-Eluting Stent

Reports describing increasing follow-up durations for the same trial are ranked according to the date of becoming publicly available. To avoid duplicate counts, data from shorter follow-up periods were omitted from the analysis after data from longer follow-up periods became available. Squares indicate odds ratios (ORs), with horizontal lines representing 95% CIs. BVS indicates bioresorbable vascular scaffolds; CE, Conformité Européene; FDA, US Food and Drug Administration.

1.
Calis  KA, Archdeacon  P, Bain  R,  et al.  Recommendations for data monitoring committees from the Clinical Trials Transformation Initiative.   Clin Trials. 2017;14(4):342-348. doi:10.1177/1740774517707743PubMedGoogle ScholarCrossref
2.
Chalmers  I, Bracken  MB, Djulbegovic  B,  et al.  How to increase value and reduce waste when research priorities are set.   Lancet. 2014;383(9912):156-165. doi:10.1016/S0140-6736(13)62229-1PubMedGoogle ScholarCrossref
3.
Bender  R, Bunce  C, Clarke  M,  et al.  Attention should be given to multiplicity issues in systematic reviews.   J Clin Epidemiol. 2008;61(9):857-865. doi:10.1016/j.jclinepi.2008.03.004PubMedGoogle ScholarCrossref
4.
Hu  M, Cappelleri  JC, Lan  KK.  Applying the law of iterated logarithm to control type I error in cumulative meta-analysis of binary outcomes.   Clin Trials. 2007;4(4):329-340. doi:10.1177/1740774507081219PubMedGoogle ScholarCrossref
5.
Ioannidis  JPA.  The proposal to lower P value thresholds to. 005.   JAMA. 2018;319(14):1429-1430. doi:10.1001/jama.2018.1536PubMedGoogle ScholarCrossref
6.
Sterne  JA, Davey Smith  G.  Sifting the evidence-what’s wrong with significance tests?   BMJ. 2001;322(7280):226-231. doi:10.1136/bmj.322.7280.226PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 839
    Citations 0
    Research Letter
    Statistics and Research Methods
    September 4, 2020

    Evaluation of Cumulative Meta-analysis of Rare Events as a Tool for Clinical Trials Safety Monitoring

    Author Affiliations
    • 1Department of Cardiology, Bern University Hospital, University of Bern, Bern, Switzerland
    • 2Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
    • 3Applied Health Research Centre, Li Ka Shing Knowledge Institute of St Michael's Hospital, Toronto, Ontario, Canada
    • 4Department of Medicine and Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
    JAMA Netw Open. 2020;3(9):e2015031. doi:10.1001/jamanetworkopen.2020.15031
    Introduction

    The continued vigilance in safety monitoring in randomized clinical trials (RCTs) is critical as more data and experience are accumulated.1 Emerging safety profiles of therapeutic interventions during longer follow-up may cast doubt on earlier conclusions about benefit-risk assessment.1,2 Along this line, cumulative meta-analysis has been proposed as a tool to evaluate evidence aggregation. We retrospectively assessed how cumulative meta-analysis could serve as a safety monitoring tool to identify the time point when firm evidence for safety concerns of a rare outcome becomes available.

    Methods

    For this meta-analysis, we assessed the withdrawn polymeric everolimus-eluting coronary bioresorbable vascular scaffold (BVS) (Absorb; Abbott Vascular). The BVS received CE mark approval in January 2011 and US Food and Drug Administration approval in July 2016. In September 2017, the manufacturer voluntarily withdrew the device owing to safety concerns (increased risk of scaffold-related thrombosis) after it had been available for clinical use for more than 6 years in Europe and 1 year in the US. This study followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline.

    We retrieved all available reports of RCTs comparing the BVS with metallic everolimus-eluting stents for percutaneous coronary interventions by searching PubMed, CENTRAL, and websites of major cardiology meetings occurring before May 31, 2019. Device-related (scaffold or stent) definite or probable thrombosis was the safety outcome of interest. We used Mantel-Haenszel (fixed-effects model) cumulative meta-analysis to summarize accumulated rare events over time and computed odds ratios (ORs) at each time point. All P values are 2-sided, and P < .05 was considered statistically significant. Analyses were performed using R, version 3.3.2 (The R Foundation for Statistical Computing).

    Results

    A total of 22 reports describing 8 RCTs including 8180 patients randomized to BVS (4553 patients) or everolimus-eluting stents (3627 patients) were included, with 96 and 20 device-related thromboses for each intervention, respectively. Patient recruitment took place over 6 years, with considerable overlap of recruitment periods (Figure 1). The cumulative meta-analysis (Figure 2) revealed that the initial uncertainty regarding the treatment effect based on early trials with follow-up to 1 year gained precision through inclusion of additional trials and follow-up time. The analysis of accumulated evidence showed initial safety concerns after the publication of ABSORB III trial on October 12, 2015, for a clinically important but non–statistically significant increase in the risk of device-related thrombosis after use of BVS (OR, 2.22; 95% CI, 0.97-5.06, P = .06). The between-group difference became statistically significant on September 18, 2016 (OR, 2.52; 95% CI, 1.12-5.71; P = .03). Availability of longer follow-up and new trials resulted in an OR of 2.87 (95% CI, 1.34-6.16; P = .007) 11 months before the Absorb BVS was withdrawn in September 2017. Between-group differences reached on March 18, 2017, had an OR of 3.15 (95% CI, 1.48-6.72; P = .003) with a lower limit of the 95% CI above 1.00 (Figure 2). The final estimate was an OR of BVS-related thrombosis of 3.68 (95% CI, 2.25-6.00; P< .001), indicating that the experimental intervention was harmful.

    Discussion

    Timely recognition of safety signals is important to patients, physicians, regulators, and the medical community at large to avoid unnecessary, clinically important adverse events and to prevent waste of research efforts, especially in studies of the comparative effectiveness of medical devices. In the absence of large clinical trials, some adverse events may not be known a priori when a new device is used and additional mechanisms, such as regulatory oversight for unexpected events, may be needed; continuously updated cumulative meta-analyses may contribute to this purpose. Of note, although cumulative statistical testing can bias this approach, it is not of particular concern in the present analysis because it was performed retrospectively and was not associated with a stopping rule for the meta-analysis. However, in a prospectively designed cumulative meta-analysis, correction for multiple testing should be considered because the examination of multiple outcomes and repeated analysis of the data over time may exacerbate the risks associated with multiplicity and further adjustments may be warranted.3,4 Under these scenarios, false-positive rates for significance tests at the conventional P < .05 are typically too high, and naive interpretations of statistical significance should be avoided.5,6

    Back to top
    Article Information

    Accepted for Publication: June 1, 2020.

    Published: September 4, 2020. doi:10.1001/jamanetworkopen.2020.15031

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Siontis GCM et al. JAMA Network Open.

    Corresponding Author: George C. M. Siontis, MD, PhD, Department of Cardiology, Bern University Hospital, University of Bern, 3010 Bern, Switzerland (georgios.siontis@insel.ch).

    Author Contributions: Dr Siontis had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Siontis, Nikolakopoulou, Räber, Jüni.

    Acquisition, analysis, or interpretation of data: Siontis, Nikolakopoulou, Efthimiou, Windecker, Jüni.

    Drafting of the manuscript: Siontis, Efthimiou, Jüni.

    Critical revision of the manuscript for important intellectual content: Nikolakopoulou, Efthimiou, Räber, Windecker, Jüni.

    Statistical analysis: Siontis, Nikolakopoulou, Efthimiou, Jüni.

    Administrative, technical, or material support: Siontis.

    Supervision: Räber, Windecker.

    Conflict of Interest Disclosures: Dr Siontis reported receiving honoraria from Abbott outside the submitted work. Dr Efthimiou reported receiving grant support from the Swiss National Science Foundation. Dr Räber reported receiving research grants to the institution from Abbott, Biotronik, Boston Scientific, Heartflow, Sanofi, and Regeneron; receiving speaker honoraria from Abbott, AstraZeneca, Amgen, CSL Behring, and Sanofi outside the submitted work; receiving personal fees from Abbott during the conduct of the study; receiving personal fees from Amgen, AstraZeneca, Canon, Occlutech, Sanofi, and Vifor; and receiving grants from Boston Scientific, Biotronik, Heartflow, and Regeneron outside the submitted work. Dr Windecker reported receiving research and educational grants from Abbott, Amgen, Boston Scientific, Biotronik, Bayer, BMS, Cardinal Health, CSL Behring, Edwards Lifesciences, Medtronic, Polares, Sanofi, and Sinomed outside the submitted work. Dr Jüni reported receiving research grants to the institution from AstraZeneca, Biotronik, Biosensors International, Eli Lilly, and The Medicines Company; serving as an unpaid member of the steering group of trials funded by AstraZeneca, Biotronik, Biosensors, St. Jude Medical, and The Medicines Company outside the submitted work; and participating on advisory boards and/or consulting for Amgen, Ava, and Fresenius. No other disclosures were reported.

    References
    1.
    Calis  KA, Archdeacon  P, Bain  R,  et al.  Recommendations for data monitoring committees from the Clinical Trials Transformation Initiative.   Clin Trials. 2017;14(4):342-348. doi:10.1177/1740774517707743PubMedGoogle ScholarCrossref
    2.
    Chalmers  I, Bracken  MB, Djulbegovic  B,  et al.  How to increase value and reduce waste when research priorities are set.   Lancet. 2014;383(9912):156-165. doi:10.1016/S0140-6736(13)62229-1PubMedGoogle ScholarCrossref
    3.
    Bender  R, Bunce  C, Clarke  M,  et al.  Attention should be given to multiplicity issues in systematic reviews.   J Clin Epidemiol. 2008;61(9):857-865. doi:10.1016/j.jclinepi.2008.03.004PubMedGoogle ScholarCrossref
    4.
    Hu  M, Cappelleri  JC, Lan  KK.  Applying the law of iterated logarithm to control type I error in cumulative meta-analysis of binary outcomes.   Clin Trials. 2007;4(4):329-340. doi:10.1177/1740774507081219PubMedGoogle ScholarCrossref
    5.
    Ioannidis  JPA.  The proposal to lower P value thresholds to. 005.   JAMA. 2018;319(14):1429-1430. doi:10.1001/jama.2018.1536PubMedGoogle ScholarCrossref
    6.
    Sterne  JA, Davey Smith  G.  Sifting the evidence-what’s wrong with significance tests?   BMJ. 2001;322(7280):226-231. doi:10.1136/bmj.322.7280.226PubMedGoogle ScholarCrossref
    ×