Automated Surveillance to Detect Postprocedure Safety Signals of Approved Cardiovascular Devices

Context Ensuring the safety of medical devices challenges current surveillance approaches, which rely heavily on voluntary reporting of adverse events. Automated surveillance of clinical registries may provide early warnings in the postmarket evaluation of medical device safety.

Objective To determine whether automated safety surveillance of clinical registries using a computerized tool can provide early warnings regarding the safety of new cardiovascular devices.

Design, Setting, and Patients Prospective propensity-matched cohort analysis of 7 newly introduced cardiovascular devices, using clinical data captured in the Massachusetts implementation of the National Cardiovascular Data Repository CathPCI Registry for all adult patients undergoing percutaneous coronary intervention from April 2003 through September 2007 in Massachusetts.

Main Outcome Measure Presence of any safety alert, triggered if the cumulative observed risk for a given device exceeded the upper 95% confidence interval (CI) of comparator control device. Predefined sensitivity analyses assessed robustness of alerts when triggered.

Results We evaluated 74 427 consecutive interventional coronary procedures. Three of 21 safety analyses triggered sustained alerts in 2 implantable devices. Patients receiving Taxus Express2 drug-eluting stents experienced a 1.28-fold increased risk of postprocedural myocardial infarction (2.87% vs 2.25%; absolute risk increase, 0.62% [95% CI, 0.25%-0.99%]) and a 1.21-fold increased risk of major adverse cardiac events (4.24% vs 3.50%; absolute increase, 0.74% [95% CI, 0.29%-1.19%]) compared with those receiving alternative drug-eluting stents. Patients receiving Angio-Seal STS vascular closure devices experienced a 1.51-fold increased risk of major vascular complications (1.09% vs 0.72%; absolute increased risk, 0.37% [95% CI, 0.03%-0.71%]) compared with those receiving alternative vascular closure devices. Sensitivity analyses confirmed increased risk following use of the Taxus Express2 stent but not the Angio-Seal STS device.

Conclusion Automated prospective surveillance of clinical registries is feasible and can identify low-frequency safety signals for new cardiovascular devices.

Monitoring the safety of approved medical products is of vital public health importance, given that in clinical practice such medical products are often used in numbers far greater and in patient populations more diverse than when studied in premarket evaluations and clinical trials.^1-6 Within the broad range of medical products, implantable medical devices represent high-risk products that are uniquely challenging to monitor because there is little consensus regarding the most appropriate methods to account for the complex interactions among devices, medications, patients, and implanting physicians. In addition, the lack of unique medical device identifiers challenges the effective use of administrative claims data and electronic health records as a primary data source to evaluate manufacturer-specific device safety.⁷

To support clinical research and quality-improvement efforts, detailed clinical registries have been established in recent years at the state, regional, and national levels for many high-risk implantable medical devices,^8-11 which may provide unique opportunities to prospectively monitor the safety of such devices.¹² The Massachusetts statewide coronary intervention registry¹³ was established in 2003 to monitor the quality of care of hospitals and physicians in the state. This registry is a mandatory clinical outcomes registry based on the American College of Cardiology National Cardiovascular Data Repository (NCDR) CathPCI data set and includes manufacturer-specific device information for every adult patient undergoing angioplasty procedures in nonfederal hospitals since April 2003.¹⁴ This mandatory registry represents a high-quality data source for safety surveillance, because it is comprehensively audited and adjudicated for major adverse events and risk factors.^13,15

Using the Massachusetts statewide coronary intervention registry, we examined in-hospital safety signals for recently introduced interventional cardiovascular devices using an automated safety surveillance system to assess the feasibility of such an approach to prospective surveillance of medical device safety.

The Massachusetts angioplasty registry collects detailed clinical data and inpatient outcome information for all adults (18 years or older) who undergo coronary intervention at all nonfederal Massachusetts inpatient facilities. All registry records between April 1, 2003, and September 30, 2007, were included in the analysis. Detailed clinical information obtained during the hospital admission was collected prospectively by trained data managers using variables defined in the NCDR CathPCI Registry data set¹⁶ and was subject to detailed review and audit procedures at the hospital level as well as at the state level, where all major adverse events are reviewed by a panel of trained volunteer physicians and nurse data managers. The study protocol was approved by the hospital's institutional review board and the US Food and Drug Administration Research Involving Human Subjects Committee.

A computerized automated safety surveillance tool, the Data Extraction and Longitudinal Trend Analysis (DELTA) system, was developed and validated on outcomes and clinical trial databases and shown to efficiently identify very low-frequency events, using an array of Bayesian and frequentist inference methods.^17-19 The system supports multiple simultaneous device-specific analyses, tracking the accumulating experience of multiple devices while monitoring multiple independent data sets simultaneously.^12,20

Tools within the system allow for joining multiple related data sets and establishing independent prospective analyses using numerous analytic options including propensity matching, risk-adjusted cumulative outcomes analysis, sequential methods, and survival methods. The system can be configured to trigger alerts at flexible levels of deviation from expected outcomes and to signal the analyst through e-mail notification when an alert is generated. The surveillance tool was implemented at a central data repository to monitor the accumulating Massachusetts interventional cardiology registry for device-specific safety signals and to trigger safety alerts when specific statistical thresholds were achieved for any monitored device.

Four classes of high-risk interventional cardiovascular devices recently approved and introduced into clinical practice during the study period were selected for safety monitoring. These included drug-eluting coronary stents, small-vessel bare-metal coronary stents, vascular closure devices, and embolic protection devices. Potential devices were selected among all high-risk devices if they met the sample size required to achieve 80% power to detect a 50% increase in adverse event rates using a type I error rate of .05. For example, assuming an average composite adverse event rate of 2.0%, a sample size of 3826 patient exposures would be required to attain 80% power to detect a 50% increase in event rate (to 3.0%). Sample size requirements varied from 853 for vascular closure device exposures to 3536 for drug-eluting stent exposures.

Based on the evolution of the NCDR data set specification over the study period, manufacturer-specific device information was available for drug-eluting stents throughout the study period. Bare-metal stents, vascular closure devices, and embolic protection devices had similar detailed information available beginning in January 2005. The unit of inference was the procedure with patients receiving multiple studied devices included in each device-outcome analysis.

Each medical device was evaluated for acute adverse outcomes specific to the device group as selected by the investigator team, based on clinical relevance to the device class and incorporating the recommendations of collaborators at the US Food and Drug Administration Center for Devices and Radiological Health. All adverse events and clinical risk factors were defined in accordance with the NCDR CathPCI data set definitions.¹⁶

For each stent and embolic protection device, adverse events included in-hospital postprocedure myocardial infarction (MI), in-hospital death, and a composite end point of major adverse cardiac events (MACE) including emergent revascularization, death, and MI. For the vascular closure devices studied, adverse events included in-hospital minor vascular complications (including access-site bleeding, hematoma >5 cm, pseudoaneurysm, and arteriovenous fistula), major vascular complications (including retroperitoneal hemorrhage, vessel dissection or occlusion, or need for urgent vascular procedure), and any vascular complication.

For each exposure of interest, a propensity score–matched concurrent control population was developed based on published risk factors for the outcome of interest as well as on factors considered by domain experts to potentially influence the selection of one device vs another in its group (eAppendix). Propensity scores were developed from a nonparsimonious hierarchical logistic regression analysis developed with the device of interest (exposure) used as the dependent variable, adjusting for baseline covariates of the factors listed in the eAppendix, as well as between-hospital differences in device use. Initial matches were selected from the population of patients exposed to an alternative device within the same group as the exposure of interest (ie, alternative drug-eluting stents). The cohorts were matched within 6 months of device implantation date and within a fixed propensity score caliper of 0.05 using a greedy matching algorithm.²¹

The relative imbalance between the exposed and unexposed groups was assessed using the absolute standardized difference (percentile) in covariate means and proportions, with values greater than 10% considered severely imbalanced.²² The propensity matching was considered insufficient to examine overall safety profile of the device if less than 50% of total exposures of a device were successfully matched to control patients (typically because of high use of the exposure of interest). In this circumstance, there was poor balance of the clinical features of patients receiving the device of interest and alternative (control) devices. In these situations, the potential control population was expanded through use of less restrictive device exposure parameters (ie, all drug-eluting and bare-metal coronary stents).

Adverse-event rates were calculated quarterly for the propensity score–matched unexposed and exposed cohorts. Safety alerts were triggered if the confidence intervals (CIs) around the difference between 2 independent proportions (as measured by the Wilson method) did not cross zero, which indicates a statistically significant difference between the exposed and unexposed groups. The CIs were established using a 95% CI corrected for multiple comparisons through use of the O’Brien-Fleming alpha-spending method.¹² The χ² test was used for comparisons of categorical data, and the 2-tailed t test was used to compare continuous variables. All prospective surveillance statistical analyses were performed within the safety monitoring system (Coping Systems, Andover, Massachusetts).^12,18,20 Population summary statistics were calculated using STATA version 8.0 (StataCorp, College Station, Texas). All statistical tests were 2-sided, with P < .05 considered statistically significant for all comparisons.

To further investigate a potential safety signal and explore potential subpopulations affected, a series of prespecified sensitivity analyses were performed through the surveillance system if 3 or more safety signal alerts were generated for a device-outcome pair during the analysis. These sensitivity analyses included periodic analysis, subpopulation analyses, and alternative risk-modeling methods. The periodic analysis used periodic, rather than cumulative, safety signal evaluation to explore consistency of elevated rates and temporal trends in outcomes. To explore whether potential imbalance of specific risk factors between exposed and matched populations might be related to an alert, univariate comparisons of matched and unmatched populations were performed.

Relative device safety was assessed using logistic regression–based risk adjustment using historical nonexposed patients. The multiple logistic regression model was developed using backward stepwise selection to identify predictors of specified complications, and final models incorporated those covariates with consistent associations of P ≤ .20. The model was developed and calibrated using control patients in the 12 months prior to the study period for the particular device and applied prospectively to the entire cohort of patients exposed to the device of interest. This method supported inclusion of the entire cohort exposed to the device of interest, rather than only the subset with an adequate match to a concurrent control population.

Patient, physician, and hospital deidentified data for 74 427 consecutive coronary interventional procedures performed from April 1, 2003, to September 30, 2007, in nonfederal Massachusetts hospitals were evaluated. Seven devices met the sample size requirements for automated safety monitoring, including 2 drug-eluting coronary stent systems (Taxus Express2 [Boston Scientific, Natick, Massachusetts] and Cypher [Cordis, Bridgewater, New Jersey]), 1 bare-metal stent (Mini-Vision [Guidant/Abbott, Abbott Park, Illinois]), 1 embolic protection device (FilterWire [Boston Scientific]), and 3 vascular closure devices (Angio-Seal STS [St Jude Medical, St Paul, Minnesota], Perclose Proglide [Abbott Vascular, Santa Clara, California], and StarClose [Abbott Vascular]).

Table 1 summarizes the 7 devices, along with the 21 safety analyses performed and the matched concurrent control populations chosen for each analysis. The proportion of exposures successfully matched using the propensity-matching algorithm ranged from 51% for the Cypher drug-eluting stent to more than 99% for the Perclose Proglide and StarClose vascular closure devices. The FilterWire embolic protection device and the Angio-Seal STS vascular closure device required expansion of control patient populations because of high utilization rates (and therefore limited concurrent controls in same device group) of the devices of interest.

Of the 21 safety analyses performed, 3 (14%) generated a repeated or sustained safety signal involving 2 implanted devices, prompting detailed sensitivity analysis per study protocol (Table 1). The safety alerts included an increased risk of postprocedural MI as well as an increased risk of MACE following implantation of Taxus Express2 drug-eluting stents. In addition, an increased rate of major vascular complications following implantation of the Angio-Seal STS vascular closure device was observed. All other safety analyses resulted in outcomes within (or superior to) the 95% CI established by the propensity-matched control population.

Figure 1 illustrates the cumulative safety analysis for the Taxus Express2 drug-eluting stent. A total of 18 277 procedures involved implantation of 1 or more Taxus Express2 stents, of which 14 893 (81.5%) were successfully matched to control patients receiving alternative drug-eluting stents (predominantly the Cypher drug-eluting stent). Although proportions of use differed significantly among the institutions, both the Taxus Express2 and alternative drug-eluting stents were used in all 22 hospitals included in the analysis.

As shown in Figure 1, by the end of the study period (October 2007) the rate of postprocedural MI was 27.6% higher for Taxus Express2 drug-eluting stents compared with alternative drug-eluting stents (2.87% vs 2.25%; absolute risk increase, 0.62% [95% CI, 0.25%-0.99%]). The surveillance system issued the first alert in quarter 5 of the analysis and then demonstrated sustained alerts for increased risk with the Taxus Express2 drug-eluting stent beginning in July 2005. Similarly, the rates of MACE were increased by 21.1%, driven by the increased postprocedural MI difference, for the Taxus Express2 drug-eluting stent relative to the MACE rate for the propensity-matched control population (4.24% vs 3.50%; absolute increase, 0.74% [95% CI, 0.29%-1.19%]) (Figure 1), and a sustained safety alert for MACE was triggered beginning in July 2007. No increased risk of death was observed among the exposure cohorts (Figure 1).

Baseline disparities between patients receiving the Taxus Express2 stents and those receiving alternative stents were virtually eliminated for nearly all covariates after the propensity match was applied, with the standardized difference measure less than 10% for all covariates (Table 2). Significant differences remained, however, in age, exposure to glycoprotein IIb/IIIa antagonists, mean final stent diameter, and maximum lesion length (as a surrogate for stent length) (Table 2); however, the findings regarding safety of the stent were unchanged after controlling for these factors in multivariate analysis.

Predefined sensitivity analyses were automatically performed to explore potential explanations of the positive safety signals. Rolling-quarter (period-based) analysis of the Taxus Express2 drug-eluting stent demonstrated consistent postprocedural MI and MACE rates at or above the safety-alerting threshold throughout the study, thereby confirming a temporally consistent increased hazard for use of the Taxus Express2 stent. Additionally, a multiple logistic regression predictive model for the risk of postprocedure MI based on all non-Taxus drug-eluting stents used in 2003-2004 (the period immediately preceding the study period) was applied prospectively to the entire cohort of patients receiving Taxus Express2 stents. We observed no significant difference from the alerting behavior observed in the original propensity match compared with the analysis using all patients receiving the stent.

A total of 8015 case patients receiving Angio-Seal STS vascular closure devices were successfully matched from a total population of 10 801 patients receiving Angio-Seal STS devices (74.2%). Although frequency of use differed, both the Angio-Seal STS device and alternative vascular closure strategies were used at all 22 institutions included in the analysis. Patients receiving the Angio-Seal STS device were found to have a consistently higher than expected rate of major vascular complications but a lower rate of minor complications compared with alternative vascular management strategies using the propensity-matched concurrent control method (Figure 2).

By the end of the observation period, the matched subset of Angio-Seal STS case patients experienced a 51.3% increased risk of major vascular complications compared with the concurrent control population (1.09% vs 0.72%; absolute increased risk, 0.37% [95% CI, 0.03%-0.71%]). Baseline disparities between patients receiving the Angio-Seal STS devices and those receiving alternative devices were reduced for most covariates after the propensity match was applied. Some differences remained between the matched cohorts in the proportion of patients presenting with symptoms of congestive heart failure and low ejection fraction; however, the standardized difference demonstrated adequate balance of all covariates after the match (Table 3).

The prespecified sensitivity analysis exploring temporal changes in outcomes for the Angio-Seal STS vascular closure device demonstrated significant heterogeneity in the outcomes observed, with an early period of increased risk of major vascular complications (March 2005 through December 2005) followed by consecutive periods of acceptable risk (January 2006 through December 2006). A detailed exploration of differences in clinical and demographic covariates was performed and demonstrated a significant increase in the use of the direct thrombin inhibitor bivalirudin as well as a reduction in the use of the Angio-Seal STS device in patients with concomitant venous access between the early and later periods. The propensity-matched rates of major vascular complications (and overall vascular complications) for patients treated with bivalirudin demonstrated that patients receiving the Angio-Seal STS device had a 38% reduced rate of major vascular complications compared with patients treated with alternative vascular management (0.49% vs 0.79%; absolute reduction, 0.30% [95% CI, 0.14%-0.46%]).

Lastly, a second propensity analysis including only patients who received alternative implantable vascular closure devices confirmed the principal findings of these analyses. However, only 48% of patients receiving Angio-Seal STS devices could be matched to this control group because of the high overall use of Angio-Seal STS devices within the registry population.

An independent multiple logistic regression predictive model for the risk of major vascular complications based on all non–Angio-Seal STS vascular closure devices used in 2004 (the period immediately preceding the propensity-matched analysis period) was developed and applied prospectively to the entire cohort of patients receiving Angio-Seal STS devices. We observed no significant difference between the alerting behavior using the logistic regression prediction method vs the original propensity match, thereby supporting the findings of the primary propensity analysis.

This study demonstrates the feasibility of automated safety surveillance of implantable devices when applied to a clinical outcomes registry through the use of computerized adverse event surveillance. The methodologies incorporated into the surveillance system were able to distinguish low-frequency medical device safety risks not highlighted in premarket approval studies.

In this study, 14% of the monitored device-outcome pairs triggered a sustained potential safety signal necessitating detailed sensitivity analysis. We found that patients receiving the Taxus Express2 drug-eluting stent experienced a significantly higher rate of postprocedural MI compared with patients receiving alternative drug-eluting stents. This finding was sustained after temporal trend analysis and alternative risk modeling approaches, although the effects of smaller stent diameters and shorter total lesion lengths in recipients of Taxus Express2 drug-eluting stents could not be fully excluded as confounders of the observed results.

Our findings regarding the increased risk of periprocedural MI with the Taxus Express2 stent are supported by trends reported in prospective randomized clinical trials, including the Taxus V trial, in which patients receiving multiple Taxus Express2 stents experienced significantly increased risk of MI within 30 days of the procedure compared with patients randomly assigned to receive bare-metal stents (8.3% vs 3.3%, respectively; P = .047).²³ A potential mechanism for this increased risk has been proposed by Popma et al²⁴ in a detailed angiographic review of the ENDEAVOR IV trial, which demonstrated an increased frequency of side-branch compromise associated with periprocedural MI when the Taxus Express2 drug-eluting stent was used as compared with the Endeavor drug-eluting stent.

Use of the Angio-Seal STS vascular closure device was associated with a significantly increased risk of major vascular complications in the early period of experience with the device. However, this risk reversed over time in association with changes in practice in antithrombotic therapy. For the Angio-Seal STS device, we conclude that case patient selection, changes in medical therapy, and potential learning curve effects likely explain a significant proportion of the increased risk observed in the early period of use.

There are few comparative data on the safety of specific vascular closure devices, although the Angio-Seal STS device has not previously been shown to pose an increased risk of major vascular complications relative to other vascular closure devices in 2 meta-analyses^25,26 and a large comparative safety study.²⁷ A learning curve effect in the use of the Angio-Seal device has been described,²⁸ as has the significant reduction in major vascular complications through the use of bivalirudin during percutaneous coronary intervention procedures that was observed during the period of safety surveillance in this analysis.^6,29,30

There was no evidence for increased risks of the analyzed outcomes in the use of the other cardiovascular devices studied, including the Cypher drug-eluting stent, the Mini-Vision bare-metal stent, the FilterWire embolic protection device, and the StarClose or Perclose Proglide vascular closure devices.

This study demonstrates the feasibility of automated surveillance of clinical device registries and provides a potential framework for temporally efficient comparative safety analysis over broad populations of real-world patients. Prospective computerized monitoring, such as that demonstrated here, can support the simultaneous monitoring of many device-outcome pairs, thereby permitting the efficient use of valuable human resources to explore specific risks identified through the automated safety screening algorithms and alerts.

While the number of data sources having features similar to those of the Massachusetts angioplasty registry are limited, such detailed clinical registries are becoming more widespread. In addition, alternative clinical data repositories, such as pooled data from increasingly available electronic health record systems as well as medical condition–specific clinical outcomes registries, may prove to be valuable resources for additional exploration of automated safety surveillance approaches. Such automated prospective surveillance of medical device safety can aid public health officials who rely on passive surveillance tools that lack denominator data (ie, comprehensive exposure information) and therefore provide accurate comparative assessments of safety risk. In addition, federally mandated postapproval studies are often of limited scope and duration, have limited control populations, and often lack statistical power to detect very low-frequency safety signals.^31-33 Automated safety surveillance may also complement plans for the recently announced Sentinel Initiative,³⁴ an active surveillance program being implemented by the US Food and Drug Administration to use existing electronic health care information sources to efficiently generate, strengthen, or confirm safety signals for medical products.⁷ In this context, registries will achieve optimal utility when linked with longitudinal data sources. This is particularly true for implantable devices, for which outcomes of interest often extend beyond the hospital stay.

It is important to note that potential signals generated in automated surveillance systems must be interpreted with caution and that system safety alerts are intended to generate hypotheses for more in-depth exploration. All potential signals identified through such methods require further evaluation, including sensitivity analyses and more formal epidemiologic studies (which may include medical record validation of outcomes as appropriate). Also, while simple alert boundaries based on statistically significant increased risk were used in this analysis, alternative alert boundary conditions potentially incorporating the severity of the adverse outcome being studied may help inform the choice of the magnitude of the signal to be identified as well as the threshold for significance that would merit additional exploration.

Several additional limitations of the present analysis may affect the generalizability of the results. The Massachusetts angioplasty registry is an audited and adjudicated data set that provides a high-quality data source, but it may not be representative of other postmarket device clinical registries. In an effort to reduce bias in our estimates, our case-matching strategy led, in some instances, to the exclusion of unique, yet high-risk, subsets of exposed patients. In this case, because matching controls could not be identified, we were unable to make any comparative statements about safety in these patient populations. Also, potential for residual confounding (from known and unknown factors, including those influencing patient selection) may remain, despite the propensity-based adjustment methods used.

In conclusion, automated safety surveillance of medical devices is feasible using automated monitoring tools applied to detailed clinical registries and can efficiently help identify emerging potential postmarket safety risks. Automated medical product surveillance can complement existing public health strategies, providing an additional mechanism to assess the comparative safety of approved medical products and improve the quality of health care delivered.

Corresponding Author: Frederic S. Resnic, MD, MSc, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115 (fresnic@partners.org).

Author Contributions: Dr Resnic had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Resnic, Gross, Donnelly, Normand, Matheny.

Acquisition of data: Resnic, Donnelly, Matheny.

Analysis and interpretation of data: Resnic, Gross, Marinac-Dabic, Loyo-Berrios, Donnelly, Normand, Matheny.

Drafting of the manuscript: Resnic, Normand.

Critical revision of the manuscript for important intellectual content: Resnic, Gross, Marinac-Dabic, Loyo-Berrios, Donnelly, Matheny.

Statistical analysis: Resnic, Normand, Matheny.

Obtained funding: Resnic, Gross, Marinac-Dabic.

Administrative, technical, or material support: Resnic, Gross, Donnelly, Normand.

Study supervision: Resnic, Gross, Marinac-Dabic, Donnelly.

Funding/Support: This study was funded, in part, by grants from the National Library of Medicine (NIH R01-LM008142) and the US Food and Drug Administration (HHSF 223200830058C) as well as from the Veterans Administration Health Services Research and Development Service (CDP 09-387).

Role of the Sponsors: The National Institutes of Health, the Veterans Administration Health Services Research and Development Service, and the US Food and Drug Administration had no role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; or the preparation, review, or approval of the manuscript. However, the preliminary proposed study protocol was provided to the Veterans Administration Health Services Research and Development Service and the US Food and Drug Administration as part of the original grant and contract application process.

Disclaimer: The opinions and assertions presented in this article are the private views of the authors and are not to be construed as conveying either official endorsement or criticism by the US Department of Health and Human Services, the Public Health Service, or the US Food and Drug Administration.

Additional Contributions: We acknowledge the significant system development and technical contributions of Richard Cope, BS, and Susan Robbins, BS, of Coping Systems Inc, Andover, Massachusetts. Coping Systems was a subcontractor to Brigham and Women's Hospital, and neither Mr Cope nor Ms Robbins received extra compensation for their contributions; Mr Cope is part owner of Coping Systems and is also named on the patent application filed by Dr Resnic and Dr Matheny. We also acknowledge the support of the Division of Healthcare Quality and Statistics, Massachusetts Department of Public Health.