Chen CE, Dhruva SS, Redberg RF. Inclusion of Comparative Effectiveness Data in High-Risk Cardiovascular Device Studies at the Time of Premarket Approval. JAMA. 2012;308(17):1740-1742. doi:10.1001/jama.2012.14491
Letters Section Editor: Jody W. Zylke, MD, Senior Editor.
Author Affiliations: Department of Medicine, Stanford Hospital and Clinics, Palo Alto, California (Dr Chen); Division of Cardiovascular Medicine, University of California, Davis (Dr Dhruva); and Division of Cardiology, University of California, San Francisco (Dr Redberg; email@example.com).
To the Editor: Use of a comparator group is not required in studies used for approval of medical devices, in contrast to drugs, which require 2 randomized controlled trials. The US Food and Drug Administration (FDA) defines valid scientific evidence as “partially controlled studies, studies and objective trials without matched controls, well-documented case histories conducted by qualified experts, and reports of significant human experience.”1 A study of new drugs in the United States found that 51% were not supported by comparative effectiveness data at the time of FDA approval.2 Unlike drugs, device trials frequently use control data from previous trials (historical controls) and performance benchmarks imputed from prior studies (objective performance criteria; OPC) in evaluating outcomes. However, the lack of active controls raises concerns about selection bias and validity.
High-risk cardiovascular devices requiring premarket approval are defined as those that support or sustain life and include many invasive or implantable devices. Minor variations in manufacture do not require approval. In a review of such devices approved from 2000-2007, 52% of primary end points had controls, 31% of which were retrospective.3 In this study, we examined the type of controls used by the FDA in evaluating comparative effectiveness during premarket approval of cardiovascular devices.
Data were extracted from publicly available FDA summaries of safety and effectiveness data (herein called summaries).4 Summaries representing all high-risk cardiovascular devices approved by the FDA between January 1, 2000, and December 31, 2011, were analyzed. The selection and characteristics of premarket applications have been previously described.2
All primary end points were examined to determine the type of controls with which the device was compared (Box).3 We defined comparative effectiveness research as use of active controls in evaluating primary end points.1 We described control group characteristics by approval year and device class and examined whether number needed to treat was calculated. The Kruskal-Wallis test was used to examine differences between device classes. Statistical significance was determined at P < .05 (2-sided). Statistical analyses were performed using Stata version 10 (StataCorp).
The control group comprised individuals recruited for current study and received a validated intervention that was not the device being studied (ie, comparing a drug-eluting stent with a bare metal stent). Use of a historical control group was excluded.
For first-in-class devices, the active control group could be standard surgical intervention (carotid endarterectomy vs carotid stenting) or maximum medical management (catheter ablation vs flecainide for recurrent paroxysmal atrial fibrillation).
Few trials relied on placebo groups such as sham intravascular brachytherapy or cardiac resynchronization device with biventricular pacing functionality turned off. Active control trials could be blinded or unblinded.
The data for the control group was generated from previously conducted studies.
The significance of the end point of interest had to be evaluated against the historical data using a case-control or other statistically matched method.
Objective performance criteria or performance goal
A study that relied on a numerical benchmark derived from previously published studies of comparable devices or from expert opinion (such as previous Food and Drug Administration consensus statements).
No control group was recruited for the current study and also used none of the performance goals described above.
As an example, 2 single-group studies of a new porcine aortic valve provided data on postoperative outcomes (heart failure and hemodynamic evaluation) without comparison with other available valves. Twenty-three valve-related deaths were noted among the 337 treatment patients without comparison with a control.
The 121 summaries identified contained 203 supporting clinical studies. These studies used 353 primary end points, of which 40% (n = 140) were evaluated against an active control, 13% (n = 45) against a historical control, 26% (n = 90) against an OPC, and 22% (n = 78) against no control. Thirty-six percent (74/203) of studies included at least 1 primary end point evaluated against an active control; 48% (58/121) of devices had an active control in at least 1 supporting study, while 35% (42/121) of devices were approved without data from any control or OPC.
The use of active controls varied by device category (Table). The number of devices approved each year was too small to allow statistical analysis, but there did not appear to be any trends over time. Of the 90 end points defined using OPC, 28% (n = 25) relied on performance goals established in previous FDA guidance documents, 26% (n = 23) on previous trial data, and 19% (n = 17) on reviews of previously published literature. Twenty-one percent (25/121) of studies did not provide an OPC source.
Only 3 end points were identified in which an active control may not have been possible. No summary commented on the number needed to treat for any primary end point.
Most high-risk cardiovascular devices were approved without comparative effectiveness data. A minority of end points, studies, and summaries since 2000 included an active comparator group. Use of active controls varied significantly by device class.
We categorized the control group used in studies conducted for premarket approval of high-risk cardiovascular devices, but we cannot comment on internal validity or rigor of the studies included or on other key aspects of the trial designs.
It is hard to know if a new device is safe and more effective than alternative treatments unless it is compared with conventional treatment. While occasionally use of active controls may not be possible, such as ventricular assist devices, more frequent use of a comparator group may help to better inform clinical and regulatory decisions.
Author Contributions: Dr Redberg had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Chen, Dhruva, Redberg.
Acquisition of data: Chen, Dhruva, Redberg.
Analysis and interpretation of data: Chen, Dhruva, Redberg.
Drafting of the manuscript: Chen, Dhruva.
Critical revision of the manuscript for important intellectual content: Chen, Dhruva, Redberg.
Statistical analysis: Chen.
Study supervision: Redberg.
Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Redberg reported being a member of the US Food and Drug Administration Circulatory System Devices Panel and a member of the California Technology Assessment Forum. Dr Redberg is the editor of the Archives of Internal Medicine. No other conflicts of interest were reported.
Additional Contributions: We thank Jean Slutsky, PA, MSPH (Center for Outcomes and Evidence at Agency for Healthcare Research and Quality), and Vanessa Jacoby, MD, MAS (University of California, San Francisco, Department of Obstetrics, Gynecology, and Reproductive Sciences), for their comments and contributions. Neither received compensation for their contributions.