Characteristics of Clinical Studies Used for US Food and Drug Administration Approval of High-Risk Medical Device Supplements

Key Points

Question  What is the quality of clinical studies and data used to approve modifications to high-risk devices by the US Food and Drug Administration (FDA) panel-track supplement pathway?

Findings  In this descriptive study of 83 clinical studies for 78 panel-track supplements approved between 2006 and 2015, 45% were randomized clinical trials and 30% were blinded. Of the 150 primary end points in these studies, 81% were surrogates and 38% were compared with controls.

Meaning  There are limitations in the quality of the studies and data evaluated by the FDA to support modifications of high-risk devices.

Importance  High-risk medical devices often undergo modifications, which are approved by the US Food and Drug Administration (FDA) through various kinds of premarket approval (PMA) supplements. There have been multiple high-profile recalls of devices approved as PMA supplements.

Objective  To characterize the quality of the clinical studies and data (strength of evidence) used to support FDA approval of panel-track supplements (a type of PMA supplement pathway that is used for significant changes in a device or indication for use and always requires clinical data).

Design and Setting  Descriptive study of clinical studies supporting panel-track supplements approved by the FDA between April 19, 2006, and October 9, 2015.

Exposure  Panel-track supplement approval.

Main Outcomes and Measures  Methodological quality of studies including randomization, blinding, type of controls, clinical vs surrogate primary end points, use of post hoc analyses, and reporting of age and sex.

Results  Eighty-three clinical studies supported the approval of 78 panel-track supplements, with 71 panel-track supplements (91%) supported by a single study. Of the 83 studies, 37 (45%) were randomized clinical trials and 25 (30%) were blinded. The median number of patients per study was 185 (interquartile range, 75-305), and the median follow-up duration was 180 days (interquartile range, 84-270 days). There were a total of 150 primary end points (mean [SD], 1.8 [1.2] per study), and 57 primary end points (38%) were compared with controls. Of primary end points with controls, 6 (11%) were retrospective controls and 51 (89%) were active controls. One hundred twenty-one primary end points (81%) were surrogate end points. Thirty-three studies (40%) did not report age and 25 (30%) did not report sex for all enrolled patients. The FDA required postapproval studies for 29 of 78 (37%) panel-track supplements.

Conclusions and Relevance  Among clinical studies used to support FDA approval of high-risk medical device modifications, fewer than half were randomized, blinded, or controlled, and most primary outcomes were based on surrogate end points. These findings suggest that the quality of studies and data evaluated to support approval by the FDA of modifications of high-risk devices should be improved.

Quiz Ref IDHigh-risk medical devices in the United States are regulated by the US Food and Drug Administration (FDA) to ensure safety and effectiveness. These devices, defined as those that “support or sustain human life, are of substantial importance in preventing impairment of human health, or which present a potential, unreasonable risk of illness or injury,”¹ are evaluated via premarket approval (PMA), the most rigorous FDA device approval pathway. Examples include coronary stents, hip prostheses, and cosmetic facial injectable implants. Only 1% of devices enter the market as original PMAs.² The majority of clinical studies supporting approval of original PMA devices are nonrandomized, are unblinded, and often use surrogate end points that are not compared with active controls.^3-6

Quiz Ref IDThe development pathway for devices differs from drugs in that devices may undergo many iterative modifications after entering the market. PMA device manufacturers must submit supplementary applications for any change affecting safety and effectiveness. Devices often have hundreds of supplements. For example, from 1979 through 2012, there were more than 5800 supplements for 77 original PMAs for cardiac implantable electronic devices.⁷ These supplementary changes mean many high-risk devices differ substantially from the originally approved device.⁸ Furthermore, the total number of supplements has been increasing.^7,9,10

Quiz Ref IDBy statute, the FDA must require the “least burdensome” supporting evidence for device approval.¹¹ The 6 different PMA supplement pathways¹² are less rigorous than the original PMA pathway and require variable types of evidence necessary to support safety and effectiveness.⁸ Only panel-track supplements (1 of the 6 different supplement pathways), which are used for a “significant change in design or performance of the device, or a new indication for use of the device,”¹³ always require clinical data. Given increasing use of PMA supplements,^7,9,10 this study characterized the strength of evidence of clinical studies used in FDA-approved panel-track supplements during the past decade.

On October 9, 2015, the FDA’s PMA database¹⁴ was searched using the term “Supplement Type: Panel Track” and all panel-track supplements approved between April 19, 2006, and October 9, 2015, were identified. These data were used to perform a descriptive study of all devices either implanted or tested in humans. Devices used for in vitro testing of laboratory samples were excluded.

Data were abstracted from clinical studies included in each device’s Summary of Safety and Effectiveness Data (hereafter referred to as Summary), which is “intended to present a reasoned, objective, and balanced critique of the scientific evidence which served as the basis of the decision to approve or deny the PMA.”¹⁵ The FDA determines safety and effectiveness of a medical device by considering, among other relevant factors, “(1) The persons for whose use the device is represented or intended; (2) The conditions of use for the device, including conditions of use prescribed, recommended, or suggested in the labeling or advertising of the device, and other intended conditions of use; (3) The probable benefit to health from the use of the device weighed against any probable injury or illness from such use; (4) The reliability of the device.”¹⁶

The following data were recorded for each panel-track supplement: device trade name, applicant name, device category (for example, ophthalmic or cardiovascular), reason for supplement (labeling change or changes to device design, components, or specifications), date received by the FDA, and FDA decision date. Data were classified by one of us (S.Y.Z.) and reviewed by one or two of us (S.S.D. and R.F.R.) in cases of uncertainty, which were resolved by consensus.

Data abstracted from the clinical studies evaluated for approval of the device were number of patients enrolled, mean age (with standard deviation), sex, and race. Number of patients enrolled was considered stated in the studies only if the Summary explicitly identified participants as “enrolled.” For both age and sex analyses, any discrepancies between the number of participants enrolled in the study and the number reported for mean age and sex proportion were noted.

Strength of evidence and risk of bias and confounding were determined based on study design and characterization of primary end point(s) (PEP[s]). For studies, the use of randomization and the use of blinding were characterized. The number and location of study sites were also characterized because single-site studies generally have more limitations than multisite studies, including lack of external validity.¹⁷ The PEPs were characterized by single component vs composite, type of controls, follow-up duration, and type (clinical or surrogate, with surrogate defined as “a laboratory measurement or a physical sign used as a substitute for a clinically meaningful endpoint that measures directly how a patient feels, functions or survives”¹⁸). Examples of surrogate outcomes include echocardiographic parameters such as Doppler velocity index for a heart valve, ischemia-driven target vessel revascularization for a coronary stent (because asymptomatic patients may be labeled as ischemic based on stress testing), and blinded evaluation of lip fullness for an injectable gel (because this is not a patient’s own evaluation of how he or she feels). The type of analysis also was characterized (superiority, equivalence, noninferiority, or objective performance criteria); although these designs depend on context of their use, superiority usually requires a higher evidentiary bar than the other types of analyses. An FDA Guidance Document states that a positive superiority trial is interpretable without further assumptions, while a noninferiority trial is dependent on knowing that the active control had its expected effect; if the active control did not have an effect, then showing noninferiority “provides no evidence that the test drug is effective.”¹⁹

PEPs were identified only if the Summary specifically referred to end points, objectives, outcomes, parameters, measures or measurements, criteria, variables, or assessments as “primary.” Examples of PEPs include all-cause mortality at 12 months, progression-free survival, and mean (logMAR [logarithm of the minimum angle of resolution] chart) distance-corrected near visual acuity under photopic conditions at 40 cm. When a study did not explicitly refer to any end point as primary, up to 3 end points were designated as PEPs. In such cases, the first 3 end points mentioned in the Summary were designated as PEPs. Any discrepancies between the number of patients enrolled and number examined for PEP analyses were quantified.

The presence of a post hoc analysis was recorded if analyses were conducted after any data were examined or if the Summary stated there were post hoc analyses; “not prespecified” was used for analyses that were not in the original protocol but were added to the study before data examination. Studies were characterized as not prespecified if any part of the study design—enrollment, protocol, PEP designation, or statistical analysis—was modified after study initiation, if an application was approved despite PEPs not being met, or if it was not stated whether changes occurred before data examination. If sex bias was addressed, such as through sex-specific analyses, as stated in a 1994 FDA Directive, this was recorded.²⁰ This Directive states each FDA Summary should address the following: “Was the selection ratio of men versus women in the study reflective of the underlying distribution of the disease for that given age group, ethnic group, stage of disease, etc.? Was any selection bias on the basis of gender identified during review? Was there any difference in the safety and effectiveness of the device based on gender? For example, was the device more/less effective in women?”²⁰

Additional recorded information included whether a panel-track supplement was reviewed by an FDA advisory panel and whether the FDA mandated any postapproval studies (PASs). The FDA advisory panels consist of experts who provide FDA guidance on specific questions, generally related to medical device safety and effectiveness and generally meaning that a more in-depth examination was warranted of a complex or controversial issue. PASs provide important data about the safety and effectiveness of devices in real-world clinical practice and can provide larger sample sizes and longer duration of follow-up.

Data were summarized across PMA supplements, studies, and PEPs. These summary data are presented as number (PMAs, studies, or PEPs) and as a percentage of the category to which they belong. Mean (standard deviation) and median (interquartile range [IQR]) were calculated and reported as appropriate. The statistical software used was Microsoft Excel version 14.0.0 (Microsoft Corp).

Eighty-four panel-track supplements were approved between April 19, 2006, and October 9, 2015 (Figure and eTable in the Supplement). Six supplements not involving human studies were excluded because they involved in vitro devices used to test laboratory samples. Forty-one of the 78 supplements (53%) were categorized as cardiovascular, 15 (19%) as ophthalmic, 9 (12%) as general and plastic surgery, 6 (8%) as clinical chemistry, 7 (9%) as others (orthopedic, neurology, anesthesiology, radiology, gastroenterology/urology). Sixty-two (79%) were submitted for a labeling change, such as modification of a device’s indications; 14 (18%) were submitted to support changes to device design, components, or specifications; and 2 (3%) did not state the reason for submission (Table 1). The mean (SD) number of studies supporting each supplement was 1.1 (0.5) and the mean (SD) time between submission and FDA approval was 355 (327) days (range, 104-2376 days). Of the 78 panel-track supplements, 12 (15%) underwent FDA advisory panel review. The FDA required PASs for 29 (37%).

Eighty-three clinical studies supported the 78 panel-track supplements. Five studies supported multiple supplement approvals. Most supplements (71 of 78 [91%]) were supported by a single study.

Of the 83 studies, 72 (87%) reported the number of patients enrolled (Table 2); the median number of enrolled patients was 185 (IQR, 75-305). Enrollment by age was reported for 70 studies (84%), sex for 77 (93%), and race for 49 (59%). The mean (SD) age was 57 (10) years, 51% of the study participants were male, and 82% were white. A total of 33 studies (40%) did not report age for all enrolled patients: 13 did not report any age data and 20 had incomplete age reporting. When both number enrolled and number for age were reported, a median of 28 enrolled patients (11%) per study did not have age reported as a characteristic. Similarly, 25 studies (30%) did not report sex for all enrolled patients: 6 did not report any sex data and 19 had incomplete sex reporting. When both number enrolled and number for sex were reported, a median of 27 enrolled patients (11%) per study were excluded from reporting patient sex as a characteristic.

Of the 83 studies, 37 (45%) were randomized clinical studies and 25 (30%) were blinded (16 [19%] single blinded and 9 [11%] double blinded) (Table 2). There was variation by device type: all 12 general and plastic surgery supplement studies were randomized clinical studies, compared with 16 of 41 cardiovascular studies (39%). Similarly, all 12 studies for general and plastic surgery supplements were either single or double blinded, compared with 6 of 41 studies (15%) for cardiovascular supplements.

The number of enrollment sites was not specified for 9 of 83 studies (11%) (Table 2). Of the 74 studies that specified the number of sites, the median number of sites per study was 15 (IQR, 7-24). Seventy-three studies were multicenter, of which 1 (1%) reported the number of participants enrolled at each site. Site location was reported for 59 of 83 studies (71%); of these, 29 studies (49%) were conducted solely in the United States, whereas 6 (10%) did not have any United States sites.

Comments on study results by patient sex, including additional sex subgroup analyses, were available for 40 studies (48%). Of the 83 studies, 9 (11%) used a post hoc analysis and 11 (13%) used a not-prespecified analysis, such as changing the study population by adding additional study participants or by not meeting the stated PEP (5 studies) and using another end point.

Of the 83 studies, 7 (8%) did not state any PEPs. After designating PEPs for these studies, a total of 150 PEPs were identified, with a mean (SD) of 1.8 (1.2) PEPs per study (range, 1-7) (Table 3).

The type of PEP analysis was explicitly stated for 86 of the 150 PEPs (57%) (Table 3). Of these, 22 (26%) were superiority, 2 (2%) were equivalence, 12 (14%) were noninferiority, and 50 (58%) involved analysis against an objective performance criterion.

Of the 150 PEPs, 57 (38%) were compared with controls; of these 57 PEPs with controls, 51 (89%) had active controls and 6 (11%) had retrospective controls (Table 3). Forty-five PEPs (30%) were composites and 121 (81%) were surrogates. Both the number of participants enrolled and the number included in the analysis were reported for 97 PEPs (65%). For 64 of these 97 PEPs (66%), more patients were enrolled than included in the analysis. A median of 11% (IQR, 4%-40%) of enrolled participants were excluded from their PEP analysis, indicating incomplete follow-up. For one device, 91% of enrolled patients were not included in PEP analysis.²¹ Follow-up duration at PEP analysis was stated for 143 PEPs (95%), with a median follow-up of 180 days (IQR, 84-270 days).

Quiz Ref IDIn this descriptive study of panel-track supplements for high-risk devices approved during the past decade, the majority relied on a single nonrandomized, unblinded study that lacked controls. Panel-track supplements are the most rigorous supplement type and the only type that always requires clinical data for device modifications. Although randomization and blinding are widely accepted as prerequisites for high-quality clinical studies,²² they were used infrequently in studies to support device modifications. This means lower-quality data often supported changes in high-risk devices that are modifications to previously approved devices. For example, the LAP-BAND’s indications were expanded through a panel-track PMA supplement relying on a single-group study (N = 160) without active controls (each study participant served as his or her own control) with a PEP measured at 1 year²³; this expanded indication made an estimated 19 million more Americans able to have the gastric band placed.²⁴ However, recent data indicate important concerns about the safety and effectiveness of the LAP-BAND; 18.5% of Medicare beneficiaries who have received this device have undergone reoperation by 4.5 years, with an average of 3.8 procedures per patient.²⁴ Studies without randomization are prone to various types of bias, making it difficult to ascertain whether these modified devices are safer or more effective than previous iterations, conventional treatments, or no procedure.^5,25

Quiz Ref IDFurthermore, most studies used surrogate end points, such as percent diameter stenosis determined by quantitative coronary angiography for a coronary stent or percentage of glucose values being within 20% of a reference for a glucose monitoring system. The coding of an outcome as surrogate or clinical is not always straightforward, and while we sought to be consistent with the cited definition, we recognize that some may view a few of the outcomes differently. Surrogate end points allow for clinical trials of smaller sample size and shorter duration, so trials are less costly. For surrogate end points to be useful to patients and clinicians, they must be shown to predict meaningful clinical outcomes, which rarely happens.^18,26 Therefore, use of surrogate measures can lead to uncertainty about clinical outcomes. In addition, 30% of PEPs were composites, which are often weighted disproportionately by 1 component, usually the weakest or most subjective.²⁷ Clinically important events such as death contribute less to the composite end point than more commonly occurring but less clinically significant events. One example of a study that used a composite end point was for approval of a drug-eluting coronary stent. The PEP was target-lesion failure at 12 months following the procedure, defined as cardiac death, target-vessel myocardial infarction (Q wave and non–Q wave), or clinically driven target-lesion revascularization by percutaneous or surgical methods.²⁸ A more clinically significant outcome would have been just death and myocardial infarction.

Sixty-four of 97 PEP analyses (66%) did not include all patients enrolled in the study. Such incomplete reporting may bias study results because patients with less favorable outcomes may be preferentially lost. Additionally, the common use of post hoc or not-prespecified analyses (24% of studies) may introduce bias. The reasons for study modifications should be transparent, which was often not the case. Another finding was that 33 studies (40%) did not report age and 25 (30%) did not report sex for all enrolled patients; these are usually essential data that affect the risk-benefit profile for devices and help ascertain the representativeness of study participants to the intended target population for the medical device and generalizability of study findings.

For one panel-track supplement radiology device, Selenia Dimensions 3D System, 91% of the enrolled patients were not included in the primary analysis. This device was used to generate digital mammographic images for screening and diagnosis of breast cancer. The study excluded the majority of enrolled participants for various reasons such as training purposes, participants not meeting inclusion criteria or meeting exclusion criteria, equipment failure, participants’ withdrawal of consent, imaging obtained using incorrect technique, and quality control issues. According to the Summary, following an FDA advisory panel meeting, the FDA asked the manufacturer to provide additional data to address concerns regarding excluded participants. The FDA concluded, “The additional information supported that the study exclusions were made to accommodate the study design. The technical description of the device description was sufficient and did not raise concerns about imaging the excluded subjects. In addition, images of the types of subjects that were excluded were reviewed and considered to be of acceptable image quality for clinical use.”²¹

This study found a similar strength of evidence to a previous study of original cardiovascular PMAs, suggesting opportunities for increasing the quality of clinical data for both original and supplemental PMAs.⁴ But the “least burdensome” requirement for data necessary for “a reasonable likelihood of resulting in approval”¹¹ means nearly all PMA supplements are approved without clinical data; the recently passed 21st Century Cures Act strengthens the “least burdensome” requirements.²⁹ Panel-track supplements are rarely used. For example, only 1 of 528 PMA supplements (0.2%) for high-risk otolaryngologic devices was approved as a panel-track supplement, and 15 of more than 5800 supplements (0.3%) approved for cardiac implantable devices used the panel-track supplement pathway.^7,10

The recalls of multiple devices approved through PMA supplements without clinical data—including implantable cardioverter-defibrillator leads, knee implants, and cochlear implants, among others—demonstrate that device modifications without clinical data could contribute to patients receiving devices for which safety has not been established.^30,31 For example, a glucose-monitoring system, the Dexcom G4 PLATINUM (Pediatric) Receiver, was approved as a panel-track supplement in May 2015 based on a study with 7 days’ follow-up. In February 2016, a Class I recall was initiated for more than 19 500 of these devices because patients may not receive an intended audible alert or alarm for hypoglycemia or hyperglycemia.³² A Class I recall is defined by the FDA as “a situation in which there is a reasonable probability that the use of or exposure to a violative product will cause serious adverse health consequences or death.”³³

Additionally, the implantable cardioverter-defibrillator leads of the Medtronic Sprint Fidelis were recalled in 2007 and St Jude Riata and Riata ST in 2011. These recalled leads had undergone multiple modifications (the Medtronic Sprint Fidelis was approved as a 180-day supplement and the St Jude Riata as a real-time supplement); none of the changes were supported by clinical data.⁷ These devices were implanted in hundreds of thousands of patients worldwide and were associated with at least 22 reports of deaths in the case of Riata and Riata ST and at least a dozen deaths and more than 2200 reports of serious injuries related to Sprint Fidelis.^34,35 Additionally, in 2015, there were recalls because of high revision rates of the New Jersey LCS Total Knee System, which received multiple supplemental approvals including 2 panel-track supplements.⁹ These recalls raise concern that safety signals were missed due to lack of adequate or any premarket clinical studies.

The Riegel v Medtronic, Inc Supreme Court ruling established that PMA approval, including supplements, preempts patient lawsuits related to device safety and effectiveness.³⁶ This means that patients lack legal recourse if a PMA device is faulty and adversely affects health outcomes. Thus, it is vital to ensure safety and effectiveness by requiring high-quality clinical data before high-risk devices reach the market. The findings that few supplementary changes require clinical data and that when they do the data are often low quality raise uncertainty about performance of many commonly used devices.

Given the extensive modification of many PMA supplement devices and the median preapproval follow-up of 6 months, obtaining additional data via PASs is critical. However, the FDA required PASs for the minority (37%) of panel-track supplements. Currently, PASs are often small, nonrandomized, unblinded studies without controls^3,37—similar to the quality of preapproval studies for PMA panel-track supplements. Only 13% of initiated PASs are completed between 3 and 5 years after FDA approval,³ and the FDA has never issued a warning letter, penalty, or fine against the manufacturer for noncompliance.³⁷ Active postmarket surveillance in a National Evaluation System for Health Technology, including adoption of the FDA’s unique device identification system and mandatory device registries, will facilitate postmarket data collection when the system is implemented.^38-41 To further help physicians and patients make an informed decision about which type of device to use, each device label should include easily accessible information on all relevant supplements.

This study has several limitations. First, the FDA Summaries may have missing data that are included in the proprietary applications to the FDA. However, the Summaries contain data justifying the FDA’s rationale for approval.¹⁵ Standardized reporting requirements of clinical study data by manufacturers and standardized FDA reviewer templates could ensure that studies for the highest-risk devices meet sufficiently rigorous standards. Second, data collection was done by a single coder. However, all cases of uncertainty were reviewed by at least 1 additional author. Third, because the focus of this study was premarket clinical data, the analyses did not include preclinical data supporting panel-track supplements or postmarket studies initiated without FDA requirements. Both could help inform patients and clinicians about device performance.

Among clinical studies used to support FDA approval of high-risk medical device modifications, fewer than half were randomized, blinded, or controlled, and most primary outcomes were based on surrogate end points. These findings suggest that the quality of the studies and data evaluated to support approval by the FDA of modifications of high-risk devices should be improved.

Corresponding Author: Rita F. Redberg, MD, MSc, Division of Cardiology, University of California, San Francisco, 505 Parnassus Ave, Ste M-1180, San Francisco, CA 94143-0124 (rita.redberg@ucsf.edu).

Accepted for Publication: July 11, 2017.

Author Contributions: Drs Zheng and Redberg had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Zheng and Dhruva are co–first authors.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: Zheng, Dhruva.

Drafting of the manuscript: Zheng, Dhruva.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Zheng.

Supervision: Redberg.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: Dr Dhruva is supported by the Robert Wood Johnson Foundation Clinical Scholars Program and the US Department of Veterans Affairs.

Role of the Funder/Sponsor: The funding agencies had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: Dr Redberg is Editor of JAMA Internal Medicine, but she was not involved in any of the decisions regarding review of the manuscript or its acceptance.

Additional Contributions: William Vodra, JD (Retired Partner, Arnold & Porter, LLP), provided input on an earlier version of the manuscript and Ariel Peleg, MD (Montefiore Medical Center), and Ari Gartenberg, MD (Children’s Hospital of Philadelphia), helped with the initial search of the FDA database; they received no compensation for their roles.