Assessing the Safety and Effectiveness of Devices After US Food and Drug Administration Approval: FDA-Mandated Postapproval Studies

Importance  Postmarketing surveillance is critical to evaluating the safety and effectiveness of medical devices. The US Food and Drug Administration (FDA) may order the manufacturer of a high-risk device to conduct postmarketing surveillance studies, known as postapproval studies (PASs), at the time of approval.

Objectives  To understand the characteristics of PASs ordered in recent years and inform discussions about the direction of the PASs program.

Design  Descriptive study of the PASs ordered for medical devices using the FDA’s PASs website, the Premarket Approval database, and supplemental information provided by the FDA.

Main Outcomes and Measures  The proportion of medical devices that received a PAS order and study characteristics.

Results  Between January 1, 2005, and December 31, 2011, the FDA ordered 223 studies of 158 medical devices, including studies for 93 (48%) new high-risk devices approved during this period. The median required sample size for a study was 350 patients (interquartile range, 160-1500). If the protocol of a study was not in place at the time the device was approved, which occurred frequently, a median of 180 days elapsed until the protocol was agreed on. The FDA has never issued a warning letter or penalty owing to study delays, inadequate progress, or any other issue related to a PAS. Of the approved protocols, 41 (19%) were subsequently revised, including 29 (21%) protocols in place by application approval. Some studies generated significant clinical findings. The most common effect of a PAS finding after study completion was that the FDA requested a labeling change for 31 studies (53%).

Conclusions and Relevance  Postapproval studies have the potential to provide additional information to better understand medical device performance. However, small sample sizes, delays in reaching protocol agreement, and lack of availability of findings may hinder their ability to be clinically useful. Owing to the lack of information on the effect of studies, it is unclear whether the program achieves its aims. Improved completion and accessibility of PASs could help answer important questions of safety and effectiveness about medical devices. To better understand the real-world performance of these products, they should be better integrated with other sources of information about device performance.

Postmarketing surveillance is critical to evaluating the safety and effectiveness of medical devices, which typically enter the market with less clinical data than pharmaceuticals.^1,2 Accordingly, postmarketing assessment has tremendous potential for contributing crucial information about device performance in the real world. For example, in recent years, multiple manufacturers recalled implanted cardioverter defibrillator leads placed in hundreds of thousands of patients.^3,4 Information about these critical safety issues only emerged after widespread adoption by physicians following their approval by the US Food and Drug Administration (FDA). As a result of these types of incidents, numerous stakeholders have called on the FDA to strengthen its postmarketing surveillance of medical devices to better inform our knowledge of the safety and effectiveness of medical devices already on the market.⁵

There are now promising signs that the agency is increasing efforts to bolster its postmarketing surveillance of medical devices. The FDA is implementing its first strategic plan for postmarket surveillance of medical devices, “Strengthening Our National System for Medical Device Postmarket Surveillance.”⁶ One key element of this plan is the FDA’s recent release of regulations⁷ requiring each high-risk medical device to have a unique identifier, starting in 2014; this identifier will make it easier to identify the specific devices used in health care and assess their outcomes.

One of the FDA’s most important tools in the postmarket setting is the authority to order the manufacturer of a high-risk device to conduct 1 or more postmarketing surveillance studies, known as postapproval studies (PASs), at the time the agency approves the device. Postapproval studies are “intended to gather specific information to address questions about the postmarket performance of or experience with an approved medical device.”⁸

The FDA guidance states, “[i]deally, the final protocol for a post-approval study and the schedule for study completion are based on agreements reached between FDA and the sponsor…prior to approval of the [device].”⁸ If no agreement is ultimately reached, or the study does not progress satisfactorily, the FDA can issue a warning letter, levy civil monetary penalties, or withdraw the marketing approval for the device.

The FDA has ordered hundreds of PASs during the past decade, but no systematic analysis of the PAS program has ever been published. This article examines the number and characteristics of PASs ordered by the agency, the accessibility of information on individual PASs to clinicians and the public, and the overall effects of the program.

The FDA website is the only publicly available source of information on PASs. We abstracted all relevant information from the FDA website as of November 5, 2012, and developed a spreadsheet of approvals of high-risk devices and PASs that had been ordered for those products. We began by capturing information on all PASs posted to the FDA PASs website for products approved between January 1, 2005, the earliest date for which PAS information is available via the FDA website, and December 31, 2011. The PASs website is intended to inform clinicians and other stakeholders of the progress of each PAS. The website contains basic information regarding each PAS, such as the device name; more detailed information, such as study characteristics and status of manufacturer reports to the FDA on PAS progress, is accessible by clicking on the study link.⁹

For each PAS, we also collected information on which FDA approval pathway the device used to enter the market: premarket approval (used for new devices; typically <40/y), panel track supplement (used for significant changes in design or new indications; <15/y), non–panel track supplement (used for changes requiring less scrutiny; thousands per year) or humanitarian device exemption (used for products for small patient populations; <10/y) (Box 1).¹³

Box Section Ref ID

We reviewed the approval orders for all original premarket approval, panel track supplement, and humanitarian device exemption applications during the same period. Approval orders for these applications were retrieved from the agency’s online database of approved products, which includes information about approved devices and can be searched by a variety of fields.¹⁴

We abstracted information that would allow us to understand why each PAS was ordered, as well as other characteristics of the study (eMethods in the Supplement). Because the PASs website only contains information on the date the most recent protocol version was approved, we requested and received data from the FDA on when the first version of each PAS protocol was approved. The agency also provided updated information on when the most recent version of each PAS protocol was approved (eTable in the Supplement).

While the PASs website includes a study design field as well as free-text fields with more detailed information, it does not provide precise definitions for the terms used, some of which may have more than 1 interpretation. To understand the types of postmarketing studies the agency required, we classified each PAS into one of the following study design categories. “Nonclinical” studies only involve laboratory testing and do not involve collection of clinical data. “Enhanced surveillance” studies require additional safety monitoring but do not explicitly require a patient cohort (eg, a study that collects additional demographic and clinical information for all reported concerns and adverse events related to a device). “Follow-up” studies involve further study of the device in a premarket patient cohort that the manufacturer had previously assembled. “Prospective cohort” studies require enrollment of a new group of patients in a cohort. “Prospective randomized” studies require patient enrollment in a randomized clinical trial. We classified studies that did not meet any of the above criteria as “other or unknown.” We designated individual PASs that included more than 1 design component (eg, prospective cohort and follow-up) as a “mixed” study. All data abstraction and classification was performed independently by 2 of us (I.S.R. and K.H.P.). In cases of disagreement, the 2 reviewers reached consensus on the appropriate classification.

In total, the FDA ordered 223 PASs on 158 medical devices approved from 2005 through 2011. Analysis by approval pathway shows that 93 of the 193 premarket approval applications approved during this period had a PAS (48%; 145 total studies). Fourteen of the 44 panel track supplements (32%; 21 total studies) and 10 of the 20 humanitarian device exemptions (50%; 11 total studies) had studies. Non–panel track supplements accounted for 46 studies. We were unable to determine the total number of devices approved by this pathway, but given that the FDA received nearly 2000 non–panel track supplements in fiscal year 2011 alone, the proportion of these applications receiving a PAS order is extremely small.

Of the 223 total studies, we classified 109 of the PASs as prospective cohort studies, 60 as follow-up, 17 as nonclinical, 7 as randomized trials, 16 as mixed, 7 as enhanced surveillance, and 7 as other or unknown (Figure). Of the 16 mixed studies, 14 included both prospective cohort and follow-up study elements. Examples of studies designated as other were patient focus groups or physician surveys. Overall, the FDA required a study that mandated new patient enrollment—a prospective cohort, a prospective randomized trial, or a mixed study that included at least 1 of the 2—for 117 (74%) applications receiving a PAS study order.

For the 168 studies on the PASs website for which the number of patients was provided, the median required sample size was 350 (interquartile range, 160-1500). Information about required sample size varied across approval letters. Some approval letters specified the minimum sample size, but in other cases the approval order specified another metric, such as minimum statistical power, or recommended follow-up of all patients from the related premarket study cohort. Other approval letters did not contain sample size information.

Based on our initial analysis of data from the FDA website, we determined that 107 study protocols (48%) were finalized at the time of application approval. However, because the date on the FDA website reflects the date of the most recent protocol approval and there may have been earlier versions of an approved study protocol, we reconducted the analysis using additional data supplied by the FDA. Using this information, we concluded that 137 PAS protocols (61%) were initially agreed on by the time of application approval, while 83 (37%) were agreed on after application approval. For protocols not in place at the date of application approval, a median of 180 days elapsed before a protocol was agreed on (mean, 230 days). Fourteen PAS protocols were not present until more than 1 year after the medical device was approved. There was missing information on protocol approval dates for 3 studies (1%).

Forty-one (19%) study protocols were revised at some point (including 29 (21%) of those in place at the time of device approval and 12 (14%) of those agreed on after the approval). A median of 220 days elapsed between the date a protocol was initially agreed on and the most recent revision (mean, 373 days). No information is publicly available regarding why protocols are revised or the extent of the changes.

A search of the FDA’s warning letter database revealed that the agency has not issued a warning letter to any manufacturer related to a PAS to date.¹⁵ We also did not find any cases in which the agency has assessed a fine or any other penalty against a manufacturer related to a PAS. Similarly, the agency has never withdrawn a device’s approval owing to failure to comply with agreed-on postmarketing surveillance.

We checked the approval order and the PASs website for specific reasons for the order of each PAS and any study hypotheses. We found significant variation in whether this information was present, where it was located, and the amount of detail that was provided. There was also inconsistent information on specific end points and outcomes of interest.

The FDA provides a status for each PAS on its website (definitions in Box 2). The agency listed 99 PASs (44%) as “progress adequate,” 58 (26%) as “completed,” 40 (18%) as “progress inadequate,” 12 (5%) as “study pending,” 9 (4%) as “other,” and 3 (1%) as “terminated.” The status of 2 other studies (1%) was not provided. No information was provided on the specific reasons that individual studies were making inadequate progress, had been terminated, or had been classified as “other.” Of the 137 studies with protocols originally agreed on by time of application approval, 50 (36%) were reported as “completed,” while only 8 (10%) of the 83 studies for which protocols were not in place until after application approval were reported as completed.

Box Section Ref ID

Specific reporting schedules vary, but agency guidance recommends that manufacturers submit a report to the FDA on the progress of the PAS every 6 months for the first 2 years of the study and then annually until its completion,⁸ which can be 5 years or more. The PAS website lists the date that each of the reports is due, when the FDA received the report, and whether the report was on time or overdue. No information was available on the content of any of the interim study reports because the information is considered nonpublic. Eighty-three (60%) of active studies—those categorized as “progress adequate” or “progress inadequate”—had submitted a status report late on at least 1 occasion. No information is available on the start date of the PASs.

In the period since our analysis, the FDA made a revision to the PASs website to clarify which protocols had been completely redesigned or replaced owing to lack of study progress. In such cases, the study status is now reported as “revised/replaced study.” Six studies have this designation.

For 54 of 58 completed studies (93%), the FDA included in-depth information on the PASs website, such as actual patient enrollment, number of sites, patient follow-up rate, safety or effectiveness findings, study strengths and weaknesses, and any recommendations for a labeling change for the device. A labeling change was requested for at least 31 of the 58 completed PASs (53%).

Studies generated findings with significant potential clinical implications in some cases. The FDA asked the maker of one product to update the device labeling “to reflect myocardial infarction (MI) and stroke rates at 3 years.”¹⁶ It asked another manufacturer to include data from the PAS that revealed “the high rates of MI and cardiac death or MI at 5 years.”¹⁷ According to the FDA’s website, one product—a carotid stent—was removed from the market owing to adverse events identified during the PAS.¹⁸ Studies can also reveal new information on device effectiveness. A randomized dosing study of a device used to treat depression found no dose-response for the treatment, information that was incorporated into the device labeling.¹⁹ There is no way to easily track how long it takes manufacturers to implement any recommended labeling changes.

The FDA ordered manufacturers to conduct studies on more than 150 high-risk medical devices between 2005 and 2011, including just less than half of the devices that entered the market by premarket approval. Delays in launching and completing studies are common. There is limited information about the causes of study delays, and we found variable detail provided on the reason a PAS was ordered and on the study hypotheses. In addition, the FDA has never issued a warning letter to a manufacturer for failing to start or complete a mandated PAS, which may undermine its authority in ordering these studies. The most common effect of a PAS was a change to device labeling. The influence of such label changes is unknown.

Our analysis revealed that the FDA typically uses PASs to address specific concerns about the safety and efficacy of a device, as evidenced by the small sample sizes of the trials. Although these studies may be powered to answer the FDA’s scientific questions, the small sample sizes mean that most PASs will not be able to assess device performance in certain patient subgroups, detect uncommon adverse events, or compare outcomes across devices. At times, these small sample sizes may make it difficult to draw any conclusions from the study at all. For example, the FDA noted the following weakness to one completed PAS: “Follow-up rate was very poor. That plus the fact that the study was very small to begin with means that the data were too sparse to make any definitive conclusions.”²⁰

While the FDA’s goal is for study protocols to be in place at the time of approval, this did not occur for 83 (37%) studies. For this delayed cohort, it took a median of 6 months to reach agreement on a protocol. Combined with other barriers to initiating new studies, such as contracting with investigators and receiving institutional review board approval, there may be an extended period between device approval and the beginning of PAS data collection. Based on our findings, we recommend that manufacturers and the FDA reach an agreement on protocols by the time of approval to enhance the value of these studies for the health of the public. The agency should use the enforcement tools at its disposal, when appropriate, and clearly convey the specific reasons that any changes to the study protocol are necessary.

We found a lack of clear and consistent information on specific study hypotheses and end points in the approval orders and on the PASs website. It would be useful for the FDA to clearly articulate—in both the approval order and on the PASs website—study hypotheses, specific study end points, and the specified sample size, study duration, and comparator groups for all studies. This clarity will benefit physicians, payers, and patients who want to understand what questions about newly approved medical devices are in need of additional study.

It was impossible to determine why almost one-third of active studies were making “inadequate progress.” This lack of information prevents providers and the public from understanding the causes of such delays. The FDA guidance instructs manufacturers to include a rationale for not meeting study milestones or timelines in their interim reports to the FDA⁸; these reasons for not meeting milestones should be made available on the PASs website. This information would allow physicians and patients to hold manufacturers and the FDA accountable for the setbacks by demanding more timely completion of the studies.

Results from interim study reports were not available, even for studies of 10 years’ duration (eg, for some orthopedic implants). Because clinicians and patients must make treatment decisions throughout the study period, the FDA and manufacturers should determine, as part of the initial protocol, which interim results should be made available, with the goal of disclosing as much information as is appropriate.

Officials from the FDA have stated that they are not able to publicly release certain information about PASs, including the study hypotheses, specific target end points, and reasons that studies are making inadequate progress or have protocol revisions. Given the importance of these data to clinicians and patients in making treatment decisions, it would be appropriate for the FDA to reevaluate its ability to make this information available and seek additional authority to do so if needed. If necessary, Congress should update the statute to allow for greater transparency of this information.

We found 1 instance of removal of a device from the market owing to problems identified in the PAS, which illustrates the potential of a robust PAS program to identify problems and contribute to ensuring the safety and effectiveness of high-risk devices, given the limited amount of premarket evidence that supports their approval.^2,21 In addition, most completed PASs had results that led to labeling changes. The FDA is to be applauded for requesting changes to the device’s labeling, but additional research is needed on the value of the labeling changes for physicians and patient outcomes, including whether changes in treatment decisions are made based on the results of the studies. There are no data indicating whether device labeling changes influence clinician or patient decision making. If labeling changes alone do not affect clinician behavior, the FDA, manufacturers, and clinical societies will need to identify other ways to communicate this information.

Our analysis has some limitations. First, there is a lack of complete information for most studies on the FDA website. As a result, it is difficult to draw conclusions about the scientific purposes of the studies, the appropriateness of the sample size, the reasons for the delays in launching studies, why many studies were making inadequate progress, and the effect of the studies. Given the role that PASs may play in the postmarketing surveillance of devices, making this information more readily available would benefit patients and clinicians. If additional statutory authority is needed to make this information available, Congress should update the law. In addition, there is no other public source of data that can be used to verify the accuracy of the information. However, there are periodic updates to the site, and we believe that the FDA and manufacturers work to ensure that the information presented is correct. Finally, when there were revisions of the agreed-on PAS protocol, it was not possible to determine the reason for the revision and if the study had begun before the revision because the FDA does not make this information available.

This analysis describes how the FDA has used PASs to better understand the postmarket performance of medical devices. Given our findings—in particular, that only 1 of 223 studies has resulted in any action other than a labeling change—we encourage the agency to work together with all stakeholders to evaluate how these studies can more effectively be used to improve the public health. In particular, an assessment of the effect and value of labeling changes—one of the distinct elements of the PAS program—is needed. Understanding how best to communicate information to clinicians on postmarketing performances should be a priority for the FDA and industry. In addition, we believe there is unrealized potential to improve the safety and effectiveness of high-risk devices without lengthening premarket approval duration if studies are initiated immediately on approval and their results are made publicly available in a timely fashion.

The small sizes of most PASs mean that they are unlikely to assess device performance in patient subgroups, detect relatively rare adverse events, or compare outcomes across devices. As a result, the FDA needs to ensure that other tools, such as large registries, are capable of answering these questions, which is a goal the agency laid out in its 2013 strategic plan for postmarket surveillance.⁶

Finally, the FDA should integrate PAS data with other sources of information, such as premarket studies, clinical registries, adverse event reports, or published studies conducted by manufacturers or the research community, in one location, such as an online portal. The FDA’s recently issued Unique Device Identifier regulations will facilitate this aggregation of data and other improvements in postmarketing surveillance.²² This development and the approaches outlined in the FDA’s first strategic plan to enhance device postmarket surveillance have the potential to greatly improve clinicians’ understanding of the postmarket performance of medical devices.

Accepted for Publication: June 28, 2014.

Corresponding Author: Ian S. Reynolds, MPH, The Pew Charitable Trusts, 901 E St NW, Washington, DC 20004 (ireynolds@pewtrusts.org).

Published Online: September 29, 2014. doi:10.1001/jamainternmed.2014.4194.

Author Contributions: Mr Reynolds had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: Reynolds, Rising, Coukell, Paulson.

Drafting of the manuscript: Reynolds, Rising, Coukell, Redberg.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Reynolds.

Administrative, technical, or material support: Reynolds, Paulson.

Study supervision: Reynolds, Rising, Coukell, Redberg.

Conflict of Interest Disclosures: Mr Coukell reports being the vice chair of the Medical Device Innovation Consortium Board of Directors and serving on the Reagan-Udall Foundation Board of Directors. Dr Redberg reports being a member of the US Food and Drug Administration Cardiovascular Devices Expert Panel. No other disclosures were reported.

Disclaimer: Dr Redberg, editor of JAMA Internal Medicine, had no role in the review or editorial decision process for this article. She was not compensated.

Additional Contributions: Naomi Herz, MSc, contributed data review and analysis.