Acuity category 1 indicates elective procedure; category 2, urgent procedure status and no preprocedure shock, inotropes, mechanical assist device, or cardiac arrest; category 3, urgent or elective procedure status, preprocedure shock, inotropes, or mechanical assist device, and no cardiac arrest less than 24 hours before the procedure; and category 4, emergent or salvage procedure status and/or cardiac arrest less than 24 hours before the procedure. The upper limit of the 95% CI for acuity category 4 is 54.0%. AV indicates aortic valve; EF, ejection fraction; NYHA, New York Heart Association; and error bars, 95% CI.
Data markers represent the observed mortality rate (position on the y-axis) in relation to the expected mortality rate (position on the x-axis); error bars, 95% CIs for the observed mortality rate; and solid lines, perfect calibration (ie, observed = expected) (included as a reference to the ideal).
eTable 1. Initial Covariates Considered for the Model
eTable 2. Definitions for Model Covariates
eTable 3. Slope and Intercept of the Model Calibration Line
Customize your JAMA Network experience by selecting one or more topics from the list below.
Edwards FH, Cohen DJ, O’Brien SM, et al. Development and Validation of a Risk Prediction Model for In-Hospital Mortality After Transcatheter Aortic Valve Replacement. JAMA Cardiol. 2016;1(1):46–52. doi:10.1001/jamacardio.2015.0326
Patient selection for transcatheter aortic valve replacement (TAVR) should include assessment of the risks of TAVR compared with surgical aortic valve replacement (SAVR). Existing SAVR risk models accurately predict the risks for the population undergoing SAVR, but comparable models to predict risk for patients undergoing TAVR are currently not available and should be derived from a population that underwent TAVR.
To use a national population of patients undergoing TAVR to develop a statistical model that will predict in-hospital mortality after TAVR.
Design, Setting, and Participants
Patient data were obtained from the Society of Thoracic Surgeons/American College of Cardiology Transcatheter Valve Therapy (STS/ACC TVT) Registry. The model was developed from 13 718 consecutive US patients undergoing TAVR in centers participating in the STS/ACC TVT Registry from November 1, 2011, to February 28, 2014. Validation was conducted using 6868 records of consecutive patients undergoing TAVR from March 1 to October 8, 2014. Covariates were selected through a process of expert opinion and statistical analysis. The association between in-hospital mortality and baseline covariates was estimated using logistic regression. The final set of predictors was selected via stepwise variable selection. Data were collected and analyzed from November 1, 2011, to February 28, 2014.
Main Outcomes and Measures
In-hospital TAVR mortality.
The development sample included 13 718 patient records from 265 participant sites (of 13 672 with data available, 6680 men [48.9%]; 6992 women [51.1%]; mean [SD] age, 82.1 [8.3] years). The final validation cohort included 6868 patients from 314 participating centers (3554 men [51.7%]; 3314 women [48.3%]; mean [SD] age, 81.6 [8.8] years). In-hospital mortality occurred in 730 patients (5.3%). The C statistic for discrimination was 0.67 (95% CI, 0.65-0.69) in the development group and 0.66 (95% CI, 0.62-0.69) in the validation group. The final model covariates (reported as odds ratios; 95% CIs) were age (1.13; 1.06-1.20), glomerular filtration rate per 5-U increments (0.93; 0.91-0.95), hemodialysis (3.25; 2.42-4.37), New York Heart Association functional class IV (1.25; 1.03-1.52), severe chronic lung disease (1.67; 1.35-2.05), nonfemoral access site (1.96; 1.65- 2.33), and procedural acuity categories 2 (1.57; 1.20-2.05), 3 (2.70; 2.05-3.55), and 4 (3.34; 1.59-7.02). Calibration analysis demonstrated no significant difference between the model (predicted vs observed) calibration line (−0.18 and 0.97 for intercept and slope, respectively) compared with the ideal calibration line.
Conclusions and Relevance
Data from the STS/ACC TVT Registry have been used to develop a predictive model of in-hospital mortality for patients undergoing TAVR. Validation based on a population of patient records not used in model development demonstrates discrimination and calibration indices that are more favorable than other models used in populations with TAVR. This model should be a valuable adjunct for patient counseling, local quality improvement, and national monitoring for appropriateness of selection of patients for TAVR.
Accurate risk prediction models for cardiac procedures are an essential component of patient-centric care. The Society of Thoracic Surgeons (STS) and the American College of Cardiology (ACC) have produced several statistical models to predict procedural outcomes based on patient characteristics and disease severity.1-4 Risk models perform best when developed from a population undergoing the specific procedure that is the focus of the model.5-8 At present, STS and ACC models are based on populations undergoing well-established conventional procedures.
In recent years, this array of conventional cardiac procedures has been augmented by transcatheter aortic valve replacement (TAVR). TAVR was developed to provide a treatment option for patients with critical aortic stenosis who are not optimal candidates for surgical aortic valve replacement (SAVR). This population includes patients with a prohibitive or a high risk for SAVR mortality. To compare the expected results of SAVR vs TAVR in a given patient, we must develop TAVR risk models that are based on large populations of patients who have undergone TAVR. This ability to assess patient risk preoperatively is a key element in patient selection9-12; yet, to our knowledge, no well-accepted statistical risk model has been designed specifically for the population undergoing TAVR.5,6,9,13-17
General agreement exists that a population undergoing TAVR will be necessary to develop a model that performs well.5-7,17,18 In the United States, a national registry was developed in 2011 to capture all commercial TAVR cases. With the support of the US Food and Drug Administration, numerous stakeholders collaborated to create the STS/ACC Transcatheter Valve Therapy (TVT) Registry.18-20 Using this rich source of real-world clinical information from virtually all commercial TAVR cases in the United States, we have developed a statistical predictive model of in-hospital mortality based entirely on a population of more than 13 000 patients undergoing TAVR.
The outcome for this model is in-hospital mortality. The ACC and STS have designated Chesapeake Research Review Incorporated as their institutional review board. A TVT Registry protocol was submitted to Chesapeake Research Review Incorporated, which approved this study and granted a waiver of informed consent.
The appropriate clinical indication for TAVR was determined by at least 2 cardiothoracic surgeons. In general, the patients undergoing TAVR were considered to be unsuitable for or at extreme risk with SAVR. Patients underwent radiographic and echocardiographic assessment to ensure anatomy appropriate for the TAVR device. Patients were offered TAVR only if their expected survival was at least 1 year.
The analysis data set included procedures performed from November 1, 2011, to February 28, 2014. From a starting population of 13 887 TVT records, we excluded 169 records (1.2%) with missing data for in-hospital mortality. The validation sample included 6868 consecutive procedures performed from March 1 to October 8, 2014. The development and validation populations had similar clinical characteristics.
Candidate covariates were selected based on prior literature and expert opinion. Potential covariates were taken from the TVT Registry data collection form (eTable 1 in the Supplement). A formal physician survey was then conducted to select the model risk factors from this list. The inclusion of nonfemoral access as a covariate bears mention, because nonfemoral access might be considered a departure from the standard practice of including only those factors present at admission. We considered nonfemoral access to be a marker of increased risk that was nondiscretionary, because the operators were compelled to undertake a riskier approach to access owing to patient factors beyond their control. For this reason, the inclusion of nonfemoral access as a potential covariate was believed to be appropriate.
The association between discharge mortality and baseline covariates was estimated via logistic regression. A combination of forward and backward steps was used. Using records from the development sample, the final set of 9 predictors (eTable 2 in the Supplement) was selected via forward stepwise variable selection with P ≤ .05 to enter and P ≤ .05 to stay. Each forward-selection step was followed by 1 or more backward elimination steps. The selection process was terminated if no additional variables could be added at P ≤ .05 or if the current model was identical to a previously tested model.
Model covariates were highly complete, with most candidate covariates having less than 2% missing data and only 1 candidate covariate having more than 3% missing data (ejection fraction [3.2%]). In the rare case of missing data, unknown values were imputed to the most common categorical variable and to the median or subgroup-specific median of continuous variables. Multiple imputation was not used because of the very low rate of missing data and because multiple imputation had a negligible impact in models previously developed by the STS and ACC in cohorts with similarly low rates of missing data.1-3
We decided to perform stepwise selection after performing Monte Carlo simulations and bootstrap resampling to assess the consistency of variable selection across randomly resampled data sets. The multivariable association model covariates and mortality were summarized by reporting odds ratios with approximate 95% CIs. Confidence intervals were constructed with empirical (sandwich) SEs to account for the clustered data structure. Clustered data refers to the lack of statistical independence between patients within the same hospital. Analyses were performed from November 1, 2011, to February 28, 2014, using SAS (version 9.3; SAS Institute Inc) and R (version 3.0.3; https://cran.r-project.org) software.
Calibration was assessed by plotting observed vs expected mortality rates within prespecified subgroups and across quintiles of predicted risk among patients in the validation sample. The Hosmer-Lemeshow goodness-of-fit test was also used for objective assessment of calibration. We conducted a test of overfitting in the validation sample by estimating the association of in-hospital mortality and each patient’s predicted log odds of mortality according to the final model. The test was performed by fitting a univariable logistic regression with discharge mortality as the outcome variable and the patient’s predicted log odds of mortality according to the final risk model as the only explanatory variable. A perfectly calibrated model has an intercept of 0 and a slope of 1. A Wald test was used to assess whether the actual estimated values were significantly different from these ideal values. Discrimination was assessed by calculating the C statistic (the area under the receiver operating characteristics curve) among patients in the validation sample.
From a starting cohort of 6877 patients undergoing TAVR from March 1 to October 8, 2014, we excluded 9 patients with missing discharge mortality status, for a final validation cohort of 6868 from 314 participating sites (3554 men [51.7%]; 3314 women [48.3%]; mean [SD] age, 81.6 [8.8] years). The final model covariates were age, glomerular filtration rate, hemodialysis, New York Heart Association (NYHA) functional class IV, severe chronic lung disease, nonfemoral access site, and procedural acuity. Procedural acuity, sometimes termed operative priority, refers to the patient’s preprocedure clinical state that determines the urgency of the procedure. Higher-acuity status is associated with higher degrees of acute clinical compromise and the need for more urgent intervention. Definitions are listed in eTable 2 in the Supplement.
In the development sample, the final study cohort included 13 718 records from 265 participant sites (of 13 672 with data available, 6680 men [48.9%]; 6992 women [51.1%] mean [SD] age, 82.1 [8.3] years). In-hospital mortality occurred in 730 patients (5.3%). The model coefficients and odds ratios with 95% CIs for each covariate are given in Table 1. The model intercept was −4.72976. The C statistic was 0.67 (95% CI, 0.65-0.69) in the development sample and 0.66 (95% CI, 0.62-0.69) in the validation sample.
Calibration plots were generated using predicted vs observed results, as described above. To obtain objective quantification of the degree of calibration, the overall calibration line was compared with a line demonstrating perfect calibration (intercept, 0; slope, 1). The model values of −0.18 and 0.97 for intercept and slope, respectively, were not significantly different from the perfect calibration line. As shown in eTable 3 in the Supplement, we found no significant difference in the validation sample in the slope and intercept of the model calibration line and an ideal line denoting perfect calibration.
As shown in Figure 1, we found good agreement between predicted vs observed in-hospital mortality rates. The wide 95% CIs seen in acuity category 4 are a consequence of the small number of patients in that category. Calibration lines of predicted vs observed results for age and sex are shown in Figure 2. Model calibration was also assessed in subgroups defined by ejection fraction of less than 35% or of 35% or greater, NYHA class IV or other than class IV, acuity categories, and a prior or no prior aortic procedure. Results were similar to those shown in Figure 2. The Hosmer-Lemeshow goodness-of-fit test with 10 subgroups was found to have a P value of .20 with χ210 = 13.4.
This model based on TVT Registry data demonstrates excellent calibration in the overall population and in important clinical subsets. Model discrimination as measured by the C statistic is higher than that for previously reported models that have been used in the population undergoing TAVR.5,6,21
The STS risk models and the logistic European System for Cardiac Operative Risk Evaluation models5 were developed to predict risk in conventional surgical procedures such as SAVR. In addition, these models have been used to predict risk in the population with TAVR, but because the models were not specifically developed from a patient population undergoing TAVR, their application in this context is questionable.5,6,9,13-17
The TVT Registry provides an ideal source of information on virtually all commercial TAVR cases in the United States. The National Coverage Determination from the Centers for Medicaid & Medicare Services18 specifies that reimbursement is contingent on participation in a national registry with an array of specifications, all of which are met by the TVT Registry. This stipulation indicates that virtually all patients with TAVR are entered into this registry, thereby creating a real-world database of the TAVR experience. The registry now has more than 20 000 patient records with detailed pre-TAVR clinical information and post-TAVR outcomes, including procedural mortality. This information provides an ideal substrate for the development of a statistical model designed to predict procedural outcomes.
The Steering Committee of the TVT Registry has been committed to the development of a series of risk models that will be based on TVT Registry data. The initial outcome to be modeled was selected after a thorough review of collected data. In-hospital mortality is a clearly defined outcome with obvious clinical importance and a high degree of completeness, thereby making it the best candidate for modeling at the present time. Subsequent models based on survival of 30 days and longer are planned (see below).
The selection of covariates took into account the necessary balance between inclusion of the most important risk factors on one hand and a desire for reasonable parsimony on the other. We began with an initial set of candidate variables (eTable 1 in the Supplement) and through a process of expert opinion reduced the number to 14. Further statistical analysis determined that reducing the set to 9 covariates (eTable 2 in the Supplement) had no effect on model performance indices. After this process by the development task force, the covariates were scrutinized by the entire TVT Registry Steering Committee to arrive at the final set of 9 risk factors. This set of risk factors will be subject to continuous evaluation and is certain to change as more data are gathered into the TVT Registry.
Validation of the TVT Registry model was performed using a training set–test set approach. The model was developed from a population of 13 718 records and validated against a population of 6868 records that had not been used in model development.
Discrimination was assessed using the C statistic.22,23 When recently used in a population with TAVR, the C statistic was reported to be 0.60 for the standard STS valve mortality risk model and 0.53 for the European System for Cardiac Operative Risk Evaluation.5 The STS model C statistic was found to be 0.62 when applied to the specific TVT Registry validation sample used in the present study. The C statistic of 0.66 for the TVT Registry model presented herein is an improvement over the previously reported C statistic for the standard STS valve mortality risk model,5 the European System for Cardiac Operative Risk Evaluation,5 and the French Aortic National Core Valve and Edwards 2 (FRANCE-2) model.21 Only the TVT Registry and the FRANCE-2 models are based solely on a population with TAVR. A comparison of covariates used in each model is shown in Table 2. We should also mention that the slope and intercept of the TVT Registry model calibration line demonstrated considerable improvement over the calibration data for the STS risk model and European System for Cardiac Operative Risk Evaluation model when used in a population undergoing TAVR.5
This model was developed to assess patient-level outcome predictions, in contrast to other models that have been developed for comparison of center-level risk-adjusted outcomes. The performance of the TVT Registry model compared with other models indicates that the TVT Registry model may have a role in patient selection. The model results should not dictate which patients are candidates for TAVR; rather, the model should be used as 1 element in the selection process to be considered in concert with history, physical examination, laboratory information, and clinical judgment. The model may also provide useful information for patient counseling. Patients may be advised of their predicted TAVR procedural mortality based on a national risk-adjusted analysis of real-world patients with a clinical profile similar to their own. This expected TAVR mortality can be compared with the risk for SAVR mortality as determined from existing STS models that are derived from a broad national population. The TVT Registry model predicts in-hospital mortality, whereas the STS SAVR model predicts 30-day mortality.
As data accumulate, the TVT Registry model should yield valuable information about the evolution of the clinical profile of those patients undergoing TAVR. The change in the predicted risk over time will provide an objective way to monitor for indication creep associated with a liberalization of procedural indications over time. This type of analysis may well define clinical subsets of patients who accrue particular benefit from the procedure or, conversely, reveal subsets not well served by TAVR.
The value of providing individual centers with risk-adjusted outcome data for comparison against national benchmarks is widely recognized.3,24,25 This information will be provided to TVT Registry participant sites several times each year so that local quality programs can monitor risk-adjusted results in this challenging population. The reports will be based on TVT Registry model data and will be presented in a manner that facilitates the identification of areas that could be improved.
Although the model adjusts for numerous key covariates, not all risk factors could be included. Frailty indices such as the 5-m walk test of gait speed26 and quality-of-life measures such as the Kansas City Cardiomyopathy Questionnaire26 are generally recognized as potentially important factors, but at this point they are incompletely collected, thereby precluding consideration in the model. Using data taken entirely from the early TAVR experience, the selected risk factors in this model may not include covariates that will eventually be found to influence mortality. In keeping with the design of the TVT Registry, patients enrolled in pivotal trials are not included in the registry. All data in the TVT Registry are subject to internal data quality checks; however, the external audit process is in its early stages and data quality cannot be ensured completely. An independent auditing group was scheduled to begin formal audits in 2015.18
The model presented herein is the first of a series of risk models that will be developed by the TVT Registry. A model for 30-day mortality is particularly important because an appreciable number of patients undergoing TAVR survive the hospitalization but succumb within 30 days. Data for 30-day mortality are more difficult to capture because 30-day follow-up requires contacting patients after discharge. Aggressive efforts are under way to ensure a high rate of data return for this critical variable so that it can be the focus of upcoming models.
The TVT Registry is collecting data to determine 1-year survival. The 1-year survival information requires linking with the Centers for Medicaid & Medicare Services data in a challenging administrative process. At this early stage of the registry, these data are sparse, but in the near future adequate information for modeling should be possible.
The ability to predict and risk adjust for nonfatal outcomes is essential for this high-risk population. Models to predict the probability of neurologic deficit are now being developed, and other nonfatal outcome models will be added in the future.
Patient frailty is determined by a preprocedure 5-m walk test.26 In the elderly population with TAVR, frailty may be an important risk factor for periprocedural complications and longer-term procedural benefit.27-29 In the last 2 years, frailty data are available for approximately 75% to 80% of patient records. As more complete frailty data become available, the impact of frailty will be examined statistically to determine its value as a covariate. Frailty will likely be included as a covariate in forthcoming TVT Registry risk models.
The TVT Registry is one of the few clinical registries to collect quality-of-life data. The Kansas City Cardiomyopathy Questionnaire is administered before and at 30 days and 1 year after the procedure. Information gained from the Kansas City Cardiomyopathy Questionnaire can serve as a general index of procedural benefit. Linking this information to the wealth of clinical data in the TVT Registry will enable the creation of models that can predict the probability of the TAVR benefit. This link will be especially valuable in the population undergoing TAVR and should be taken into account with predicted survival data. For example, a patient may have a favorable predicted TAVR mortality but a low probability of procedural benefit. Information of this kind will be readily available in this new generation of risk models that should be powerful adjuncts in the process of appropriate patient selection.
Data from the STS/ACC TVT Registry from 2011 to 2014 have been used to develop a predictive model of in-hospital mortality for patients undergoing TAVR. We used 13 718 patient records to develop the model and 6868 patient records not used in model development for validation. The validation process demonstrated discrimination and calibration indices that are more favorable than other models that have been used in populations undergoing TAVR. This model is being incorporated into the standard TVT Registry software and should be a valuable adjunct for patient counseling, performance assessment, local quality improvement, and national monitoring of the appropriateness of patient selection for TAVR.
Corresponding Author: Fred H. Edwards, MD, Department of Surgery, University of Florida College of Medicine–Jacksonville, 653 W Eighth St, Jacksonville, FL 32209 (firstname.lastname@example.org).
Accepted for Publication: December 10, 2015.
Published Online: March 9, 2016. doi:10.1001/jamacardio.2015.0326.
Author Contributions: Dr Edwards had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Edwards, Cohen, Peterson, Mack, Shahian, Grover, Carroll, Brennan, Rumsfeld, Holmes.
Acquisition, analysis, or interpretation of data: Edwards, O’Brien, Peterson, Mack, Shahian, Tuzcu, Thourani, Brennan, Brindis, Rumsfeld.
Drafting of the manuscript: Edwards, Mack, Shahian, Carroll, Holmes.
Critical revision of the manuscript for important intellectual content: Edwards, Cohen, O’Brien, Peterson, Mack, Shahian, Grover, Tuzcu, Thourani, Brennan, Brindis, Rumsfeld.
Statistical analysis: O’Brien, Peterson, Mack, Shahian, Tuzcu.
Administrative, technical, or material support: Peterson, Mack, Carroll, Brennan, Holmes.
Study supervision: Edwards, Mack, Shahian, Grover, Thourani, Brennan, Holmes.
Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Peterson reported receiving grants and personal fees from Janssen and Eli Lily and personal fees from Boehringer Ingelheim, Bayer, and AstraZeneca. Dr Mack reported being a member of the Executive Committee of the Placement of Aortic Transcatheter Valve (PARTNER) Trial sponsored by Edwards Lifesciences. Dr Grover reported being vice chair of the Society of Thoracic Surgeons/American College of Cardiology Transcatheter Valve Therapy (STS/ACC TVT) Registry steering committee. Dr Thourani reported serving as research advisor for Edwards Lifesciences and Medtronic. Dr Carroll reported being local site investigator for the PARTNER 2 Trial sponsored by Edwards Lifesciences. Dr Brindis reported being senior medical officer for the National Cardiovascular Data Registry. Dr Rumsfeld reported being chief science officer for the National Cardiovascular Data Registry. Dr Holmes reported being chair of the STS/ACC/TVT Registry Steering Committee. No other disclosures were reported.
Funding/Support: This study was supported by the ACC Foundation’s National Cardiovascular Data Registry and the STS.
Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Group Information: Members of the Steering Committee of the Society of Thoracic Surgeons/American College of Cardiology Transcatheter Valve Therapy Registry include the following: David R. Holmes Jr, MD (Chair, ACC representative), Frederick L. Grover, MD (vice chair, STS representative), Fred H. Edwards, MD (member, STS representative), David M. Shahian, MD (member, STS representative), Vinod H. Thourani, MD (member, STS representative), John Carroll, MD (member, ACC representative), E. Murat Tuzcu, MD (member, ACC representative), Ralph G. Brindis, MD, MPH (ex-officio, ACC representative), Danica Marinac-Dabic, MD, PhD (ex-officio, US Food and Drug Administration representative), Lori Ashby, MA (ex-officio, Centers for Medicare & Medicaid Services representative), Eric D. Peterson, MD, MPH (ex-officio, Analytic Center liaison, Duke Clinical Research Institute), and Michael J. Mack, MD (ex-officio, National Cardiovascular Data Registry management board liaison).
Disclaimer: The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the STS or the ACC Foundation or its associated professional societies identified at http://CVQuality.ACC.org/NCDR.