Risk Prediction Tools to Improve Patient Selection for Carotid Endarterectomy Among Patients With Asymptomatic Carotid Stenosis

Key Points

Question  Which patients with asymptomatic carotid stenosis are unlikely to benefit from carotid endarterectomy?

Findings  Among 2325 veterans in this cohort study that used Veterans Administration and Medicare data, a risk prediction tool (Carotid Mortality Index) based on 23 candidate variables and a simpler model based on the number of 4 key comorbidities (any cancer, chronic obstructive pulmonary disease, congestive heart failure, and chronic kidney disease [the 4C model]) were developed to identify patients at higher risk of 5-year mortality. The models were internally validated.

Meaning  Study results suggest that both the Carotid Mortality Index and the 4C model may be used to inform clinicians whether a patient will live long enough to benefit from carotid endarterectomy.

Importance  Randomized clinical trials have demonstrated that patients with asymptomatic carotid stenosis are eligible for carotid endarterectomy (CEA) if the 30-day surgical complication rate is less than 3% and the patient’s life expectancy is at least 5 years.

Objective  To develop a risk prediction tool to improve patient selection for CEA among patients with asymptomatic carotid stenosis.

Design, Setting, and Participants  In this cohort study, veterans 65 years and older who received both carotid imaging and CEA in the Veterans Administration between January 1, 2005, and December 31, 2009 (n = 2325) were followed up for 5 years. Data were analyzed from January 2005 to December 2015. A risk prediction tool (the Carotid Mortality Index [CMI]) based on 23 candidate variables identified in the literature was developed using Veterans Administration and Medicare data. A simpler model based on the number of 4 key comorbidities that were prevalent and strongly associated with 5-year mortality was also developed (any cancer in the past 5 years, chronic obstructive pulmonary disease, congestive heart failure, and chronic kidney disease [the 4C model]). Model performance was assessed using measures of discrimination (eg, area under the curve [AUC]) and calibration. Internal validation was performed by correcting for optimism using 500 bootstrapped samples.

Main Outcome and Measure  Five-year mortality.

Results  Among 2325 veterans, the mean (SD) age was 73.74 (5.92) years. The cohort was predominantly male (98.8%) and of white race/ethnicity (94.4%). Overall, 29.5% (n = 687) of patients died within 5 years of CEA. On the basis of a backward selection algorithm, 9 patient characteristics were selected (age, chronic kidney disease, diabetes, chronic obstructive pulmonary disease, any cancer diagnosis in the past 5 years, congestive heart failure, atrial fibrillation, remote stroke or transient ischemic attack, and body mass index) for the final logistic model, which yielded an optimism-corrected AUC of 0.687 for the CMI. The 4C model had slightly worse discrimination (AUC, 0.657) compared with the CMI model; however, the calibration curve was similar to the full model in most of the range of predicted probabilities.

Conclusions and Relevance  According to results of this study, use of the CMI or the simpler 4C model may improve patient selection for CEA among patients with asymptomatic carotid stenosis.

Carotid endarterectomy (CEA) is one of the most common types of major surgery in the United States and is 1 of only 2 primary prevention surgical procedures available. Randomized clinical trials^1,2 published in 1995 and 2004 showed that, among carefully selected patients with asymptomatic carotid stenosis and experienced surgeons, the procedure reduced the risk of stroke and death compared with medical therapy alone.

Revascularization with CEA is a trade-off between higher perioperative short-term risks in exchange for a lower long-term risk of stroke. Based on the aforementioned trials,^1,2 guidelines concluded that candidate patients need to have a 30-day surgical complication rate less than 3% and a life expectancy of at least 5 years for the longer-term benefits of revascularization to outweigh the higher short-term risks of the procedure.³ Therefore, patient selection is a key factor in determining who benefits from intervention.^4-6 In a randomized clinical trial, patients are excluded based on comorbid conditions; however, such exclusions may not be observed in routine practice.^7-9 If in routine clinical practice patient selection does not conform to the original exclusion criteria outlined in the trials, patients may not live long enough to benefit from intervention. The development of risk prediction tools that can assist surgeons in identifying patients who are most likely to live long enough to benefit from CEA can improve patient outcomes.

Multiple studies^10-14 have developed risk prediction tools focused on reducing 30-day complication rates. Few studies^15-17 have focused on developing prediction models to identify patients who will live long enough to benefit from revascularization. In this cohort study, we use national Veterans Administration (VA) and Medicare data to examine factors predictive of 5-year mortality among older veterans with asymptomatic carotid stenosis who underwent CEA.

We included all veterans who received both carotid imaging and CEA within 1 year of carotid imaging in the VA between January 1, 2005, and December 31, 2009. The dates of analysis were January 2005 to December 2015. We first identified 13 383 index carotid imaging tests among those 65 years and older performed in the VA between 2005 and 2009 using a previously developed natural language processing algorithm.¹⁸ This algorithm identified all veterans who had received carotid ultrasonography, magnetic resonance angiography, and computed tomographic angiography in the VA and who had at least 50% or moderate stenosis. We then manually reviewed the carotid imaging tests and excluded patients with less than 70% stenosis and patients with a stroke or transient ischemic attack in the prior 6 months. We then identified all veterans who had received CEA in the VA within 1 year of the carotid image (n = 2532). We excluded 181 veterans who had a stroke or transient ischemic attack between the imaging and CEA and excluded 26 patients who received a coronary artery bypass graft on the same day as carotid revascularization. The final cohort included 2325 veterans. These patients were followed up for 5 years from the date of CEA (2005-2015). This study was approved by the University of California, San Francisco Institutional Review Board.

Our primary outcome was mortality due to any cause within 5 years of CEA. We also collected data on complications within 30 days of the procedure, including stroke, acute myocardial infarction, and death. We used the high-specificity algorithm by Tirschwell and Longstreth¹⁹ to identify strokes within 30 days. The VA Suicide Data Repository and the VA Vital Status Files were used to identify 30-day mortality.²⁰ We identified 30-day acute myocardial infarction rates using the International Classification of Diseases, Ninth Revision (ICD-9) code 435.x on hospital discharges for admissions within 30 days of CEA.

Candidate predictor variables were selected based on review of published models in the literature.^10,14-16,21 We identified previously published predictors of 1-year and 5-year survival.^15,17,21 Because this literature was limited, we also included variables from a previously validated model of 30-day surgical complications that focused on frailty and function as predictors of 30-day surgical complications because these variables may be relevant to outcomes among older adults.¹⁴ We identified 23 candidate variables in the literature that were previously found to be predictive of either 30-day complication rates, 1-year mortality, or 5-year mortality as summarized in Table 1. Demographic variables, including age, sex, and race/ethnicity, were derived from VA and Medicare data. Data on comorbid conditions were constructed with ICD-9 codes using both VA and Medicare data. We required at least 2 outpatient visits or 1 inpatient visit with a diagnosis code to classify a patient as having a comorbid condition. Using a previously developed algorithm (eTable 1 in the Supplement), we identified all veterans who had a cancer diagnosis in the 5 years preceding CEA.²² Nonmetastatic prostate cancer was not included in this variable given the generally favorable prognosis and high prevalence of this condition among older men. The dementia variable was constructed using a combination of ICD-9 codes and data on use of medications (eg, donepezil hydrochloride) in the year before CEA. Data on medications were extracted from the VA Pharmacy Files. Data on body mass index were extracted from the Corporate Data Warehouse.²⁰ Logical Observation Identifier Names and Codes (LOINC) were used to identify laboratory test data. We extracted data on kidney function, hemoglobin level, platelet count, and international normalized ratio from the Corporate Data Warehouse. We considered patients to have evidence of functional impairment if they were hemiplegic, had a pressure ulcer in the past year, were admitted to a nursing home, received home-based primary care, had a home health aide, or were dispensed a wheelchair, walker, or cane. Data on hemiplegia were extracted using ICD-9 codes, and data on pressure ulcers were extracted from the VA health factors files. Data on nursing home use and home health aide use were extracted from VA encounter files and VA Fee Basis files and Medicare data. Data on mobility devices were extracted from the VA prosthetics file and Medicare durable medical equipment files. Data on ejection fraction and pulmonary function were extracted from the medical record by trained abstractors (M.G., A.S.A., and A.J.Z.).

We summarized categorical variables by frequency and percentage; continuous variables were summarized with means (SDs). We also present 5-year mortality curves, stratified by the count of the 4C model comorbidities, estimated as the complement of the Kaplan-Meier survival curves. We conducted a simple univariate analysis to examine the association of each candidate predictor variable with the odds of 5-year mortality. We also report the unadjusted estimates of the association between the count of the following 4 comorbidities with 5-year mortality: (1) any cancer in the past 5 years, (2) chronic obstructive pulmonary disease (COPD), (3) congestive heart failure (CHF), and (4) chronic kidney disease (CKD) with estimated glomerular filtration rate (eGFR) less than 60 mL/min/1.73 m². Odds ratios (ORs) and 95% CIs derived from simple logistic regression of 5-year mortality with each of the predictors are listed in Table 2. For comorbid variables with data on level of severity (eg, CKD [eGFR], COPD [forced expiratory volume in the first second of expiration], and diabetes [insulin or no insulin]), we compared the discrimination (area under the curve [AUC]) and fit (Akaike information criterion and Nagelkerke R²) of the model that incorporated their dichotomously coded (yes or no) versions with those of the model that incorporated the noncollapsed version of the comorbid variable, and the best-performing version was retained in further analyses (Table 2). To reduce the number of predictors to a set small enough to be clinically expedient, we performed model selection using backward variable elimination (with a threshold of 2-sided P < .005).²³ We also tested different versions of comorbid conditions (eg, cancer in the past 1 year vs cancer in the past 5 years) by allowing them to compete in the backward elimination process. All pairwise interactions between the 9 variables in the resulting model (age, chronic kidney disease, diabetes, chronic obstructive pulmonary disease, any cancer diagnosis in the past 5 years, congestive heart failure, atrial fibrillation, remote stroke or transient ischemic attack, and body mass index) were tested and retained if 2-sided P < .001. We calculated the AUC and Nagelkerke R² for the resultant model in the full cohort. We then used a bootstrap approach to obtain optimism-corrected measures of model performance. To do so, we (1) applied all model-building steps described previously in each bootstrap sample, (2) assessed the performance of this model in the bootstrap sample and in the original data set, and (3) took the difference of the 2 estimates obtained in (2) to get an estimate of optimism. Finally, the mean of 500 bootstrapped estimates of optimism was subtracted from the initial (full cohort model) estimate of the AUC and Nagelkerke R² to obtain the bootstrap optimism-corrected estimates of performance.

Backward variable elimination is known to produce estimated regression coefficients that are biased and CIs that do not have a coverage probability of 95%. To improve estimation of regression coefficients and 95% CIs, we used the zero-corrected bootstrap (number of bootstrapped samples = 3000) model selection procedure. In a given bootstrap sample, predictor variables that are not selected for inclusion in the final regression model have their regression coefficient set to zero. Regression coefficients are averaged across the bootstrap samples, and nonparametric percentile bootstrap 95% CIs are then constructed for each regression coefficient.²⁴

Finally, a multivariable logistic model consisting only of age group and a count of the number of the 4 comorbidities (any cancer, COPD, CHF, and CKD) was created (the 4C model). The AUC and Nagelkerke R² were calculated, and internal validation was performed by correcting for optimism using 500 bootstrapped samples.²⁵

To test whether the count of comorbidities was as good as allowing the individual comorbidity coefficient to vary, likelihood ratio tests comparing the 4C model (with the 4C model augmented with each comorbid condition separately) were performed. This analysis demonstrated that the simplifying assumption of equal weight for each comorbidity was reasonable for COPD, CHF, and CKD but not for cancer. However, the Nagelkerke R² measure of model fit was similar for both the parsimonious and 4C models.

To examine and compare calibration of the 2 risk models (Carotid Mortality Index [CMI] and 4C), the observed occurrence of mortality within 5 years was regressed against the model-predicted probabilities using loess smoothing.²⁶ Optimism-corrected calibration curves (number of bootstrap samples = 500) and the 45-degree reference line represent ideal model fit and were overlaid on the same plot.

A 5-year mortality CMI was created by multiplying the zero-corrected bootstrap regression coefficients by 10 and rounding, and then summing each patient score accordingly. The resulting score was approximately normally distributed, with a mean (SD) of 8.6 (8.5). The distribution of risk scores was divided into quintiles, and 5-year mortality (and 95% CI) for these 5 groups was estimated. We also present predicted probabilities and 95% CIs from the 4C model. All analyses were performed with statistical software (SAS Enterprise Guide, version 7.1; SAS Institute Inc).

In total, 2325 veterans were included in the cohort. The mean (SD) age of the cohort was 73.74 (5.92) years. The cohort was predominantly male (98.8%) and of white race/ethnicity (94.4%). The most prevalent comorbidities were coronary artery disease (53.5% [n = 1243]), CKD (49.5% [n = 1150]), diabetes (40.1% [n = 932]), COPD (27.3% [n = 635]), any cancer (25.9% [n = 603]), and CHF (17.6% [n = 409]). Approximately 10% (n = 242) of veterans who received CEA had functional impairment (Table 1).

The 30-day complication rate (composite of stroke, acute myocardial infarction, and death) was 4.9% (n = 114), with a combined stroke and death rate of 3.0% (n = 70) (Table 1). Overall, 29.5% (n = 687) of patients died within 5 years of CEA. Veterans 80 years and older had an observed mortality rate of 42.7% (170 of 398) (Table 2 and eFigure 1 in the Supplement). Patients with 0, 1, 2, and 3 or 4 comorbidities had observed mortality rates of 18.8% (114 of 607), 24.6% (224 of 909), 37.4% (213 of 570), and 56.9% (136 of 239), respectively (Table 2 and eFigure 2 in the Supplement).

Among the 23 candidate variables evaluated, 20 were nominally associated with 5-year survival. Factors most strongly associated with mortality within 5 years included the following: age 80 years and older (OR, 2.42; 95% CI, 1.87-3.10), CHF with ejection fraction of 35% or less (OR, 3.40; 95% CI, 2.50-4.70), CKD with eGFR of less than 30 mL/min/1.73 m² (OR, 4.01; 95% CI, 2.60-6.10), dialysis in the past 3 months (OR, 10.86; 95% CI, 2.30-50.00), moderate or severe COPD (OR, 2.95; 95% CI, 2.20-3.90), diabetes taking insulin (OR, 1.81; 95% CI, 1.36-2.40), dementia (OR, 1.85; 95% CI, 1.33-2.60), atrial fibrillation (OR, 2.29; 95% CI, 1.78-3.00), underweight (OR, 4.64; 95% CI, 1.91-11.30), and hemoglobin level of less than 10 g/dL (OR, 3.58; 95% CI, 1.85-6.90) (Table 2) (to convert hemoglobin level to grams per liter, multiply by 10.0).

A risk prediction tool (CMI) based on 23 candidate variables identified in the literature was developed using VA and Medicare data. On the basis of our backward selection algorithm, we selected 9 patient characteristics (age, CKD, diabetes, COPD, any cancer diagnosis in the past 5 years, CHF, atrial fibrillation, remote stroke or transient ischemic attack, and body mass index) for the final logistic model for 5-year mortality. The final model had an AUC of 0.706. These 9 variables represent 97.5% of the potential AUC if all 23 candidate variables had been included in the model. Internal validation by bootstrapping analysis demonstrated an optimism-corrected AUC of 0.687 for the CMI (Table 3). A nomogram is provided in eTable 2 in the Supplement. An online calculator is also available (https://is.gd/CEA_Risk_Calculator).

The interaction between age and the number of comorbidities was not statistically significant. The count of the 4C model comorbidities was strongly associated with 5-year mortality, with the number of comorbidities demonstrating increasing odds of 5-year mortality with increasing comorbidity as follows: 1 (OR, 1.31; 95% CI, 1.02-1.70), 2 (OR, 2.33; 95% CI, 1.78-3.10), and 3 or 4 (OR, 5.22; 95% CI, 3.70-7.30). The model AUC was 0.661, and the optimism-corrected AUC was 0.657 (Table 3). The 4C model had slightly worse discrimination compared with the CMI model; however, the calibration curve was similar to the CMI model in most of the range of predicted probabilities.

The 4C model had slightly worse fit (Table 3) than the CMI. However, calibration was as good as for the CMI for most of the range of predicted probabilities (Figure 1), with both models underestimating risk at higher levels (ie, calibration curves lay above the perfect-fit line).

Figure 2 shows the estimated probability of 5-year mortality based on quintiles of the CMI. eFigure 3 in the Supplement shows the estimated probability of 5-year mortality based on age and the number of comorbidities using the 4C model.

In this cohort study that used VA and Medicare data, 29.5% (687 of 2325) of veterans 65 years and older who received CEA died within 5 years. Outcomes among older veterans are similar to those of patients with Medicare coverage and patients revascularized in other health systems.^27-30 Our data suggest that health systems can improve patient selection to ensure that patients will achieve a net benefit from carotid revascularization. We developed both simple and more complex risk prediction tools that can be used to improve patient selection for CEA in the VA.

Our study replicates and builds on the findings of 2 other published 5-year risk prediction models in the literature. One was developed in a sample of 4114 patients with asymptomatic carotid stenosis cared for by the Vascular Study Group of New England.¹⁶ This model found that age, diabetes, CHF, COPD, renal function, dialysis dependency, severity of contralateral stenosis, and statin use were associated with 5-year mortality. The other, an analysis of both 1-year and 5-year outcomes using the CEA Vascular Quality Initiative (VQI) registry, found that age, diabetes, coronary artery disease, COPD, renal function, dialysis dependency, and absence of statin use were associated with mortality.^21,31 Contralateral stenosis was not evaluated in the VQI-derived model. In addition, patients whose data were included in the VQI registry who had evidence of a major medical contraindication (CHF class III/IV, left ventricular ejection fraction <30%, unstable angina, recent acute myocardial infarction, or COPD with forced expiratory volume in the first second of expiration <30%) were not eligible for CEA. Therefore, it appears that the severity of disease in the VQI population was different from that of our cohort, although the mean age was similar. We confirmed that all variables identified in the VQI model^21,31 and by the Vascular Study Group of New England¹⁶ were important predictors of 5-year survival. However, our final model included several factors that were not evaluated in these articles. For example, the model based on the VQI registry did not include CHF as a variable, and neither study included cancer. Both of these variables contributed to model performance and were included in our final model. Our model did not include dialysis as a factor because only 11 patients in our sample were on dialysis. Our data also confirm prior work³² suggesting that patients on dialysis are not appropriate candidates for CEA, with 5-year mortality herein among those with dialysis in the past 3 months of 81.8% (9 of 11) (Table 2). The present study builds on the prior literature and includes all relevant variables identified in this earlier work.

This emerging body of research^16,17 suggests that these risk prediction models should be used to improve patient selection for CEA. The CMI was developed based on a population of veterans with carotid stenosis who received CEA in the VA and is especially informative to practice in the VA. General mortality models (eg, the Lee Index³³) have been developed using data from the US adult population and can also be incorporated into assessments regarding the benefits of CEA among patients with asymptomatic disease. The decision to operate could be individualized with use of multiple different mortality calculators^15-17,33 that have been developed in recent years. However, at this juncture, it is unclear what 5-year mortality risk is acceptable from a public health perspective to ensure benefit from revascularization for an individual patient. Given that multiple risk scores developed in different populations have found comparable results, risk scores estimating 5-year mortality should be incorporated into updated guidelines on the management of patients with asymptomatic carotid stenosis. Expert panels should weigh in on what constitutes an acceptable risk of 5-year mortality before consideration of surgery and provide guidance to clinicians on how many patients should be exposed to a potentially unnecessary surgical procedure to prevent 1 stroke. Five-year survival among participants who received medical therapy in asymptomatic carotid stenosis trials could offer a basis for guidelines and provide the outer bounds of an acceptable 5-year mortality range to target using risk prediction tools.

To facilitate implementation clinically, we also developed a simpler model that can be easily remembered in clinical practice (any cancer in the past 5 years, COPD, CHF, and CKD [the 4C model]). This model, based on the number of 4 key comorbidities that were prevalent and strongly associated with 5-year mortality, demonstrated that patients with multiple 4C model comorbidities are less likely to benefit from revascularization. While the model is limited by equally weighting any cancer diagnosis, COPD, CHF, and CKD, it is easy to remember and provides a quick assessment of potential 5-year survival.

Our study has several limitations that deserve comment. First, we did not include patient self-reported assessments of function because these data are unavailable in the VA electronic medical record. We used measures that were suggestive of poor function in the medical record, such as a recent pressure ulcer, recent admission to a nursing home, and use of mobility devices. This may be the reason why functional status was not retained in our final model, although other investigators have found it to be an important predictor of survival.³³ Second, similar to other models,³³ we found that cancer predicts poor survival. Given our sample size, we could not determine which cancers were the most important to consider and combined all types of cancer. The results of our study and other studies suggest that cancer diagnosis is important. A patient’s cancer type and overall prognosis should be weighed carefully in individual decision making. Third, we used backward selection to select predictor variables for inclusion in the final model; however, the candidate predictor variables with which we began were not selected based on convenience or availability but were selected based on prior research demonstrating that these factors were important in other 5-year mortality models. Fourth, the models in this work underwent only internal validation. However, we used computer-intensive methods to adjust our estimates of model performance for optimism. Future work should focus on external validation in a more recent period in the VA. Fifth, models derived in older male veterans may not be generalizable to younger veterans or the general adult population; however, CEA is more commonly performed among older men. Therefore, while the models are limited in generalizability, they are still instructive.

Both a risk prediction tool based on 9 variables (the CMI) and a simpler model based on the number of 4 key comorbidities (the 4C model) developed in this study can improve patient selection for CEA among patients with asymptomatic carotid stenosis in the VA. Five-year risk stratification scores should be incorporated into guidelines and routine practice to identify asymptomatic patients who are most likely to benefit from carotid revascularization.

Accepted for Publication: September 30, 2018.

Corresponding Author: Salomeh Keyhani, MD, MPH, Medical Service, San Francisco Veterans Affairs Medical Center, Box 111A1, 4150 Clement St, San Francisco, CA 94121 (salomeh.keyhani@va.gov).

Published Online: January 9, 2019. doi:10.1001/jamasurg.2018.5119

Author Contributions: Dr Keyhani had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Keyhani, Madden, Cheng, Johanning.

Acquisition, analysis, or interpretation of data: Keyhani, Madden, Cheng, Bravata, Halm, Austin, Ghasemiesfe, Abraham, Zhang.

Drafting of the manuscript: Keyhani, Madden, Ghasemiesfe, Abraham.

Critical revision of the manuscript for important intellectual content: Keyhani, Madden, Cheng, Bravata, Halm, Austin, Abraham, Zhang, Johanning.

Statistical analysis: Madden, Bravata.

Obtained funding: Keyhani, Cheng, Bravata.

Administrative, technical, or material support: Ghasemiesfe, Abraham, Zhang, Johanning.

Supervision: Keyhani, Halm, Johanning.

Conflict of Interest Disclosures: Dr Cheng reported having received grants from the National Institutes of Health (NIH) during before or during the conduct of the study. Dr Bravata reported having received grants from the NIH before or during the conduct of the study. Dr Johanning reported a patent to Frailty Assessment pending. No other disclosures were reported.

Funding/ Support: This project was supported by grant R01 HL114563-01A1 from the National Institute of Health and by Quality Enhancement Research Initiative grant 1IP1HX001994 from the Veterans Administration Health Services Research & Development.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Meeting Presentation: This study was presented at the 2018 Annual Meeting of the Association of VA Surgeons; May 8, 2018; Miami, Florida.