Risk Prediction Tools to Improve Patient Selection for Carotid Endarterectomy Among Patients With Asymptomatic Carotid Stenosis | Cerebrovascular Disease | JAMA Surgery | JAMA Network
[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.  Calibration of the CMI vs the 4C Model
Calibration of the CMI vs the 4C Model

CMI indicates Carotid Mortality Index; 4C model, any cancer, chronic obstructive pulmonary disease, congestive heart failure, and chronic kidney disease.

Figure 2.  Estimated Probability of 5-Year Mortality Based on the CMI Binned Into 5 Groups
Estimated Probability of 5-Year Mortality Based on the CMI Binned Into 5 Groups

Estimated mortality by quintile of the Carotid Mortality Index (CMI). The number of patients is 2325, and the number of events is 687. Error bars indicate 95% CIs.

Table 1.  Characteristics of 2325 Veterans Who Received Carotid Endarterectomy in the Veterans Administration Between 2005 and 2010
Characteristics of 2325 Veterans Who Received Carotid Endarterectomy in the Veterans Administration Between 2005 and 2010
Table 2.  Univariate Predictors of 5-Year Mortality
Univariate Predictors of 5-Year Mortality
Table 3.  Comparison of 2 Multivariable Prediction Risk Models
Comparison of 2 Multivariable Prediction Risk Models
1.
Executive Committee for the Asymptomatic Carotid Atherosclerosis Study.  Endarterectomy for asymptomatic carotid artery stenosis.  JAMA. 1995;273(18):1421-1428. doi:10.1001/jama.1995.03520420037035PubMedGoogle ScholarCrossref
2.
Halliday  A, Mansfield  A, Marro  J,  et al; MRC Asymptomatic Carotid Surgery Trial (ACST) Collaborative Group.  Prevention of disabling and fatal strokes by successful carotid endarterectomy in patients without recent neurological symptoms: randomised controlled trial.  Lancet. 2004;363(9420):1491-1502. doi:10.1016/S0140-6736(04)16146-1PubMedGoogle ScholarCrossref
3.
Chaturvedi  S, Bruno  A, Feasby  T,  et al; Therapeutics and Technology Assessment Subcommittee of the American Academy of Neurology.  Carotid endarterectomy: an evidence-based review: report of the Therapeutics and Technology Assessment Subcommittee of the American Academy of Neurology.  Neurology. 2005;65(6):794-801. doi:10.1212/01.wnl.0000176036.07558.82PubMedGoogle ScholarCrossref
4.
Chaturvedi  S.  Has evidence changed practice? appropriateness of carotid endarterectomy after the clinical trials.  Neurology. 2007;69(2):224. doi:10.1212/01.wnl.0000271911.54805.13PubMedGoogle ScholarCrossref
5.
Halm  EA, Tuhrim  S, Wang  JJ, Rojas  M, Hannan  EL, Chassin  MR.  Has evidence changed practice? appropriateness of carotid endarterectomy after the clinical trials.  Neurology. 2007;68(3):187-194. doi:10.1212/01.wnl.0000251197.98197.e9PubMedGoogle ScholarCrossref
6.
Halm  EA, Tuhrim  S, Wang  JJ,  et al.  Racial and ethnic disparities in outcomes and appropriateness of carotid endarterectomy: impact of patient and provider factors.  Stroke. 2009;40(7):2493-2501. doi:10.1161/STROKEAHA.108.544866PubMedGoogle ScholarCrossref
7.
Murthy  VH, Krumholz  HM, Gross  CP.  Participation in cancer clinical trials: race-, sex-, and age-based disparities.  JAMA. 2004;291(22):2720-2726. doi:10.1001/jama.291.22.2720PubMedGoogle ScholarCrossref
8.
Krumholz  HM, Gross  CP, Peterson  ED,  et al.  Is there evidence of implicit exclusion criteria for elderly subjects in randomized trials? evidence from the GUSTO-1 study.  Am Heart J. 2003;146(5):839-847. doi:10.1016/S0002-8703(03)00408-3PubMedGoogle ScholarCrossref
9.
Masoudi  FA, Havranek  EP, Wolfe  P,  et al.  Most hospitalized older persons do not meet the enrollment criteria for clinical trials in heart failure.  Am Heart J. 2003;146(2):250-257. doi:10.1016/S0002-8703(03)00189-3PubMedGoogle ScholarCrossref
10.
Calvillo-King  L, Xuan  L, Zhang  S, Tuhrim  S, Halm  EA.  Predicting risk of perioperative death and stroke after carotid endarterectomy in asymptomatic patients: derivation and validation of a clinical risk score.  Stroke. 2010;41(12):2786-2794. doi:10.1161/STROKEAHA.110.599019PubMedGoogle ScholarCrossref
11.
Press  MJ, Chassin  MR, Wang  J, Tuhrim  S, Halm  EA.  Predicting medical and surgical complications of carotid endarterectomy: comparing the risk indexes.  Arch Intern Med. 2006;166(8):914-920. doi:10.1001/archinte.166.8.914PubMedGoogle ScholarCrossref
12.
Chaudhry  SA, Afzal  MR, Kassab  A, Hussain  SI, Qureshi  AI.  A new risk index for predicting outcomes among patients undergoing carotid endarterectomy in large administrative data sets.  J Stroke Cerebrovasc Dis. 2016;25(8):1978-1983. doi:10.1016/j.jstrokecerebrovasdis.2016.01.023PubMedGoogle ScholarCrossref
13.
Gupta  PK, Ramanan  B, Mactaggart  JN,  et al.  Risk index for predicting perioperative stroke, myocardial infarction, or death risk in asymptomatic patients undergoing carotid endarterectomy.  J Vasc Surg. 2013;57(2):318-326. doi:10.1016/j.jvs.2012.08.116PubMedGoogle ScholarCrossref
14.
Melin  AA, Schmid  KK, Lynch  TG,  et al.  Preoperative frailty Risk Analysis Index to stratify patients undergoing carotid endarterectomy.  J Vasc Surg. 2015;61(3):683-689. doi:10.1016/j.jvs.2014.10.009PubMedGoogle ScholarCrossref
15.
Conrad  MF, Kang  J, Mukhopadhyay  S, Patel  VI, LaMuraglia  GM, Cambria  RP.  A risk prediction model for determining appropriateness of CEA in patients with asymptomatic carotid artery stenosis.  Ann Surg. 2013;258(4):534-538.PubMedGoogle Scholar
16.
Wallaert  JB, Cronenwett  JL, Bertges  DJ,  et al; Vascular Study Group of New England.  Optimal selection of asymptomatic patients for carotid endarterectomy based on predicted 5-year survival.  J Vasc Surg. 2013;58(1):112-118. doi:10.1016/j.jvs.2012.12.056PubMedGoogle ScholarCrossref
17.
Carmo  M, Barbetta  I, Bissacco  D,  et al.  Development and validation of a score to predict life expectancy after carotid endarterectomy in asymptomatic patients.  J Vasc Surg. 2018;67(1):175-182. doi:10.1016/j.jvs.2017.05.107PubMedGoogle ScholarCrossref
18.
Mowery  DL, Chapman  BE, Conway  M,  et al.  Extracting a stroke phenotype risk factor from Veteran Health Administration clinical reports: an information content analysis.  J Biomed Semantics. 2016;7:26. doi:10.1186/s13326-016-0065-1PubMedGoogle ScholarCrossref
19.
Tirschwell  DL, Longstreth  WT  Jr.  Validating administrative data in stroke research.  Stroke. 2002;33(10):2465-2470. doi:10.1161/01.STR.0000032240.28636.BDPubMedGoogle ScholarCrossref
20.
U.S. Department of Veterans Affairs. The researcher’s guide to VA data: VA Information Research Center (VIReC). http://www.virec.research.va.gov/. Accessed September 19, 2011.
21.
DeMartino  RR, Brooke  BS, Neal  D,  et al.  Development of a validated model to predict 30-day stroke and 1-year survival after carotid endarterectomy for asymptomatic stenosis using the Vascular Quality Initiative.  J Vasc Surg. 2017;66(2):433-444.e2. doi:10.1016/j.jvs.2017.03.427PubMedGoogle ScholarCrossref
22.
Rassekh  SR, Lorenzi  M, Lee  L, Devji  S, McBride  M, Goddard  K.  Reclassification of ICD-9 codes into meaningful categories for oncology survivorship research.  J Cancer Epidemiol. 2010;2010:569517. doi:10.1155/2010/569517PubMedGoogle ScholarCrossref
23.
Austin  PC.  Bootstrap model selection had similar performance for selecting authentic and noise variables compared to backward variable elimination: a simulation study.  J Clin Epidemiol. 2008;61(10):1009-1017.e1. doi:10.1016/j.jclinepi.2007.11.014PubMedGoogle ScholarCrossref
24.
Austin  PC.  Using the bootstrap to improve estimation and confidence intervals for regression coefficients selected using backwards variable elimination.  Stat Med. 2008;27(17):3286-3300. doi:10.1002/sim.3104PubMedGoogle ScholarCrossref
25.
Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387. doi:10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4PubMedGoogle ScholarCrossref
26.
Austin  PC, Steyerberg  EW.  Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers.  Stat Med. 2014;33(3):517-535. doi:10.1002/sim.5941PubMedGoogle ScholarCrossref
27.
Go  C, Avgerinos  ED, Chaer  RA,  et al.  Long-term clinical outcomes and cardiovascular events after carotid endarterectomy.  Ann Vasc Surg. 2015;29(6):1265-1271. doi:10.1016/j.avsg.2015.03.031PubMedGoogle ScholarCrossref
28.
Jalbert  JJ, Nguyen  LL, Gerhard-Herman  MD,  et al.  Outcomes after carotid artery stenting in Medicare beneficiaries, 2005 to 2009.  JAMA Neurol. 2015;72(3):276-286. doi:10.1001/jamaneurol.2014.3638PubMedGoogle ScholarCrossref
29.
Hertzer  NR, Arison  R.  Cumulative stroke and survival ten years after carotid endarterectomy.  J Vasc Surg. 1985;2(5):661-668. doi:10.1016/0741-5214(85)90035-7PubMedGoogle ScholarCrossref
30.
Kragsterman  B, Björck  M, Lindbäck  J, Bergqvist  D, Pärsson  H; Swedish Vascular Registry (Swedvasc).  Long-term survival after carotid endarterectomy for asymptomatic stenosis.  Stroke. 2006;37(12):2886-2891. doi:10.1161/01.STR.0000248967.44015.88PubMedGoogle ScholarCrossref
31.
Eslami  MH, Rybin  D, Doros  G, Farber  A.  An externally validated robust risk predictive model of adverse outcomes after carotid endarterectomy.  J Vasc Surg. 2016;63(2):345-354. doi:10.1016/j.jvs.2015.09.003PubMedGoogle ScholarCrossref
32.
Yuo  TH, Sidaoui  J, Marone  LK, Makaroun  MS, Chaer  RA.  Revascularization of asymptomatic carotid stenosis is not appropriate in patients on dialysis.  J Vasc Surg. 2015;61(3):670-674. doi:10.1016/j.jvs.2014.10.002Google ScholarCrossref
33.
Lee  SJ, Lindquist  K, Segal  MR, Covinsky  KE.  Development and validation of a prognostic index for 4-year mortality in older adults.  JAMA. 2006;295(7):801-808. doi:10.1001/jama.295.7.801PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Association of VA Surgeons
    January 9, 2019

    Risk Prediction Tools to Improve Patient Selection for Carotid Endarterectomy Among Patients With Asymptomatic Carotid Stenosis

    Author Affiliations
    • 1Division of General Internal Medicine, Department of Medicine, University of California, San Francisco
    • 2Medical Service, San Francisco Veterans Affairs Medical Center, San Francisco, California
    • 3Northern California Institute of Research and Education, San Francisco
    • 4Department of Neurology, University of California, Los Angeles
    • 5VA Greater Los Angeles Healthcare System, Los Angeles, California
    • 6Department of Internal Medicine, Indiana University School of Medicine, Indianapolis
    • 7Department of Neurology, Indiana University School of Medicine, Indianapolis
    • 8Health Services Research and Development, Center for Health Information and Communication, Richard L. Roudebush VA Medical Center, Department of Veterans Affairs, Indianapolis, Indiana
    • 9Division of General Internal Medicine, UT Southwestern Medical Center, Dallas, Texas
    • 10Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
    • 11Visiting Scholar, Department of Medicine, University of California, San Francisco
    • 12Department of Surgery, University of Nebraska, Omaha
    • 13Omaha VA Medical Center, Omaha, Nebraska
    JAMA Surg. 2019;154(4):336-344. doi:10.1001/jamasurg.2018.5119
    Key Points

    Question  Which patients with asymptomatic carotid stenosis are unlikely to benefit from carotid endarterectomy?

    Findings  Among 2325 veterans in this cohort study that used Veterans Administration and Medicare data, a risk prediction tool (Carotid Mortality Index) based on 23 candidate variables and a simpler model based on the number of 4 key comorbidities (any cancer, chronic obstructive pulmonary disease, congestive heart failure, and chronic kidney disease [the 4C model]) were developed to identify patients at higher risk of 5-year mortality. The models were internally validated.

    Meaning  Study results suggest that both the Carotid Mortality Index and the 4C model may be used to inform clinicians whether a patient will live long enough to benefit from carotid endarterectomy.

    Abstract

    Importance  Randomized clinical trials have demonstrated that patients with asymptomatic carotid stenosis are eligible for carotid endarterectomy (CEA) if the 30-day surgical complication rate is less than 3% and the patient’s life expectancy is at least 5 years.

    Objective  To develop a risk prediction tool to improve patient selection for CEA among patients with asymptomatic carotid stenosis.

    Design, Setting, and Participants  In this cohort study, veterans 65 years and older who received both carotid imaging and CEA in the Veterans Administration between January 1, 2005, and December 31, 2009 (n = 2325) were followed up for 5 years. Data were analyzed from January 2005 to December 2015. A risk prediction tool (the Carotid Mortality Index [CMI]) based on 23 candidate variables identified in the literature was developed using Veterans Administration and Medicare data. A simpler model based on the number of 4 key comorbidities that were prevalent and strongly associated with 5-year mortality was also developed (any cancer in the past 5 years, chronic obstructive pulmonary disease, congestive heart failure, and chronic kidney disease [the 4C model]). Model performance was assessed using measures of discrimination (eg, area under the curve [AUC]) and calibration. Internal validation was performed by correcting for optimism using 500 bootstrapped samples.

    Main Outcome and Measure  Five-year mortality.

    Results  Among 2325 veterans, the mean (SD) age was 73.74 (5.92) years. The cohort was predominantly male (98.8%) and of white race/ethnicity (94.4%). Overall, 29.5% (n = 687) of patients died within 5 years of CEA. On the basis of a backward selection algorithm, 9 patient characteristics were selected (age, chronic kidney disease, diabetes, chronic obstructive pulmonary disease, any cancer diagnosis in the past 5 years, congestive heart failure, atrial fibrillation, remote stroke or transient ischemic attack, and body mass index) for the final logistic model, which yielded an optimism-corrected AUC of 0.687 for the CMI. The 4C model had slightly worse discrimination (AUC, 0.657) compared with the CMI model; however, the calibration curve was similar to the full model in most of the range of predicted probabilities.

    Conclusions and Relevance  According to results of this study, use of the CMI or the simpler 4C model may improve patient selection for CEA among patients with asymptomatic carotid stenosis.

    Introduction

    Carotid endarterectomy (CEA) is one of the most common types of major surgery in the United States and is 1 of only 2 primary prevention surgical procedures available. Randomized clinical trials1,2 published in 1995 and 2004 showed that, among carefully selected patients with asymptomatic carotid stenosis and experienced surgeons, the procedure reduced the risk of stroke and death compared with medical therapy alone.

    Revascularization with CEA is a trade-off between higher perioperative short-term risks in exchange for a lower long-term risk of stroke. Based on the aforementioned trials,1,2 guidelines concluded that candidate patients need to have a 30-day surgical complication rate less than 3% and a life expectancy of at least 5 years for the longer-term benefits of revascularization to outweigh the higher short-term risks of the procedure.3 Therefore, patient selection is a key factor in determining who benefits from intervention.4-6 In a randomized clinical trial, patients are excluded based on comorbid conditions; however, such exclusions may not be observed in routine practice.7-9 If in routine clinical practice patient selection does not conform to the original exclusion criteria outlined in the trials, patients may not live long enough to benefit from intervention. The development of risk prediction tools that can assist surgeons in identifying patients who are most likely to live long enough to benefit from CEA can improve patient outcomes.

    Multiple studies10-14 have developed risk prediction tools focused on reducing 30-day complication rates. Few studies15-17 have focused on developing prediction models to identify patients who will live long enough to benefit from revascularization. In this cohort study, we use national Veterans Administration (VA) and Medicare data to examine factors predictive of 5-year mortality among older veterans with asymptomatic carotid stenosis who underwent CEA.

    Methods
    Cohort

    We included all veterans who received both carotid imaging and CEA within 1 year of carotid imaging in the VA between January 1, 2005, and December 31, 2009. The dates of analysis were January 2005 to December 2015. We first identified 13 383 index carotid imaging tests among those 65 years and older performed in the VA between 2005 and 2009 using a previously developed natural language processing algorithm.18 This algorithm identified all veterans who had received carotid ultrasonography, magnetic resonance angiography, and computed tomographic angiography in the VA and who had at least 50% or moderate stenosis. We then manually reviewed the carotid imaging tests and excluded patients with less than 70% stenosis and patients with a stroke or transient ischemic attack in the prior 6 months. We then identified all veterans who had received CEA in the VA within 1 year of the carotid image (n = 2532). We excluded 181 veterans who had a stroke or transient ischemic attack between the imaging and CEA and excluded 26 patients who received a coronary artery bypass graft on the same day as carotid revascularization. The final cohort included 2325 veterans. These patients were followed up for 5 years from the date of CEA (2005-2015). This study was approved by the University of California, San Francisco Institutional Review Board.

    Outcome

    Our primary outcome was mortality due to any cause within 5 years of CEA. We also collected data on complications within 30 days of the procedure, including stroke, acute myocardial infarction, and death. We used the high-specificity algorithm by Tirschwell and Longstreth19 to identify strokes within 30 days. The VA Suicide Data Repository and the VA Vital Status Files were used to identify 30-day mortality.20 We identified 30-day acute myocardial infarction rates using the International Classification of Diseases, Ninth Revision (ICD-9) code 435.x on hospital discharges for admissions within 30 days of CEA.

    Candidate Predictor Variables

    Candidate predictor variables were selected based on review of published models in the literature.10,14-16,21 We identified previously published predictors of 1-year and 5-year survival.15,17,21 Because this literature was limited, we also included variables from a previously validated model of 30-day surgical complications that focused on frailty and function as predictors of 30-day surgical complications because these variables may be relevant to outcomes among older adults.14 We identified 23 candidate variables in the literature that were previously found to be predictive of either 30-day complication rates, 1-year mortality, or 5-year mortality as summarized in Table 1. Demographic variables, including age, sex, and race/ethnicity, were derived from VA and Medicare data. Data on comorbid conditions were constructed with ICD-9 codes using both VA and Medicare data. We required at least 2 outpatient visits or 1 inpatient visit with a diagnosis code to classify a patient as having a comorbid condition. Using a previously developed algorithm (eTable 1 in the Supplement), we identified all veterans who had a cancer diagnosis in the 5 years preceding CEA.22 Nonmetastatic prostate cancer was not included in this variable given the generally favorable prognosis and high prevalence of this condition among older men. The dementia variable was constructed using a combination of ICD-9 codes and data on use of medications (eg, donepezil hydrochloride) in the year before CEA. Data on medications were extracted from the VA Pharmacy Files. Data on body mass index were extracted from the Corporate Data Warehouse.20 Logical Observation Identifier Names and Codes (LOINC) were used to identify laboratory test data. We extracted data on kidney function, hemoglobin level, platelet count, and international normalized ratio from the Corporate Data Warehouse. We considered patients to have evidence of functional impairment if they were hemiplegic, had a pressure ulcer in the past year, were admitted to a nursing home, received home-based primary care, had a home health aide, or were dispensed a wheelchair, walker, or cane. Data on hemiplegia were extracted using ICD-9 codes, and data on pressure ulcers were extracted from the VA health factors files. Data on nursing home use and home health aide use were extracted from VA encounter files and VA Fee Basis files and Medicare data. Data on mobility devices were extracted from the VA prosthetics file and Medicare durable medical equipment files. Data on ejection fraction and pulmonary function were extracted from the medical record by trained abstractors (M.G., A.S.A., and A.J.Z.).

    Statistical Analysis

    We summarized categorical variables by frequency and percentage; continuous variables were summarized with means (SDs). We also present 5-year mortality curves, stratified by the count of the 4C model comorbidities, estimated as the complement of the Kaplan-Meier survival curves. We conducted a simple univariate analysis to examine the association of each candidate predictor variable with the odds of 5-year mortality. We also report the unadjusted estimates of the association between the count of the following 4 comorbidities with 5-year mortality: (1) any cancer in the past 5 years, (2) chronic obstructive pulmonary disease (COPD), (3) congestive heart failure (CHF), and (4) chronic kidney disease (CKD) with estimated glomerular filtration rate (eGFR) less than 60 mL/min/1.73 m2. Odds ratios (ORs) and 95% CIs derived from simple logistic regression of 5-year mortality with each of the predictors are listed in Table 2. For comorbid variables with data on level of severity (eg, CKD [eGFR], COPD [forced expiratory volume in the first second of expiration], and diabetes [insulin or no insulin]), we compared the discrimination (area under the curve [AUC]) and fit (Akaike information criterion and Nagelkerke R2) of the model that incorporated their dichotomously coded (yes or no) versions with those of the model that incorporated the noncollapsed version of the comorbid variable, and the best-performing version was retained in further analyses (Table 2). To reduce the number of predictors to a set small enough to be clinically expedient, we performed model selection using backward variable elimination (with a threshold of 2-sided P < .005).23 We also tested different versions of comorbid conditions (eg, cancer in the past 1 year vs cancer in the past 5 years) by allowing them to compete in the backward elimination process. All pairwise interactions between the 9 variables in the resulting model (age, chronic kidney disease, diabetes, chronic obstructive pulmonary disease, any cancer diagnosis in the past 5 years, congestive heart failure, atrial fibrillation, remote stroke or transient ischemic attack, and body mass index) were tested and retained if 2-sided P < .001. We calculated the AUC and Nagelkerke R2 for the resultant model in the full cohort. We then used a bootstrap approach to obtain optimism-corrected measures of model performance. To do so, we (1) applied all model-building steps described previously in each bootstrap sample, (2) assessed the performance of this model in the bootstrap sample and in the original data set, and (3) took the difference of the 2 estimates obtained in (2) to get an estimate of optimism. Finally, the mean of 500 bootstrapped estimates of optimism was subtracted from the initial (full cohort model) estimate of the AUC and Nagelkerke R2 to obtain the bootstrap optimism-corrected estimates of performance.

    Backward variable elimination is known to produce estimated regression coefficients that are biased and CIs that do not have a coverage probability of 95%. To improve estimation of regression coefficients and 95% CIs, we used the zero-corrected bootstrap (number of bootstrapped samples = 3000) model selection procedure. In a given bootstrap sample, predictor variables that are not selected for inclusion in the final regression model have their regression coefficient set to zero. Regression coefficients are averaged across the bootstrap samples, and nonparametric percentile bootstrap 95% CIs are then constructed for each regression coefficient.24

    Finally, a multivariable logistic model consisting only of age group and a count of the number of the 4 comorbidities (any cancer, COPD, CHF, and CKD) was created (the 4C model). The AUC and Nagelkerke R2 were calculated, and internal validation was performed by correcting for optimism using 500 bootstrapped samples.25

    To test whether the count of comorbidities was as good as allowing the individual comorbidity coefficient to vary, likelihood ratio tests comparing the 4C model (with the 4C model augmented with each comorbid condition separately) were performed. This analysis demonstrated that the simplifying assumption of equal weight for each comorbidity was reasonable for COPD, CHF, and CKD but not for cancer. However, the Nagelkerke R2 measure of model fit was similar for both the parsimonious and 4C models.

    To examine and compare calibration of the 2 risk models (Carotid Mortality Index [CMI] and 4C), the observed occurrence of mortality within 5 years was regressed against the model-predicted probabilities using loess smoothing.26 Optimism-corrected calibration curves (number of bootstrap samples = 500) and the 45-degree reference line represent ideal model fit and were overlaid on the same plot.

    A 5-year mortality CMI was created by multiplying the zero-corrected bootstrap regression coefficients by 10 and rounding, and then summing each patient score accordingly. The resulting score was approximately normally distributed, with a mean (SD) of 8.6 (8.5). The distribution of risk scores was divided into quintiles, and 5-year mortality (and 95% CI) for these 5 groups was estimated. We also present predicted probabilities and 95% CIs from the 4C model. All analyses were performed with statistical software (SAS Enterprise Guide, version 7.1; SAS Institute Inc).

    Results
    Cohort Characteristics

    In total, 2325 veterans were included in the cohort. The mean (SD) age of the cohort was 73.74 (5.92) years. The cohort was predominantly male (98.8%) and of white race/ethnicity (94.4%). The most prevalent comorbidities were coronary artery disease (53.5% [n = 1243]), CKD (49.5% [n = 1150]), diabetes (40.1% [n = 932]), COPD (27.3% [n = 635]), any cancer (25.9% [n = 603]), and CHF (17.6% [n = 409]). Approximately 10% (n = 242) of veterans who received CEA had functional impairment (Table 1).

    30-Day Complication Rate and 5-Year Mortality

    The 30-day complication rate (composite of stroke, acute myocardial infarction, and death) was 4.9% (n = 114), with a combined stroke and death rate of 3.0% (n = 70) (Table 1). Overall, 29.5% (n = 687) of patients died within 5 years of CEA. Veterans 80 years and older had an observed mortality rate of 42.7% (170 of 398) (Table 2 and eFigure 1 in the Supplement). Patients with 0, 1, 2, and 3 or 4 comorbidities had observed mortality rates of 18.8% (114 of 607), 24.6% (224 of 909), 37.4% (213 of 570), and 56.9% (136 of 239), respectively (Table 2 and eFigure 2 in the Supplement).

    Univariate Analyses

    Among the 23 candidate variables evaluated, 20 were nominally associated with 5-year survival. Factors most strongly associated with mortality within 5 years included the following: age 80 years and older (OR, 2.42; 95% CI, 1.87-3.10), CHF with ejection fraction of 35% or less (OR, 3.40; 95% CI, 2.50-4.70), CKD with eGFR of less than 30 mL/min/1.73 m2 (OR, 4.01; 95% CI, 2.60-6.10), dialysis in the past 3 months (OR, 10.86; 95% CI, 2.30-50.00), moderate or severe COPD (OR, 2.95; 95% CI, 2.20-3.90), diabetes taking insulin (OR, 1.81; 95% CI, 1.36-2.40), dementia (OR, 1.85; 95% CI, 1.33-2.60), atrial fibrillation (OR, 2.29; 95% CI, 1.78-3.00), underweight (OR, 4.64; 95% CI, 1.91-11.30), and hemoglobin level of less than 10 g/dL (OR, 3.58; 95% CI, 1.85-6.90) (Table 2) (to convert hemoglobin level to grams per liter, multiply by 10.0).

    Multivariable Models
    Carotid Mortality Index

    A risk prediction tool (CMI) based on 23 candidate variables identified in the literature was developed using VA and Medicare data. On the basis of our backward selection algorithm, we selected 9 patient characteristics (age, CKD, diabetes, COPD, any cancer diagnosis in the past 5 years, CHF, atrial fibrillation, remote stroke or transient ischemic attack, and body mass index) for the final logistic model for 5-year mortality. The final model had an AUC of 0.706. These 9 variables represent 97.5% of the potential AUC if all 23 candidate variables had been included in the model. Internal validation by bootstrapping analysis demonstrated an optimism-corrected AUC of 0.687 for the CMI (Table 3). A nomogram is provided in eTable 2 in the Supplement. An online calculator is also available (https://is.gd/CEA_Risk_Calculator).

    4C Model

    The interaction between age and the number of comorbidities was not statistically significant. The count of the 4C model comorbidities was strongly associated with 5-year mortality, with the number of comorbidities demonstrating increasing odds of 5-year mortality with increasing comorbidity as follows: 1 (OR, 1.31; 95% CI, 1.02-1.70), 2 (OR, 2.33; 95% CI, 1.78-3.10), and 3 or 4 (OR, 5.22; 95% CI, 3.70-7.30). The model AUC was 0.661, and the optimism-corrected AUC was 0.657 (Table 3). The 4C model had slightly worse discrimination compared with the CMI model; however, the calibration curve was similar to the CMI model in most of the range of predicted probabilities.

    Comparison of Model Performance

    The 4C model had slightly worse fit (Table 3) than the CMI. However, calibration was as good as for the CMI for most of the range of predicted probabilities (Figure 1), with both models underestimating risk at higher levels (ie, calibration curves lay above the perfect-fit line).

    Predicted Probabilities of 5-Year Mortality Based on the CMI and the 4C Model

    Figure 2 shows the estimated probability of 5-year mortality based on quintiles of the CMI. eFigure 3 in the Supplement shows the estimated probability of 5-year mortality based on age and the number of comorbidities using the 4C model.

    Discussion

    In this cohort study that used VA and Medicare data, 29.5% (687 of 2325) of veterans 65 years and older who received CEA died within 5 years. Outcomes among older veterans are similar to those of patients with Medicare coverage and patients revascularized in other health systems.27-30 Our data suggest that health systems can improve patient selection to ensure that patients will achieve a net benefit from carotid revascularization. We developed both simple and more complex risk prediction tools that can be used to improve patient selection for CEA in the VA.

    Our study replicates and builds on the findings of 2 other published 5-year risk prediction models in the literature. One was developed in a sample of 4114 patients with asymptomatic carotid stenosis cared for by the Vascular Study Group of New England.16 This model found that age, diabetes, CHF, COPD, renal function, dialysis dependency, severity of contralateral stenosis, and statin use were associated with 5-year mortality. The other, an analysis of both 1-year and 5-year outcomes using the CEA Vascular Quality Initiative (VQI) registry, found that age, diabetes, coronary artery disease, COPD, renal function, dialysis dependency, and absence of statin use were associated with mortality.21,31 Contralateral stenosis was not evaluated in the VQI-derived model. In addition, patients whose data were included in the VQI registry who had evidence of a major medical contraindication (CHF class III/IV, left ventricular ejection fraction <30%, unstable angina, recent acute myocardial infarction, or COPD with forced expiratory volume in the first second of expiration <30%) were not eligible for CEA. Therefore, it appears that the severity of disease in the VQI population was different from that of our cohort, although the mean age was similar. We confirmed that all variables identified in the VQI model21,31 and by the Vascular Study Group of New England16 were important predictors of 5-year survival. However, our final model included several factors that were not evaluated in these articles. For example, the model based on the VQI registry did not include CHF as a variable, and neither study included cancer. Both of these variables contributed to model performance and were included in our final model. Our model did not include dialysis as a factor because only 11 patients in our sample were on dialysis. Our data also confirm prior work32 suggesting that patients on dialysis are not appropriate candidates for CEA, with 5-year mortality herein among those with dialysis in the past 3 months of 81.8% (9 of 11) (Table 2). The present study builds on the prior literature and includes all relevant variables identified in this earlier work.

    This emerging body of research16,17 suggests that these risk prediction models should be used to improve patient selection for CEA. The CMI was developed based on a population of veterans with carotid stenosis who received CEA in the VA and is especially informative to practice in the VA. General mortality models (eg, the Lee Index33) have been developed using data from the US adult population and can also be incorporated into assessments regarding the benefits of CEA among patients with asymptomatic disease. The decision to operate could be individualized with use of multiple different mortality calculators15-17,33 that have been developed in recent years. However, at this juncture, it is unclear what 5-year mortality risk is acceptable from a public health perspective to ensure benefit from revascularization for an individual patient. Given that multiple risk scores developed in different populations have found comparable results, risk scores estimating 5-year mortality should be incorporated into updated guidelines on the management of patients with asymptomatic carotid stenosis. Expert panels should weigh in on what constitutes an acceptable risk of 5-year mortality before consideration of surgery and provide guidance to clinicians on how many patients should be exposed to a potentially unnecessary surgical procedure to prevent 1 stroke. Five-year survival among participants who received medical therapy in asymptomatic carotid stenosis trials could offer a basis for guidelines and provide the outer bounds of an acceptable 5-year mortality range to target using risk prediction tools.

    To facilitate implementation clinically, we also developed a simpler model that can be easily remembered in clinical practice (any cancer in the past 5 years, COPD, CHF, and CKD [the 4C model]). This model, based on the number of 4 key comorbidities that were prevalent and strongly associated with 5-year mortality, demonstrated that patients with multiple 4C model comorbidities are less likely to benefit from revascularization. While the model is limited by equally weighting any cancer diagnosis, COPD, CHF, and CKD, it is easy to remember and provides a quick assessment of potential 5-year survival.

    Limitations

    Our study has several limitations that deserve comment. First, we did not include patient self-reported assessments of function because these data are unavailable in the VA electronic medical record. We used measures that were suggestive of poor function in the medical record, such as a recent pressure ulcer, recent admission to a nursing home, and use of mobility devices. This may be the reason why functional status was not retained in our final model, although other investigators have found it to be an important predictor of survival.33 Second, similar to other models,33 we found that cancer predicts poor survival. Given our sample size, we could not determine which cancers were the most important to consider and combined all types of cancer. The results of our study and other studies suggest that cancer diagnosis is important. A patient’s cancer type and overall prognosis should be weighed carefully in individual decision making. Third, we used backward selection to select predictor variables for inclusion in the final model; however, the candidate predictor variables with which we began were not selected based on convenience or availability but were selected based on prior research demonstrating that these factors were important in other 5-year mortality models. Fourth, the models in this work underwent only internal validation. However, we used computer-intensive methods to adjust our estimates of model performance for optimism. Future work should focus on external validation in a more recent period in the VA. Fifth, models derived in older male veterans may not be generalizable to younger veterans or the general adult population; however, CEA is more commonly performed among older men. Therefore, while the models are limited in generalizability, they are still instructive.

    Conclusions

    Both a risk prediction tool based on 9 variables (the CMI) and a simpler model based on the number of 4 key comorbidities (the 4C model) developed in this study can improve patient selection for CEA among patients with asymptomatic carotid stenosis in the VA. Five-year risk stratification scores should be incorporated into guidelines and routine practice to identify asymptomatic patients who are most likely to benefit from carotid revascularization.

    Back to top
    Article Information

    Accepted for Publication: September 30, 2018.

    Corresponding Author: Salomeh Keyhani, MD, MPH, Medical Service, San Francisco Veterans Affairs Medical Center, Box 111A1, 4150 Clement St, San Francisco, CA 94121 (salomeh.keyhani@va.gov).

    Published Online: January 9, 2019. doi:10.1001/jamasurg.2018.5119

    Author Contributions: Dr Keyhani had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Keyhani, Madden, Cheng, Johanning.

    Acquisition, analysis, or interpretation of data: Keyhani, Madden, Cheng, Bravata, Halm, Austin, Ghasemiesfe, Abraham, Zhang.

    Drafting of the manuscript: Keyhani, Madden, Ghasemiesfe, Abraham.

    Critical revision of the manuscript for important intellectual content: Keyhani, Madden, Cheng, Bravata, Halm, Austin, Abraham, Zhang, Johanning.

    Statistical analysis: Madden, Bravata.

    Obtained funding: Keyhani, Cheng, Bravata.

    Administrative, technical, or material support: Ghasemiesfe, Abraham, Zhang, Johanning.

    Supervision: Keyhani, Halm, Johanning.

    Conflict of Interest Disclosures: Dr Cheng reported having received grants from the National Institutes of Health (NIH) during before or during the conduct of the study. Dr Bravata reported having received grants from the NIH before or during the conduct of the study. Dr Johanning reported a patent to Frailty Assessment pending. No other disclosures were reported.

    Funding/ Support: This project was supported by grant R01 HL114563-01A1 from the National Institute of Health and by Quality Enhancement Research Initiative grant 1IP1HX001994 from the Veterans Administration Health Services Research & Development.

    Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Meeting Presentation: This study was presented at the 2018 Annual Meeting of the Association of VA Surgeons; May 8, 2018; Miami, Florida.

    References
    1.
    Executive Committee for the Asymptomatic Carotid Atherosclerosis Study.  Endarterectomy for asymptomatic carotid artery stenosis.  JAMA. 1995;273(18):1421-1428. doi:10.1001/jama.1995.03520420037035PubMedGoogle ScholarCrossref
    2.
    Halliday  A, Mansfield  A, Marro  J,  et al; MRC Asymptomatic Carotid Surgery Trial (ACST) Collaborative Group.  Prevention of disabling and fatal strokes by successful carotid endarterectomy in patients without recent neurological symptoms: randomised controlled trial.  Lancet. 2004;363(9420):1491-1502. doi:10.1016/S0140-6736(04)16146-1PubMedGoogle ScholarCrossref
    3.
    Chaturvedi  S, Bruno  A, Feasby  T,  et al; Therapeutics and Technology Assessment Subcommittee of the American Academy of Neurology.  Carotid endarterectomy: an evidence-based review: report of the Therapeutics and Technology Assessment Subcommittee of the American Academy of Neurology.  Neurology. 2005;65(6):794-801. doi:10.1212/01.wnl.0000176036.07558.82PubMedGoogle ScholarCrossref
    4.
    Chaturvedi  S.  Has evidence changed practice? appropriateness of carotid endarterectomy after the clinical trials.  Neurology. 2007;69(2):224. doi:10.1212/01.wnl.0000271911.54805.13PubMedGoogle ScholarCrossref
    5.
    Halm  EA, Tuhrim  S, Wang  JJ, Rojas  M, Hannan  EL, Chassin  MR.  Has evidence changed practice? appropriateness of carotid endarterectomy after the clinical trials.  Neurology. 2007;68(3):187-194. doi:10.1212/01.wnl.0000251197.98197.e9PubMedGoogle ScholarCrossref
    6.
    Halm  EA, Tuhrim  S, Wang  JJ,  et al.  Racial and ethnic disparities in outcomes and appropriateness of carotid endarterectomy: impact of patient and provider factors.  Stroke. 2009;40(7):2493-2501. doi:10.1161/STROKEAHA.108.544866PubMedGoogle ScholarCrossref
    7.
    Murthy  VH, Krumholz  HM, Gross  CP.  Participation in cancer clinical trials: race-, sex-, and age-based disparities.  JAMA. 2004;291(22):2720-2726. doi:10.1001/jama.291.22.2720PubMedGoogle ScholarCrossref
    8.
    Krumholz  HM, Gross  CP, Peterson  ED,  et al.  Is there evidence of implicit exclusion criteria for elderly subjects in randomized trials? evidence from the GUSTO-1 study.  Am Heart J. 2003;146(5):839-847. doi:10.1016/S0002-8703(03)00408-3PubMedGoogle ScholarCrossref
    9.
    Masoudi  FA, Havranek  EP, Wolfe  P,  et al.  Most hospitalized older persons do not meet the enrollment criteria for clinical trials in heart failure.  Am Heart J. 2003;146(2):250-257. doi:10.1016/S0002-8703(03)00189-3PubMedGoogle ScholarCrossref
    10.
    Calvillo-King  L, Xuan  L, Zhang  S, Tuhrim  S, Halm  EA.  Predicting risk of perioperative death and stroke after carotid endarterectomy in asymptomatic patients: derivation and validation of a clinical risk score.  Stroke. 2010;41(12):2786-2794. doi:10.1161/STROKEAHA.110.599019PubMedGoogle ScholarCrossref
    11.
    Press  MJ, Chassin  MR, Wang  J, Tuhrim  S, Halm  EA.  Predicting medical and surgical complications of carotid endarterectomy: comparing the risk indexes.  Arch Intern Med. 2006;166(8):914-920. doi:10.1001/archinte.166.8.914PubMedGoogle ScholarCrossref
    12.
    Chaudhry  SA, Afzal  MR, Kassab  A, Hussain  SI, Qureshi  AI.  A new risk index for predicting outcomes among patients undergoing carotid endarterectomy in large administrative data sets.  J Stroke Cerebrovasc Dis. 2016;25(8):1978-1983. doi:10.1016/j.jstrokecerebrovasdis.2016.01.023PubMedGoogle ScholarCrossref
    13.
    Gupta  PK, Ramanan  B, Mactaggart  JN,  et al.  Risk index for predicting perioperative stroke, myocardial infarction, or death risk in asymptomatic patients undergoing carotid endarterectomy.  J Vasc Surg. 2013;57(2):318-326. doi:10.1016/j.jvs.2012.08.116PubMedGoogle ScholarCrossref
    14.
    Melin  AA, Schmid  KK, Lynch  TG,  et al.  Preoperative frailty Risk Analysis Index to stratify patients undergoing carotid endarterectomy.  J Vasc Surg. 2015;61(3):683-689. doi:10.1016/j.jvs.2014.10.009PubMedGoogle ScholarCrossref
    15.
    Conrad  MF, Kang  J, Mukhopadhyay  S, Patel  VI, LaMuraglia  GM, Cambria  RP.  A risk prediction model for determining appropriateness of CEA in patients with asymptomatic carotid artery stenosis.  Ann Surg. 2013;258(4):534-538.PubMedGoogle Scholar
    16.
    Wallaert  JB, Cronenwett  JL, Bertges  DJ,  et al; Vascular Study Group of New England.  Optimal selection of asymptomatic patients for carotid endarterectomy based on predicted 5-year survival.  J Vasc Surg. 2013;58(1):112-118. doi:10.1016/j.jvs.2012.12.056PubMedGoogle ScholarCrossref
    17.
    Carmo  M, Barbetta  I, Bissacco  D,  et al.  Development and validation of a score to predict life expectancy after carotid endarterectomy in asymptomatic patients.  J Vasc Surg. 2018;67(1):175-182. doi:10.1016/j.jvs.2017.05.107PubMedGoogle ScholarCrossref
    18.
    Mowery  DL, Chapman  BE, Conway  M,  et al.  Extracting a stroke phenotype risk factor from Veteran Health Administration clinical reports: an information content analysis.  J Biomed Semantics. 2016;7:26. doi:10.1186/s13326-016-0065-1PubMedGoogle ScholarCrossref
    19.
    Tirschwell  DL, Longstreth  WT  Jr.  Validating administrative data in stroke research.  Stroke. 2002;33(10):2465-2470. doi:10.1161/01.STR.0000032240.28636.BDPubMedGoogle ScholarCrossref
    20.
    U.S. Department of Veterans Affairs. The researcher’s guide to VA data: VA Information Research Center (VIReC). http://www.virec.research.va.gov/. Accessed September 19, 2011.
    21.
    DeMartino  RR, Brooke  BS, Neal  D,  et al.  Development of a validated model to predict 30-day stroke and 1-year survival after carotid endarterectomy for asymptomatic stenosis using the Vascular Quality Initiative.  J Vasc Surg. 2017;66(2):433-444.e2. doi:10.1016/j.jvs.2017.03.427PubMedGoogle ScholarCrossref
    22.
    Rassekh  SR, Lorenzi  M, Lee  L, Devji  S, McBride  M, Goddard  K.  Reclassification of ICD-9 codes into meaningful categories for oncology survivorship research.  J Cancer Epidemiol. 2010;2010:569517. doi:10.1155/2010/569517PubMedGoogle ScholarCrossref
    23.
    Austin  PC.  Bootstrap model selection had similar performance for selecting authentic and noise variables compared to backward variable elimination: a simulation study.  J Clin Epidemiol. 2008;61(10):1009-1017.e1. doi:10.1016/j.jclinepi.2007.11.014PubMedGoogle ScholarCrossref
    24.
    Austin  PC.  Using the bootstrap to improve estimation and confidence intervals for regression coefficients selected using backwards variable elimination.  Stat Med. 2008;27(17):3286-3300. doi:10.1002/sim.3104PubMedGoogle ScholarCrossref
    25.
    Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387. doi:10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4PubMedGoogle ScholarCrossref
    26.
    Austin  PC, Steyerberg  EW.  Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers.  Stat Med. 2014;33(3):517-535. doi:10.1002/sim.5941PubMedGoogle ScholarCrossref
    27.
    Go  C, Avgerinos  ED, Chaer  RA,  et al.  Long-term clinical outcomes and cardiovascular events after carotid endarterectomy.  Ann Vasc Surg. 2015;29(6):1265-1271. doi:10.1016/j.avsg.2015.03.031PubMedGoogle ScholarCrossref
    28.
    Jalbert  JJ, Nguyen  LL, Gerhard-Herman  MD,  et al.  Outcomes after carotid artery stenting in Medicare beneficiaries, 2005 to 2009.  JAMA Neurol. 2015;72(3):276-286. doi:10.1001/jamaneurol.2014.3638PubMedGoogle ScholarCrossref
    29.
    Hertzer  NR, Arison  R.  Cumulative stroke and survival ten years after carotid endarterectomy.  J Vasc Surg. 1985;2(5):661-668. doi:10.1016/0741-5214(85)90035-7PubMedGoogle ScholarCrossref
    30.
    Kragsterman  B, Björck  M, Lindbäck  J, Bergqvist  D, Pärsson  H; Swedish Vascular Registry (Swedvasc).  Long-term survival after carotid endarterectomy for asymptomatic stenosis.  Stroke. 2006;37(12):2886-2891. doi:10.1161/01.STR.0000248967.44015.88PubMedGoogle ScholarCrossref
    31.
    Eslami  MH, Rybin  D, Doros  G, Farber  A.  An externally validated robust risk predictive model of adverse outcomes after carotid endarterectomy.  J Vasc Surg. 2016;63(2):345-354. doi:10.1016/j.jvs.2015.09.003PubMedGoogle ScholarCrossref
    32.
    Yuo  TH, Sidaoui  J, Marone  LK, Makaroun  MS, Chaer  RA.  Revascularization of asymptomatic carotid stenosis is not appropriate in patients on dialysis.  J Vasc Surg. 2015;61(3):670-674. doi:10.1016/j.jvs.2014.10.002Google ScholarCrossref
    33.
    Lee  SJ, Lindquist  K, Segal  MR, Covinsky  KE.  Development and validation of a prognostic index for 4-year mortality in older adults.  JAMA. 2006;295(7):801-808. doi:10.1001/jama.295.7.801PubMedGoogle ScholarCrossref
    ×