Analysis of a Machine Learning–Based Risk Stratification Scheme for Chronic Limb-Threatening Ischemia

Key Points

Question  At the time of chronic limb-threatening ischemia (CLTI) diagnosis, how are increasing patient risk, wound severity, and degree of atherosclerotic occlusions associated with CLTI-free survival?

Findings  This cohort study analyzed nested cohort data for 1238 patients from the Project of Ex Vivo Vein Graft Engineering via Transfection (PREVENT) III trial, the largest prospective randomized clinical trial evaluating the efficacy of edifoligide on outcomes in infrainguinal bypass for CLTI. The study identified 3 distinct clusters within the cohort, with 1-year CLTI-free survival rates of 82.3% for stage 1, 61.1% for stage 2, and 53.4% for stage 3.

Meaning  These findings suggest that, based on presenting characteristics, there are distinct clusters of patients with different 1-year CLTI-free survival; therefore, CLTI-free survival can be accurately and reproducibly quantified and may be used as a patient-centric outcome.

Importance  Valid risk stratification schemes are key to performing comparative effectiveness research; however, for chronic limb-threatening ischemia (CLTI), risk stratification schemes have limited efficacy. Improved, accurate, comprehensive, and reproducible risk stratification models for CLTI are needed.

Objective  To evaluate the use of topic model cluster analysis to generate an accurate risk prediction model for CLTI.

Design, Setting, and Participants  This multicenter, nested cohort study of existing Project of Ex Vivo Vein Graft Engineering via Transfection (PREVENT) III clinical trial data assessed data from patients undergoing infrainguinal vein bypass for the treatment of ischemic rest pain or ischemic tissue loss. Original data were collected from January 1, 2001, to December 31, 2003, and were analyzed in September 2021. All patients had 1 year of follow-up.

Exposures  Supervised topic model cluster analysis was applied to nested cohort data from the PREVENT III randomized clinical trial. Given a fixed number of clusters, the data were used to examine the probability that a patient belonged to each of the clusters and the distribution of the features within each cluster.

Main Outcomes and Measures  The primary outcome was 1-year CLTI-free survival, a composite of survival with remission of ischemic rest pain, wound healing, and freedom from major lower-extremity amputation without recurrent CLTI.

Results  Of the original 1404 patients, 166 were excluded because of a lack of sufficient feature and/or outcome data, leaving 1238 patients for analysis (mean [SD] age, 68.4 [11.2] years; 800 [64.6%] male; 894 [72.2%] White). The Society for Vascular Surgery Wound, Ischemia, and Foot Infection grade 2 wounds were present in 543 patients (43.8%), with rest pain present in 645 (52.1%). Three distinct clusters were identified within the cohort (130 patients in stage 1, 578 in stage 2, and 530 in stage 3), with 1-year CLTI-free survival rates of 82.3% (107 of 130 patients) for stage 1, 61.1% (353 of 578 patients) for stage 2, and 53.4% (283 of 530 patients) for stage 3. Stratified by stage, 1-year mortality was 10.0% (13 of 130 observed deaths in stage 1) for stage 1, 13.5% (78 of 578 patients) for stage 2, and 20.2% (105 of 521 patients) for stage 3. Similarly, stratifying by stage revealed major limb amputation rates of 4.2% (5 of 119 observed major limb amputations in stage 1) for stage 1, 10.8% (55 of 509 patients) for stage 2, and 18.4% (81 of 440 patients) for stage 3. Among survivors without a major amputation, the rates of CLTI recurrence were 9.2% (11 of 119 observed recurrences in stage 1) for stage 1, 24.9% (130 of 523 patients) for stage 2, and 29.6% (132 of 446 patients) for stage 3.

Conclusions and Relevance  The topic model cluster analysis in this cohort study identified 3 distinct stages within CLTI. Findings suggest that CLTI-free survival is an end point that can be accurately and reproducibly quantified and may be used as a patient-centric outcome.

Peripheral arterial disease is a common and debilitating manifestation of cardiovascular disease that impairs blood flow to the lower extremities. Chronic limb-threatening ischemia (CLTI) is the most severe manifestation of peripheral arterial disease and encompasses all patients with objective evidence of arterial insufficiency and symptoms of ischemic pain at rest and/or tissue loss (ulcerations, wounds, or gangrene). Required intensive medical therapy, revascularization, and/or wound management for patients with CLTI is complex, with highly variable patient responses and frequent reinterventions and even hospitalizations per year. Outcomes for patients with CLTI remain suboptimal despite evolving devices, medical therapeutics, and modern surgical techniques. Comparative effectiveness research (CER) is essential to iteratively improve data-driven outcomes in CLTI, yet the overwhelming heterogeneity of treatments and patient presentations in CLTI stymies CER by preventing objective outcome comparisons.^1,2 Patients and physicians have limited high-quality data to balance the competing mortality risk, limb threat, procedural invasiveness, hemodynamic efficacy, and longevity of a proposed procedure for a given patient with CLTI.^1,2

Although multiple researchers have developed baseline risk stratification schemes to enable CER in CLTI, none have been universally adopted.² All have modest predictive ability in CLTI, with model accuracy less than 72%.^1,2 No prior risk stratification model has been prospectively validated against large external data sets. End points of prior models are variable, often focusing on limb outcomes or survival. Few risk stratification schemes use modern computational advances, such as cluster analyses and/or machine learning to optimize model accuracy.^3,4 The Society for Vascular Surgery Wound, Ischemia, and Foot Infection (WIfI) score has been validated with regard to limb-based outcomes in retrospective cohorts.^5,6 Unfortunately, WIfI was designed to predict major amputation at only 1 year and is not understood with regard to mortality or other non–limb-specific composite outcomes.

Moreover, efficacy of infrainguinal bypass has been challenged over the past 2 decades because of the penetrance of endovascular devices and techniques.⁷ Indeed, training patterns suggest that experience with infrainguinal bypass and in particular infrageniculate bypasses, may be waning.⁸ No one is certain whether current outcomes reflect ideal infrainguinal bypass outcomes, and the comparisons between endovascular and infrainguinal bypass outcomes are potentially biased by this unmeasured confounding. Hence, it remains germane to model CLTI outcomes after infrainguinal bypass within a CLTI cohort in which the outcomes reflect what is truly achievable with infrainguinal bypass. Finally, only a small amount of literature focuses on patient-centric outcomes in CLTI. For example, outcomes analyses in CLTI focus frequently on amputation-free survival or patency. Although important, these outcomes fail to capture patient-centric outcomes, such as relief of ischemic rest pain and wound healing.

In this context, we aim to recapitulate our novel technique to generate a machine learning–based risk stratification scheme for CLTI within the nested cohort data from the previously published Project of Ex Vivo Vein Graft Engineering via Transfection (PREVENT) III trial.^4,9 We also aim to evaluate the ability to accurately and reproducibly use CLTI-free survival as an end point in CLTI outcomes research.

The PREVENT III randomized clinical trial comprised 1404 patients undergoing infrainguinal vein bypass for the treatment of ischemic rest pain or ischemic tissue loss.⁹ Patients participating in the trial, which was monitored by a data and safety monitoring board, provided written informed consent on enrolling in the trial. In our cohort, we included all patients who had complete trial enrollment and outcome data. Original data were collected from January 1, 2001, to December 31, 2003, and were analyzed in September 2021. All patients had 1 year of follow-up. Nested cohort data from the PREVENT III trial were anonymized before data analysis. Because of the retrospective nature of the analysis and impracticality of obtaining patient consent years after the closure of the clinical trial, the institutional review board at the University of North Carolina waived consent. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.¹⁰

The primary outcome measured was 1-year CLTI-free survival, which is a composite end point that includes patients who are alive with remission of CLTI symptoms (wound healing and/or freedom from ischemic pain), no ipsilateral major limb amputation, and no recurrence of ipsilateral CLTI at 1 year. Secondary outcomes are the individual components of this composite end point. We considered 1-year CLTI-free survival as a binary outcome, and we classified patients as having 1-year CLTI-free survival if they were alive, were free of major lower-extremity amputation, were free of recurrence, and had healed their index wound at 1 year after the index procedure. Specifically, mortality, major limb amputation, and recurrence of CLTI and their respective dates were recorded in the case report forms and used directly from the trial data set. For patients who had an index wound, we used the follow-up wound examination records to determine whether the index wound was healed. If the index wound was recorded as healed before or at 1 year after the index procedure, the wound was considered healed. If it was recorded as unhealed at 1 year after the index procedure, the wound was considered unhealed.

For the wound grade of the WIfI score, we used data from the baseline examination of the index leg. Records indicated the presence of gangrene and/or ulcers for each toe, the forefoot, the hindfoot, or other location. Free text responses to wound locations coded as other were recoded to the appropriate corresponding category (toe[s], forefoot, hindfoot, or leg). Wound scores were assigned as follows: wound grade 0, no index wound; wound grade 1, an ulcer was reported on only 1 toe; wound grade 2, an ulcer was reported on multiple toes OR gangrene was reported for 1 or more toes or the forefoot; wound grade 3, an ulcer was reported on the hindfoot or leg OR gangrene was reported for the forefoot, hindfoot, or leg; and highest score, multiple wounds (eg, a toe ulcer [grade 1] and a gangrenous hindfoot [grade 3]).

Demographic, comorbid, and CLTI-related characteristics for the cohort studied were described using medians (IQRs) for continuous variables and numbers (percentages) for categorical and binary variables. In accordance with the STROBE statement for reporting outcomes from observational studies, we do not include P value comparisons between the 2 groups in our description.¹⁰

Using the strategy developed in McGinigle et al,⁴ we used topic modeling from the natural language processing literature to cluster patients. Topic models are used to model the words in a corpus of documents to discover latent topics within those documents. In many cases, documents may have multiple topics; for example, movie reviews may be clustered by movie genre (eg, drama or horror) or by whether it was a positive review or not. Supervision is introduced to the model to guide topic formation so that topics are predictive of the desired response (eg, movie genre). In the context of our work, we considered our sample as a corpus of documents, with each individual in the sample as a document and the comorbid and CLTI-related features of each individual as the words in a document; supervision is provided by the outcome of interest, which in our case is CLTI-free survival.

To learn the latent topics or clusters in our sample, we specified a supervised latent Dirichlet allocation model,¹¹ which models responses, latent topics, documents, and words within documents. The generative model supposes that documents are generated from a set of latent topics and those topics induce an unknown distribution over the vocabulary. Thus, each topic has words that are more likely or less likely to appear in a document that arose from that topic. Translating from text documents to patients with CLTI, this means that patients in a particular cluster are more likely to share common features (common vocabulary), for example, wound grade and history of myocardial infarction, than patients in a different cluster. In addition to looking for common patterns, the latent clusters are formed with supervision from the patient outcome of 1-year CLTI-free survival. Specifically, CLTI-free survival enters the model as conditional on the proportion of features that are assigned to each cluster for the patient; we use a probit model, a type of generalized linear model for regression with binary outcomes. Given a fixed number of topics or clusters, we use the data to learn the probability that a patient belongs to each of the clusters and the distribution over the features within each cluster.

We slightly modified the model so that the regression for the response included demographic variables, such as sex and race, that were not included in the features used for topic formation, because demographic characteristics are likely to be associated with differential response owing to systemic disparities in health and health care but are not features on which we would want to use to characterize CLTI subtypes. To learn the model parameters, we implemented the collapsed Gibbs sampling approach described by Griffiths and Steyvers¹² and used the posterior samples of the topic assignments to calculate sufficient statistics of the model parameters.

The data were analyzed using K = 2,…,9 clusters. The number of clusters in the final model was 3 because there was a clear point of diminishing returns for adding more clusters, and clinician examination of the patient features associated with each cluster confirmed a practical clinical applicability. In addition, we used bayesian model selection to approximate the posterior probability of the observed data (features) given that the model in the study by Griffiths and Steyvers¹² and the model with K = 3 clusters yielded the highest posterior probability. After selecting the model with 3 clusters, the probabilities of patients belonging to a particular cluster were also computed. We considered patients to be in stage 1 if the probability of them being in stage 1 was higher than the probability of them being in stage 2 or stage 3 and similarly for stages 2 and 3. Clusters were characterized by the features with which they were most often associated and further characterized by analyzing the characteristics of the patients within them.

For this hypothesis-generating study, we emphasize that our model describes our cohort and provides insight into the feasibility of delineating limb-based and survival-based stages of CLTI. Based on guidance from the American Statistical Association, we did not conduct null hypothesis significance tests to compare the latent stages discovered as one may in a formal confirmatory study.^13-15 Analyses were performed using R, version 4.0.4 (R Foundation for Statistical Computing) and Julia, version 1.6 (MIT Lincoln Laboratory).

Of the 1404 patients originally randomized in PREVENT III, 166 patients were excluded because of a lack of significant feature and/or outcome data, leaving 1238 patients (mean [SD] age, 68.4 [11.3] years; 800 [64.6%] male; 10 [0.8%] Asian, 221 [17.9%] Black, 96 [7.8%] Hispanic, 894 [72.2%] White, and 17 [1.4%] other [no further detail available from the trial data set]) for analysis. The most prevalent comorbidities were hypercholesterolemia (680 patients [54.9%]) and diabetes (800 patients [64.6%]). With respect to presentation severity, 645 patients (52.1%) presented with rest pain, and 313 (25.3%) presented with a combination of rest pain and ischemic wounds. A total of 542 patients (43.8%) presented with a grade 2 wound as per the WIfI classification.⁵ Most bypasses (680 patients [54.9%]) were performed to a tibial artery target (Table 1).

Overall CLTI-free survival at 1 year was 60.0% (743 patients). We identified 3 clusters within the nested cohort (130 patients clustered in stage 1, 578 in stage 2, and 530 in stage 3). Results of our cluster analysis revealed 1-year CLTI-free survival as follows: 82.3% (107 of 130 patients) at stage 1, 61.1% (353 of 578 patients) at stage 2, and 53.4% (283 of 530 patients) at stage 3. Patients were considered to be in stage 1 if the probability of them being in stage 1 was higher than the probability of being stage 2 or stage 3 and similarly for stage 2 and stage 3. Of the 130 patients clustered into stage 1, 113 (86.9%) had an index wound. Of the 578 patients clustered into stage 2, 528 (91.4%) had an index wound. Among the 530 patients in stage 3, 501 (94.5%) had an index wound. Other significant differences between each stage are numerically quantified in Table 2 and graphically represented in Figure 1.

Comparisons to the overall population revealed that the 130 patients in stage 1 had the most variation from the overall pattern of patient features. Patients in stage 1 were more likely to have isolated femoropopliteal disease (with bypass targets more often to the above- or below-knee popliteal arteries), a history of smoking and hyperlipidemia, and smaller wounds. They were less likely to have diabetes, history of stroke or myocardial infarction, dialysis dependence, and grade 2 or 3 wounds.

Patients in stage 2 (n = 578) and stage 3 (n = 530) were more similar to the cohort pattern of features but had some key differences. Patients in stage 2 were more likely to be current smokers and have a more significant medical history, including renal disease, myocardial infarction, and stroke. Of interest, patients in stage 2 were also the most likely to have grade 3 wounds. Patients in stage 3 were more likely to have never smoked, but their peripheral arterial disease seems to be more driven by type 1 diabetes, kidney disease, and dialysis dependence. This finding correlates with known patterns of atherosclerotic disease, and patients in stage 3 were more likely than the cohort mean to undergo bypasses to tibial or pedal arteries. Patients in stage 3 were also more likely to present with grade 2 and 3 wounds. The prevalence of each feature by stage is graphically presented in Figure 2.

For the entire cohort, 1-year mortality was 16.0%. When stratified by stage, mortality was as follows: 10.0% (13 of 130 observed deaths in stage 1) for stage 1, 13.5% (78 of 576 patients) for stage 2, and 20.2% (105 of 521 patients) for stage 3. We note a difference in denominators for stages 2 and 3 in the calculation of mortality that differs from the denominator used for the calculation of CLTI-free survival. Mortality information was not available for 2 patients in stage 2 and 9 patients in stage 3, but because 1-year recurrence for these patients was recorded before death, they were included in the CLTI-free survival calculations. We indicate that our calculations are based on the total number of observed deaths in the stage with the word observed and use this convention for calculations for the other secondary outcomes. Similarly, stratifying by stage revealed the following major limb amputation rates: 4.2% (5 of 119 observed major limb amputations in stage 1) for stage 1, 10.8% (55 of 509 patients) for stage 2, 18.4% (81 of 440 patients) for stage 3. Of those who remained alive and with an intact limb, the rates of CLTI recurrence were 9.2% for stage 1 (11 of 1 119 patients [with 11 patients excluded]), 24.9% for stage 2 (130 of 2 523 patients [with 55 patients excluded]), and 29.6% for stage 3 3 (132 of 446 patients [with 84 patients excluded]).

In this cohort study, topic model cluster analysis showed the ability to discern 3 distinct clusters of patients within the nested cohort of the PREVENT III data. Chronic limb-threatening ischemia–free survival at 1 year by our modeled stages was 82.3% in stage 1; 61.1% in stage 2; and 53.4% in stage 3. This method may ultimately prove superior to other methods of risk stratification in CLTI because of its ability to incorporate a wider variety of features compared with other recent risk stratification schemes, such as WIfI. Moreover, topic model cluster analysis can adjust as the prevalence and severity of risk factors change over time within a particular disease state. The malleability of topic model cluster analysis may prove more useful than prior methods of generating risk scores to evaluate modern outcome data and quality improvement projects.

In contrast to the findings of prior publications,^3,16 wound and ischemic severity were not necessarily associated with worse outcomes, which intuitively makes sense, because not all patients with ischemic rest pain will have superior outcomes compared with those with ulcers or gangrene, and patients with less systemic disease may have better wound healing even for grade 2 and 3 wounds. This lack of strict association with ischemic and wound severity shows the importance of using CLTI-free survival as a composite end point. We should disclose that perfect discrimination between each wound grade cannot be performed retrospectively. Still, our data also show the novel promise of precisely measuring CLTI-free survival as a clinical end point. Although CLTI-free survival has been recently used by other authors,^17,18 to our knowledge, our study represents the largest to accurately use CLTI-free survival as the primary end point. This is significant because CLTI-free survival succinctly encapsulates patient-centric outcomes, such as wound healing, and alleviation of ischemic rest pain. Further external validation of our hypothesis-generating topic cluster model will be required to confirm our findings.

Prior literature does not include the relief of ischemic rest pain as a clinical outcome measure. Moreover, prior risk stratification schemes were not able to discern subpopulations with rest pain that may have worse outcomes even though, on average, patients with rest pain have improved outcomes compared with those with ischemic ulcerations. The use of topic model cluster analysis was likely able to mathematically distinguish the subpopulations of rest pain populations that may not have superior outcomes compared with those with minor tissue loss. The difference is how topic model cluster analysis identifies the subpopulations that may have a differential risk based on the totality of their risk factors than would otherwise be suggested by an individual clinical feature, such as rest pain.

Of interest, when splitting the primary end point, we found that patients in stage 1 had a 1-year mortality of 10.0% after infrainguinal bypass, which doubled to 20.2% in stage 3. The significance of this finding is that even though the mortality rate is high overall, there is a subpopulation of patients with CLTI who are expected to achieve significantly better survival. In this subpopulation of patients in stage 1, infrainguinal bypass is safe, and endovascular therapy may not mitigate morbidity and/or mortality because of other patient comorbidities. Our study does not incorporate data regarding technical feasibility of types of revascularization, which will be required in the future. In addition, it is unclear whether these outcomes are completely generalizable, because most trial participants were White and male. Still, it is important to have criterion-standard outcomes after infrainguinal bypass to compare at least within the scientific literature. This approach permits benchmarking of current outcomes, which will be particularly important when contextualizing the results of the Best Endovascular vs Best Surgical Therapy in Patients With Critical Limb Ischemia trial.⁷

There are several strengths to this study. The in-depth analysis of CLTI-free survival was long overdue and will contribute to a staging strategy to guide patients on treatment options based on outcomes more nuanced than just limb loss or death. Our novel machine learning and natural language processing methods represent a significant advance over prior literature regarding risk prediction and patient staging. This method that uses topic model cluster analysis can easily be applied to larger, more comprehensive data sets that also include specific vascular anatomy and target artery patency to further improve the predictive ability.

This research also has several limitations. A decade has passed since PREVENT III finished enrollment, and penetrance of wound care and optimal atherosclerotic medical care has changed over time, which may alter the expected outcomes of the clusters derived. Further delineation of specific distinguishing risk factors will be required also in analysis of other cohorts because the patients’ features in stages 2 and 3 were similar to those of the overall cohort mean with the exception of smoking history and dialysis dependence, which are the current differentiators between those stages. In addition, we did not have precise hemodynamic and infectious data, which limited our ability to calculate a complete WIfI score. Still, we were able to incorporate the wound severity, which is the most significant component of the WIfI score among patients who have undergone revascularization.³ The data were derived from a nested cohort of infrainguinal bypasses only; hence, we cannot extrapolate our results to populations undergoing endovascular and/or hybrid revascularizations.

This cohort study represents one of the largest and most comprehensive mathematically derived risk stratification schemes for CLTI. By topic model cluster analysis, 3 distinct stages of 1-year CLTI-free survival were identified. This study also showed that CLTI-free survival is an end point that can be accurately measured, which shows promise in improving patient-centric outcome assessments in CLTI.

Accepted for Publication: January 31, 2022.

Published: March 22, 2022. doi:10.1001/jamanetworkopen.2022.3424

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2022 Chung J et al. JAMA Network Open.

Corresponding Author: Katharine L. McGinigle, MD, MPH, Division of Vascular Surgery, Department of Surgery, University of North Carolina at Chapel Hill, 3021 Burnett-Womack Bldg, Campus Box 7212, Chapel Hill, NC 27599-7212 (katharine_mcginigle@med.unc.edu).

Author Contributions: Ms Freeman and Dr McGinigle had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Chung, Kosorok, Marston, Conte, McGinigle.

Acquisition, analysis, or interpretation of data: Chung, Freeman, McGinigle.

Drafting of the manuscript: Chung, Freeman, McGinigle.

Critical revision of the manuscript for important intellectual content: Chung, Kosorok, Marston, Conte, McGinigle.

Statistical analysis: Freeman, Kosorok, McGinigle.

Obtained funding: McGinigle.

Administrative, technical, or material support: Marston, Conte, McGinigle.

Supervision: Chung, Kosorok, McGinigle.

Conflict of Interest Disclosures: Dr Marston reported grants from Society for Vascular Surgery Foundation Award during the conduct of the study; personal fees from Boston Scientific and personal fees from Cook Medical outside the submitted work. No other disclosures were reported.

Funding/Support: The project was supported by grant UL1TR002489 from the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH) (Dr McGinigle).

Role of the Funder/Sponsor: Neither the NCATS nor the NIH participated in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.