Development and Validation of a Risk Stratification Model Using Disease Severity Hierarchy for Mortality or Major Cardiovascular Event

Key Points Question Does incorporating clinical domain knowledge regarding diseases, disease severity, and treatment pathways into machine learning improve risk stratification? Findings In this retrospective cohort study involving 51 969 patients, a new representation of patient data was developed and used to train machine learning models to predict mortality and major cardiovascular events. Results showed substantial improvement in prediction performance compared with traditional patient data representation methods. Meaning The findings of this study suggest that methods that can extract and represent the clinical knowledge contained in electronic medical records should be incorporated into machine learning models for use in clinical decision support systems.

This supplementary material has been provided by the authors to give readers additional information about their work.

eTable. Major Cardiovascular Event Internal and External Validation
Comparing the predictive ability of the different patient representations (Models) in discriminating MCE at age-time points 60, 65, 75, and 80 years. CPH models are trained for each representation through 10-fold cross-validation on the REP data and external validation on the FHS data. Data are expressed as mean (95% CI). The best AUC performance values are highlighted if bold font.   An example of a 4-level DSH tree and corresponding risk score allocation for Obesity. At the root node, a patient is considered obese if the patients' BMI was ≥ 30 kg/m 2 . Several different obesity control levels can be considered and risk scores allocated accordingly.

Internal
eFigure 5: Predict all-cause mortality (ACM) at age 80 years. Internal validation performance plots for the CPH models in predicting ACM at age 80 years. The blue, red and green curves represent the performance of DSH-RS, COM, and COM+LB/VS respectively. The AUC and Gini index of DSH-RS are significantly greater than that of COM and COM + LB/VS. Similarly, the ROC, gain, net benefit, sensitivity and positive predictive value curves for the DSH-RS model are significantly higher than those for COM and COM + LB/VS. The class distributions (Yes = dead at age 80, No = survive beyond 80 years) provided by the predicted density plot for DSH-RS are well separated, showing that DSH-RS accurately discriminates between low and high-risk patients compared to COM.
The Gini score was computed by dividing the area between the gain curve and the random classifier (gray diagonal line) by the area between the perfect classifier (purple curve) and the random classifier. The dotted diagonal line in the calibration plot represents the line of perfect calibration.

eFigure 7. Predicted Major Cardiovascular Event at Age 60 Years
Internal validation performance plots for the CPH models in predicting MCE at age 60 years. The blue, red and green curves represent the performance of DSH-RS, COM, and COM+LB/VS respectively. The AUC and Gini index of DSH-RS are significantly greater than that of COM and COM + LB/VS. Similarly, the ROC, gain, net benefit, sensitivity and positive predictive value curves for the DSH-RS model are all higher than those for COM and COM + LB/VS. The class distributions (Yes = MCE at age 60, No = no MCE beyond 60 years) for both DSH-RS and COM show some degree of overlap, however, the degree of overlap is very significant for COM. This illustrated that DSH-RS can more accurately discriminates between low and high-risk patients compared to COM.
The Gini score was computed by dividing the area between the gain curve and the random classifier (gray diagonal line) by the area between the perfect classifier (purple curve) and the random classifier. The dotted diagonal line in the calibration plot represents the line of perfect calibration.