[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.
Distribution of Patient Risk Scores and Probability of 1-Year Major Cardiovascular Events by Risk Score
Distribution of Patient Risk Scores and Probability of 1-Year Major Cardiovascular Events by Risk Score

The histograms show the distribution of risk scores for training (A), test (B), and validation (C) samples, and the curves show the probability of major cardiovascular events at 1 year. The black curve represents a fitted density curve on the risk score histogram. A risk score, ranging from 0 to 100, was constructed at the patient level based on the regression coefficients estimated from the final risk model with the training sample. A higher risk score indicates a higher probability of major cardiovascular events 1 year after discharge.

Figure 2.
Risk Stratification by Risk Scores
Risk Stratification by Risk Scores

For the training, test, and validation samples, respectively, the highest risk group includes 11.3%, 12.1%, and 11.7% of the patients; the average risk group includes 81.0%, 81.8%, and 81.1% of the patients; and the lowest risk group includes 7.7%, 6.1%, and 7.2% of the patients.

Figure 3.
Observed Probability of Being Free From 1-Year Major Cardiovascular Events by Risk Groups
Observed Probability of Being Free From 1-Year Major Cardiovascular Events by Risk Groups
Table 1.  
Model-Selected Patient Risk Factors by Training, Test, and Validation Samples
Model-Selected Patient Risk Factors by Training, Test, and Validation Samples
Table 2.  
Final Risk Prediction Model for 1-Year Major Cardiovascular Events After Discharge for Acute Myocardial Infarction Based on Training Sample
Final Risk Prediction Model for 1-Year Major Cardiovascular Events After Discharge for Acute Myocardial Infarction Based on Training Sample
1.
Tu  JV, Austin  PC, Walld  R, Roos  L, Agras  J, McDonald  KM.  Development and validation of the Ontario acute myocardial infarction mortality prediction rules.  J Am Coll Cardiol. 2001;37(4):992-997. doi:10.1016/S0735-1097(01)01109-3PubMedGoogle ScholarCrossref
2.
Smolderen  KG, Buchanan  DM, Gosch  K,  et al.  Depression treatment and 1-year mortality following acute myocardial infarction: insights from the TRIUMPH registry (Translational Research Investigating Underlying Disparities in Acute Myocardial Infarction Patients’ Health Status).  Circulation. 2017;135(18):1681-1689. doi:10.1161/CIRCULATIONAHA.116.025140PubMedGoogle ScholarCrossref
3.
Plakht  Y, Shiyovich  A, Gilutz  H.  Predictors of long-term (10-year) mortality postmyocardial infarction: age-related differences. Soroka Acute Myocardial Infarction (SAMI) Project.  J Cardiol. 2015;65(3):216-223. doi:10.1016/j.jjcc.2014.06.001PubMedGoogle ScholarCrossref
4.
Ketchum  ES, Dickstein  K, Kjekshus  J,  et al.  The Seattle Post Myocardial Infarction Model (SPIM): prediction of mortality after acute myocardial infarction with left ventricular dysfunction.  Eur Heart J Acute Cardiovasc Care. 2014;3(1):46-55. doi:10.1177/2048872613502283PubMedGoogle ScholarCrossref
5.
Chaudhry  SI, Khan  RF, Chen  J,  et al.  National trends in recurrent AMI hospitalizations 1 year after acute myocardial infarction in Medicare beneficiaries: 1999-2010.  J Am Heart Assoc. 2014;3(5):e001197. doi:10.1161/JAHA.114.001197PubMedGoogle ScholarCrossref
6.
Wang  Y, Lichtman  JH, Dharmarajan  K,  et al.  National trends in stroke after acute myocardial infarction among Medicare patients in the United States: 1999 to 2010.  Am Heart J. 2015;169(1):78-85.e4. doi:10.1016/j.ahj.2014.06.011PubMedGoogle ScholarCrossref
7.
Cahill  TJ, Kharbanda  RK.  Heart failure after myocardial infarction in the era of primary percutaneous coronary intervention: mechanisms, incidence and identification of patients at risk.  World J Cardiol. 2017;9(5):407-415. doi:10.4330/wjc.v9.i5.407PubMedGoogle ScholarCrossref
8.
Sampson  UK, Pfeffer  MA, McMurray  JJ, Lokhnygina  Y, White  HD, Solomon  SD; VALIANT Trial Investigators.  Predictors of stroke in high-risk patients after acute myocardial infarction: insights from the VALIANT trial.  Eur Heart J. 2007;28(6):685-691. doi:10.1093/eurheartj/ehl197PubMedGoogle ScholarCrossref
9.
Li  J, Dreyer  RP, Li  X,  et al; China PEACE Collaborative Group.  China patient-centered evaluative assessment of cardiac events prospective study of acute myocardial infarction: study design.  Chin Med J (Engl). 2016;129(1):72-80. doi:10.4103/0366-6999.172596PubMedGoogle ScholarCrossref
10.
Moons  KG, Altman  DG, Reitsma  JB,  et al.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration.  Ann Intern Med. 2015;162(1):W1-W73. doi:10.7326/M14-0698PubMedGoogle ScholarCrossref
11.
Sterne  JA, White  IR, Carlin  JB,  et al.  Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls.  BMJ. 2009;338:b2393. doi:10.1136/bmj.b2393PubMedGoogle ScholarCrossref
12.
Dellaportas  P, Forster  JJ, Ntzoufras  I.  On Bayesian model and variable selection using MCMC.  Stat Comput. 2002;12(1):27-36. doi:10.1023/A:1013164120801Google ScholarCrossref
13.
Lichtman  JH, Leifheit-Limson  EC, Jones  SB, Wang  Y, Goldstein  LB.  Preventable readmissions within 30 days of ischemic stroke among Medicare beneficiaries.  Stroke. 2013;44(12):3429-3435. doi:10.1161/STROKEAHA.113.003165PubMedGoogle ScholarCrossref
14.
Wang  Y.  A multinomial logistic regression modeling approach for anomaly intrusion detection.  Comput Secur. 2005;24(8):662-674. doi:10.1016/j.cose.2005.05.003Google ScholarCrossref
15.
Harrell  FE.  Regression Modeling Strategies With Applications to Linear Models, Logistic Regression, and Survival Analysis. 3rd ed. New York, NY: Springer; 2001.
16.
Heagerty  PJ, Lumley  T, Pepe  MS.  Time-dependent ROC curves for censored survival data and a diagnostic marker.  Biometrics. 2000;56(2):337-344. doi:10.1111/j.0006-341X.2000.00337.xPubMedGoogle ScholarCrossref
17.
Schemper  M, Henderson  R.  Predictive accuracy and explained variation in Cox regression.  Biometrics. 2000;56(1):249-255. doi:10.1111/j.0006-341X.2000.00249.xPubMedGoogle ScholarCrossref
18.
Schoenfeld  D.  Partial residuals for the proportional hazards regression model.  Biometrika. 1982;69(1):239-241. doi:10.1093/biomet/69.1.239Google ScholarCrossref
19.
Krumholz  HM, Wang  Y, Mattera  JA,  et al.  An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction.  Circulation. 2006;113(13):1683-1692. doi:10.1161/CIRCULATIONAHA.105.611186PubMedGoogle ScholarCrossref
20.
Yeh  RW, Secemsky  EA, Kereiakes  DJ,  et al; DAPT Study Investigators.  Development and validation of a prediction rule for benefit and harm of dual antiplatelet therapy beyond 1 year after percutaneous coronary intervention.  JAMA. 2016;315(16):1735-1749. doi:10.1001/jama.2016.3775PubMedGoogle ScholarCrossref
21.
Rassi  A  Jr, Rassi  A, Little  WC,  et al.  Development and validation of a risk score for predicting death in Chagas’ heart disease.  N Engl J Med. 2006;355(8):799-808. doi:10.1056/NEJMoa053241PubMedGoogle ScholarCrossref
22.
Lanza  ST, Collins  LM, Lemmon  DR, Schafer  JL.  PROC LCA: a SAS procedure for latent class analysis.  Struct Equ Modeling. 2007;14(4):671-694. doi:10.1080/10705510701575602PubMedGoogle ScholarCrossref
23.
Miller  CR, Sabagh  G, Dingman  HF.  Latent class analysis and differential mortality.  J Am Stat Assoc. 1962;57(298):430-438. doi:10.1080/01621459.1962.10480669Google ScholarCrossref
24.
Kim  M, Wall  MM, Li  G.  Applying latent class analysis to risk stratification for perioperative mortality in patients undergoing intraabdominal general surgery.  Anesth Analg. 2016;123(1):193-205. doi:10.1213/ANE.0000000000001279PubMedGoogle ScholarCrossref
25.
Crow  SJ, Swanson  SA, Peterson  CB, Crosby  RD, Wonderlich  SA, Mitchell  JE.  Latent class analysis of eating disorders: relationship to mortality.  J Abnorm Psychol. 2012;121(1):225-231. doi:10.1037/a0024455PubMedGoogle ScholarCrossref
26.
Leigh  L, Hudson  IL, Byles  JE.  Sleeping difficulty, disease and mortality in older women: a latent class analysis and distal survival analysis.  J Sleep Res. 2015;24(6):648-657. doi:10.1111/jsr.12324PubMedGoogle ScholarCrossref
27.
Ferrat  E, Audureau  E, Paillaud  E,  et al; ELCAPA Study Group.  Four distinct health profiles in older patients with cancer: latent class analysis of the prospective ELCAPA cohort.  J Gerontol A Biol Sci Med Sci. 2016;71(12):1653-1660. doi:10.1093/gerona/glw052PubMedGoogle ScholarCrossref
28.
Evenson  KR, Herring  AH, Wen  F.  Accelerometry-assessed latent class patterns of physical activity and sedentary behavior with mortality.  Am J Prev Med. 2017;52(2):135-143. doi:10.1016/j.amepre.2016.10.033PubMedGoogle ScholarCrossref
29.
Krumholz  HM, Radford  MJ, Wang  Y, Chen  J, Heiat  A, Marciniak  TA.  National use and effectiveness of beta-blockers for the treatment of elderly patients after acute myocardial infarction: National Cooperative Cardiovascular Project.  JAMA. 1998;280(7):623-629. doi:10.1001/jama.280.7.623PubMedGoogle ScholarCrossref
30.
Krumholz  HM, Radford  MJ, Ellerbeck  EF,  et al.  Aspirin in the treatment of acute myocardial infarction in elderly Medicare beneficiaries: patterns of use and outcomes.  Circulation. 1995;92(10):2841-2847. doi:10.1161/01.CIR.92.10.2841PubMedGoogle ScholarCrossref
31.
Barron  HV, Harr  SD, Radford  MJ, Wang  Y, Krumholz  HM.  The association between white blood cell count and acute myocardial infarction mortality in patients > or =65 years of age: findings from the cooperative cardiovascular project.  J Am Coll Cardiol. 2001;38(6):1654-1661. doi:10.1016/S0735-1097(01)01613-8PubMedGoogle ScholarCrossref
32.
Mehta  SK, Frutkin  AD, Lindsey  JB,  et al; National Cardiovascular Data Registry.  Bleeding in patients undergoing percutaneous coronary intervention: the development of a clinical risk algorithm from the National Cardiovascular Data Registry.  Circ Cardiovasc Interv. 2009;2(3):222-229. doi:10.1161/CIRCINTERVENTIONS.108.846741PubMedGoogle ScholarCrossref
33.
Rao  SV, McCoy  LA, Spertus  JA,  et al.  An updated bleeding model to predict the risk of post-procedure bleeding among patients undergoing percutaneous coronary intervention: a report using an expanded bleeding definition from the National Cardiovascular Data Registry CathPCI Registry.  JACC Cardiovasc Interv. 2013;6(9):897-904. doi:10.1016/j.jcin.2013.04.016PubMedGoogle ScholarCrossref
34.
Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387. doi:10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 889
    Original Investigation
    Cardiology
    August 10, 2018

    Risk Factors Associated With Major Cardiovascular Events 1 Year After Acute Myocardial Infarction

    Author Affiliations
    • 1Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
    • 2Center for Outcomes Research and Evaluation, Yale-New Haven Hospital, New Haven, Connecticut
    • 3Richard and Susan Smith Center for Outcomes Research in Cardiology, Division of Cardiology, Beth Israel Deaconess Medical, Harvard Medical School, Boston, Massachusetts
    • 4National Clinical Research Center of Cardiovascular Diseases, State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
    • 5Department of Cardiology, Peking University People’s Hospital, Beijing Key Laboratory of Early Prediction and Intervention of Acute Myocardial Infarction, Beijing, China
    • 6Brigham and Women’s Hospital Heart & Vascular Center, Harvard Medical School, Boston, Massachusetts
    • 7Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts
    • 8Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
    • 9Department of Health Policy and Management, Yale School of Public Health, New Haven, Connecticut
    JAMA Network Open. 2018;1(4):e181079. doi:10.1001/jamanetworkopen.2018.1079
    Key Points

    Question  What are the risk factors associated with major cardiovascular events within 1 year among patients who survive an acute myocardial infarction?

    Findings  In this cohort study of 4227 patients with acute myocardial infarction, 19 risk factors associated with major cardiovascular events 1 year after acute myocardial infarction, comprising 15 unique variables, were identified. A risk model to predict 1-year cardiovascular events after an acute myocardial infarction was developed and evaluated based on these risk factors.

    Meaning  Identification of individuals at the highest risk of long-term cardiovascular events after acute myocardial infarction may aid in the provision of targeted, intensive, and higher-quality longitudinal care following discharge.

    Abstract

    Importance  Patients who survive acute myocardial infarction (AMI) have a high risk of subsequent major cardiovascular events. Efforts to identify risk factors for recurrence have primarily focused on the period immediately following AMI admission.

    Objectives  To identify risk factors and develop and evaluate a risk model that predicts 1-year cardiovascular events after AMI.

    Design, Setting, and Participants  Prospective cohort study. Patients with AMI (n = 4227), aged 18 years or older, discharged alive from 53 acute-care hospitals across China from January 1, 2013, to July 17, 2014. Patients were randomly divided into samples: training (50% [2113 patients]), test (25% [1057 patients]), and validation (25% [1057 patients]). Risk factors were identified by a Cox model with Markov chain Monte Carlo simulation and further evaluated by latent class analysis. Analyses were conducted from May 1, 2017, to January 21, 2018.

    Main Outcomes and Measures  Major cardiovascular events, including recurrent AMI, stroke, heart failure, and death, within 1 year after discharge for the index AMI hospitalization.

    Results  The mean (SD) age of the cohort was 60.8 (11.8) years and 994 of 4227 patients (23.5%) were female. Common comorbidities included hypertension (2358 patients [55.8%]), coronary heart disease (1798 patients [42.5%]), and dyslipidemia (1290 patients [30.5%]). One-year event rates were 8.1% (95% CI, 6.91%-9.24%), 9.0% (95% CI, 7.22%-10.70%), and 6.4% (95% CI, 4.89%-7.85%) for the training, test, and validation samples, respectively. Nineteen risk factors comprising 15 unique variables (age, education, prior AMI, prior ventricular tachycardia or fibrillation, hypertension, angina, prearrival medical assistance, >4 hours from onset of symptoms to admission, ejection fraction, renal dysfunction, heart rate, systolic blood pressure, white blood cell count, blood glucose, and in-hospital complications) were identified. In the training, test, and validation samples, respectively, the risk model had C statistics of 0.79 (95% CI, 0.75-0.83), 0.73 (95% CI, 0.68-0.78), and 0.77 (95% CI, 0.70-0.83) and a predictive range of 1.2% to 33.9%, 1.2% to 37.9%, and 1.3% to 34.3%. The C statistic was 0.69 (95% CI, 0.65-0.74) for the latent class model in the training data. The risk model stratified 11.3%, 81.0%, and 7.7% of patients to high-, average-, and low-risk groups, with respective probabilities of 0.32, 0.06, and 0.01 for 1-year events.

    Conclusions and Relevance  Nineteen risk factors were identified, and a model was developed and evaluated to predict risk of 1-year cardiovascular events after AMI. This may aid clinicians in identifying high-risk patients who would benefit most from intensive follow-up and aggressive risk factor reduction.

    Introduction

    Patients who survive an acute myocardial infarction (AMI) are at high risk of major cardiovascular events following discharge, although risk may vary considerably between patients. Lifestyle changes, quality of care, and treatment with guideline-directed medical therapy after AMI are all important for secondary prevention. Patient characteristics are also associated with risk of cardiovascular events. Identification of these characteristics would better inform patients, families, and physicians about the risk of cardiovascular events following discharge and help ensure intensive follow-up and risk factor modification. Efforts to identify risk factors have focused on the period immediately after initial hospitalization for AMI and have not provided a longer-term perspective. Moreover, studies on long-term outcomes after an AMI predominately focus on mortality.1-4 Major cardiovascular events following discharge, such as recurrent AMI, stroke, and heart failure, all adversely affect a patient’s quality of life and are, in aggregate, much more common than death.1,5-8 There is a need to focus on long-term outcomes beyond mortality to capture the complete health care experience of patients, particularly in non-Western countries.

    Accordingly, we used the China Patient-Centered Evaluative Assessment of Cardiac Events Prospective Study of AMI (China PEACE Prospective AMI study)9 data to identify clinically important risk factors and develop and evaluate a risk model to predict major cardiovascular events up to 1 year following discharge for AMI. This study, which includes data abstracted from medical records and detailed longitudinal information about patients with AMI, is ideally positioned to generate tools to further stratify risk in the year after hospital discharge.

    Methods
    Study Sample

    The China PEACE Prospective AMI study consecutively enrolled 5901 patients with AMI aged 18 years or older who were discharged alive from acute-care hospitals in China from January 1, 2013, through July 17, 2014.9 Patient information was collected during the index AMI hospitalization and at 1, 6, and 12 months following discharge. Medical records were abstracted independently by 2 trained abstractors; agreement was 98% for key data elements. During the 1-year follow-up period, trained staff interviewed patients and systematically collected information on all clinical events, including recurrent AMI, angina, ischemic or hemorrhagic stroke, heart failure, transient ischemic attack, repeated angiography procedures, revascularization, bleeding, and rehospitalizations. Death information was obtained from the family or physician for patients who died during follow-up. For patients who were unable to attend an onsite interview, information was obtained by telephone through direct correspondence with the patient, the patient’s family members, or the patient’s physician.

    We excluded patients who were transferred to another acute-care hospital (n = 113), died prior to discharge (n = 287), did not agree to participate in the follow-up interviews (n = 1206), or were lost to follow-up immediately after discharge (n = 68). The final sample included 4227 unique patients who we randomly divided into 3 mutually exclusive samples for training (50% [2113 patients]), test (25% [1057 patients]), and validation (25% [1057 patients]) analyses. The training sample was used to select risk factors; test and validation samples were used for subsequent evaluation. Our rationale for creating the test and evaluation samples, rather than just 1 test sample, was to gain an additional opportunity to assess the stability and performance of the risk model. The Human Investigation Committee of China’s National Clinical Research Center of Cardiovascular Diseases approved this study. All patients provided written informed consent. The study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.10

    Potential Risk Factors

    We selected candidate risk factors that were clinically meaningful, were supported by the literature, were reliable and easily collected, occurred at a frequency of more than 1%, and were present before or during the index AMI hospitalization. Initial factors included patient demographic characteristics (age, sex, education, occupation, insurance), medical history (hypertension, diabetes mellitus, angina, coronary heart disease, prior ventricular tachycardia or ventricular fibrillation, atrial fibrillation, heart failure, prior AMI, prior percutaneous coronary intervention, prior coronary artery bypass grafting), lifestyle factors (smoking, drinking, snoring during natural sleep), time between onset of symptoms and admission greater than 4 hours, prearrival medical assistance, in-hospital diagnoses (ST segment elevation, AMI inferior or anterior, Killip class 3 or 4, ejection fraction [EF] <40%), renal dysfunction (blood urea nitrogen >40 mg/dL [to convert to mmol/L, multiply by 0.357] or creatinine >2.5 mg/dL [to convert to μmol/L, multiply by 88.4]), admission temperature (°C), heart rate greater than 90 beats per minute, blood glucose greater than 216 mg/dL (to convert to mmol/L, multiply by 0.0555), systolic blood pressure greater than 100 mm Hg, white blood cell count (WBC) 6000 to 12 000 per μL or greater than 12 000 per μL (to convert to ×109/L, multiply by 0.001), in-hospital treatments (coronary artery bypass grafting, percutaneous coronary intervention, medications), and number of in-hospital complications (recurrent angina, recurrent AMI, pericardial effusion, atrial fibrillation, cardiopulmonary resuscitation, ventricular tachycardia or fibrillation, new heart failure, infection, stroke, major bleeding) rated for severity on a scale of 0 to 10, with 10 being the most severe. Factors with missing values were imputed using multiple imputation with 10 imputations.11 The final imputed value was an average of the 10 imputations. Rates of missing ranged from 0.2% (systolic blood pressure and WBC) to 2.7% (blood glucose).

    Outcome

    The outcome for the prediction model was 1-year major cardiovascular events, a binary variable defined as the occurrence of recurrent AMI, stroke (ischemic or hemorrhagic), heart failure, or death within 1 year of discharge for the index AMI hospitalization. Information on the outcome was obtained through interviews and confirmed using medical records (eAppendix 1 in the Supplement).

    Statistical Analysis
    Risk Factor Selection and Evaluation

    Using the training sample with all candidate risk factors, we first fit a stepwise Cox proportional hazards model to eliminate factors that were unlikely to predict the outcome, with a P value threshold of .35 for adding factors and .25 for removing factors. All P values were 2-sided. Next, we fit the Cox model with Markov chain Monte Carlo (MCMC) simulation and calculated a posterior probability for each factor selected by the stepwise procedure.12 The posterior probability measures the strength of an association between a factor and the outcome. A factor with a posterior probability greater than 0.95 (or <0.05 for factors with estimates <0.0) was considered significant for predicting the outcome and therefore included in the final risk factor list.13,14 We then constructed the final risk model to predict 1-year events by refitting the Cox model to the training sample, without stepwise elimination, using the selected risk factors.

    We calculated the following indicators to evaluate the risk model performance: the Harrell C statistic to assess the overall predictive accuracy,15 time-dependent area under the receiver operating characteristic (ROC) curves to assess the predictive accuracy over the entire 1-year period,16 the Schemper-Henderson measure to assess explained variation,17 the Hosmer-Lemeshow goodness-of-fit test to assess calibration, and Schoenfeld residuals to assess the proportional hazards assumption.18 Discrimination was assessed among the observed outcomes in strata defined by deciles of the predictive probabilities.19 We divided patients in the training sample into 10 mutually exclusive risk classes based on these deciles, ranking the classes from lowest risk (class 1) to highest risk (class 10) for evaluation.

    To further evaluate the selected risk factors, we used the training data to conduct a latent class analysis, which uses an unsupervised machine learning algorithm that does not require an outcome. Conceptually, if the selected risk factors are substantially associated with the outcome, they will have the ability to assign patients to risk classes with unsupervised learning algorithms. Thus, we used latent class analysis to classify patients into 10 mutually exclusive classes and ranked them from lowest risk (class 1) to highest risk (class 10) on the basis of the observed outcome. We chose 10 classes to align with the decile-specific classes based on the risk model described previously. We calculated a Spearman correlation coefficient between the risk classes based on the risk model and the risk classes based on latent class analysis. A high coefficient indicates good agreement between the 2 classified results, which provides information on the robustness of the selected risk factors. We further validated the risk model by comparing its performance in the training sample with its performance in the test and validation samples (eAppendix 2 in the Supplement).

    Risk Score

    To facilitate the use of the selected risk factors and the risk model, we constructed a simple risk score for each patient based on the regression coefficients estimated from the risk model with the training sample. Points for each risk factor were calculated by dividing the risk factor’s coefficient by the sum of all coefficients in the risk model, multiplying by 100, and rounding to the nearest integer. We then calculated a risk score for each patient by summing that patient’s points.20,21 We stratified patients into 3 risk groups based on the distribution of the risk score: low (<10th percentile), average (10th-90th percentile), and high (>90th percentile). We further generated Kaplan-Meier survival curves to illustrate the probability of being free from 1-year events by the 3 risk groups.

    Analyses were conducted between May 1, 2017, and January 21, 2018, using SAS statistical software version 9.4 (SAS Institute Inc). Latent class analysis was conducted using the PROC LCA procedure, version 1.3.2 beta.22

    Results
    Study Sample

    A total of 4227 (2113 training, 1057 test, and 1057 validation) patients across 53 hospitals were included. The mean (SD) age was 60.8 (11.8) years, and 994 patients (23.5%) were female (eTable 1 in the Supplement). The most common comorbidities were hypertension (2358 patients [55.8%]), coronary heart disease (1798 patients [42.5%]), and dyslipidemia (1290 patients [30.5%]). Patient characteristics across the training, test, and validation samples were similar (eTable 1 in the Supplement). Median (interquartile range [IQR]) length of hospital stay was 11 (8-14) days.

    One-Year Cardiovascular Events

    For the training, test, and validation samples, respectively, the rates of 1-year cardiovascular events were 8.1% (95% CI, 6.91%-9.24%; 171 of 2113 patients), 9.0% (95% CI, 7.22%-10.70%; 94 of 1057 patients), and 6.4% (95% CI, 4.89%-7.85%; 67 of 1057 patients) (P = .36), and the number of cardiovascular events per patient ranged from 1 to 6, 1 to 7, and 1 to 6. Approximately 79.2% of patients (263 of 332) had 1 event, and 15.7% (52 of 332) had 2 events. The most common event was heart failure (249 of 4227 patients [5.9%]), followed by recurrent AMI (139 of 4227 patients [3.3%]) and stroke (42 of 4227 patients [1.0%]). In 4227 patients, the nonfatal cardiovascular event rate was 5.1% (216 patients). The 1-year cardiovascular-related and non–cardiovascular-related mortality rates were 2.4% (101 patients) and 0.3% (13 patients), respectively (total mortality rate, 2.7% [114 patients]). The median (IQR) time from discharge to an event was 89 (35-207) days; recurrent AMI occurred earlier than other events (median [IQR], 78 [10-198] days) (eFigure 1 in the Supplement). The individual events that made up the composite cardiovascular event outcome were positively correlated, with the tetrachoric correlation coefficient ranging from 0.06 (stroke and heart failure) to 0.57 (AMI and death).

    Risk Factor Selection and Validation

    The stepwise Cox regression identified 23 candidate factors, and MCMC simulation removed 4 that had a posterior probability of less than 0.95. The remaining 19 risk factors (Table 1), represented by 15 unique variables, include patient demographics (age, college degree), comorbidities (prior AMI, prior ventricular tachycardia or fibrillation, hypertension, angina), hospital diagnoses and tests (EF <40%, EF unable to measure, renal dysfunction, heart rate >90 beats per minute, blood glucose >216 mg/dL, SBP <100 mm Hg, WBC 6000/μL-12 000/μL, WBC >12 000/μL), access to care (prearrival medical assistance, time from onset of symptoms to admission >4 hours), and in-hospital complications (eFigure 2 in the Supplement). The risk model based on the 19 risk factors and the training sample demonstrated good discrimination, calibration, and fit. For the risk model, the overall C statistic was 0.79 (95% CI, 0.75-0.83), and the time-dependent area under the ROC curve ranged from 0.79 (95% CI, 0.75-0.83) to 0.75 (95% CI, 0.71-0.79) (eFigure 3 in the Supplement). The mean observed 1-year outcome ranged from 1.2% in the lowest predicted decile to 33.9% in the highest predicted decile, a range of 32.7% (eFigure 4 in the Supplement). The P value of the Hosmer-Lemeshow goodness-of-fit test was .77, suggesting the model was fitted well (eFigure 5 in the Supplement). The explained variation was 0.14, and the Schoenfeld residuals test indicated that all risk factors, except for WBC greater than 12 000/μL and renal dysfunction, met the proportional hazards assumption.

    The latent class analysis assigned each of the 2113 patients in the training sample into 10 classes based on the combination of 19 risk factors (eFigure 6 in the Supplement). The area under the ROC curve for this analysis was 0.69 (95% CI, 0.65-0.74), and the mean observed outcome rate ranged from 3.4% in the lowest rating group to 19.3% in the highest rating group, a range of 15.9%. The Spearman correlation coefficient between the predicted decile based on the Cox risk model and the latent class analysis was 0.60 (95% CI, 0.57-0.63).

    The probabilities of 1-year events based on regression coefficients estimated from the risk model in the training sample and the risk model in the test and validation samples were highly correlated (eFigure 7 in the Supplement). Model performance in test and validation samples was comparable to that in the training sample. For the test and validation samples, respectively, the overall C statistic was 0.73 (95% CI, 0.68-0.78) and 0.77 (95% CI, 0.70-0.83); the rate of observed 1-year events ranged from 1.2% and 1.3% in the lowest predicted decile to 37.9% and 34.3% in the highest predicted decile; the explained variations were 0.09 and 0.16.

    Risk Score

    The risk factor–specific points ranged from 2 (hypertension and number of in-hospital complications) to 14 (age ≥85 years) (Table 2). Age, EF, WBC, prior ventricular tachycardia or ventricular fibrillation, prior angina, and heart rate were the top 5 factors with a risk ratio greater than 2.00. The training sample had a mean (SD) risk score of 20.2 (7.9). Scores varied little across the test and validation samples. The mean (SD) score was 20.6 (8.1) for the test sample and 20.4 (7.8) for the validation sample (Figure 1). In the training sample, 11.3%, 81.0%, and 7.7% of patients, respectively, were stratified to the high-, average-, and low-risk groups, with corresponding probabilities of 0.32, 0.06, and 0.01 for 1-year events (Figure 2). Figure 3 shows Kaplan-Meier estimates of the probability of being free from 1-year major cardiovascular events by risk groups. The stratifications for the test and validation samples were not substantially different from those for the training sample (Figure 1, Figure 2, and Figure 3; eTable 2 in the Supplement).

    Discussion

    In this study, we identified 19 risk factors and developed and evaluated a risk model that predicts 1-year major cardiovascular events after discharge for AMI. These factors, selected based on abstracted data from medical records and robust statistical learning algorithms, are easy to collect and are readily available at time of discharge. The risk model and its corresponding risk scores allow clinicians to identify patients who are at heightened risk of 1-year cardiovascular events and might also help patients understand their risk of not only death but also adverse events that could impair their quality of life. The ability to identify individuals with the highest risk of long-term cardiovascular events after AMI may aid in the provision of targeted, intensive, and higher-quality longitudinal care following discharge.

    Our study, based on abstracted information and patient interviews, represents a unique, large, and contemporary investigation of the risk factors associated with long-term outcomes following discharge for AMI in China. In contrast to prior studies that focused solely on mortality, we evaluated major cardiovascular events in addition to mortality. Although the overall 1-year mortality rate for patients surviving an AMI is important, many patients experience nonfatal adverse events that can dramatically affect their quality of life. In our sample, 1-year mortality was 2.7% but the rate of 1-year nonfatal cardiovascular events was 5.1%, approximately 89% higher than the mortality rate. Insight regarding the long-term risk of major but nonfatal adverse events is important from a patient perspective, and educating patients regarding their long-term risk might provide an even stronger incentive to follow up and adhere to medications. Through risk stratification, we found that 1 in 10 patients (11.3%) were at high risk of experiencing a major cardiovascular event after discharge, highlighting the importance of identifying these patients at discharge to ensure they receive appropriate, evidence-based, longitudinal care after discharge. Because this high-risk group was small and many events occurred among those at average risk, a population risk reduction approach that focuses on average-risk individuals is also important and may save more lives.

    We used the MCMC algorithm to further assess the strength of the risk factors associated with the outcome, in addition to a traditional stepwise approach. One of the challenges in developing a predictive model is balancing the number of factors included in the model and the variance. More factors are desirable for prediction. The MCMC algorithm, which provides a robust Bayesian variable selection approach, allowed us to select a factor based on the marginal posterior probability that the factor should be in the model. Additionally, we used latent class analysis to validate the selected risk factors. The occurrence of major cardiovascular events following discharge is unobservable at discharge and associated with unknown factors after discharge. Such characteristics present a challenge for predictive model development based on conventional regression methods. Latent class analysis is increasingly being used in outcomes research23-28 and can stratify patients into subgroups based on their risk factors at discharge when outcome information is absent in a training sample. The method is suitable to address at least some of the challenges of a regression approach. By conjoining a traditional Cox regression with latent class analysis, our study used the strengths from both methods and increased the reliability and validity of the selected risk factors. Given the nature of unsupervised learning of the latent class analysis, the discrimination reported by this study is also reasonable.

    Our risk factors were identified based on a large patient sample and subsequently tested and validated with high agreement with the training results. Potential factors used for developing a predictive model may influence the strength of the final model in several ways. An ideal risk factor should be unaffected by clinical interpretation, widely accepted, easily collected, and available on admission, during hospitalization, and at discharge. The 19 risk factors identified in this study met these criteria, and many of them have been identified in other studies.29-31 Some of the selected risk factors are modifiable, providing multiple targets for patient and physician interventions (eg, blood pressure control, lifestyle changes, and appropriate care after discharge) to prevent major cardiovascular events after discharge. One risk factor, the number of in-hospital complications, is related to hospitals and could reflect hospital quality of care. Our data showed that each additional in-hospital complication was associated with a 24% increase in the risk of cardiovascular events after discharge. If a hospital can reduce its complication rate, it could improve its patients’ outcomes after discharge. Although the 19 risk factors were selected based on a Chinese AMI population, factors such as insurance, patient occupation, and hospital length of stay, which may vary across countries, were included in the initial stepwise process but were not identified as the final risk factors. Thus, the 19 risk factors are likely relevant outside of China and could provide clinicians with targets for treatment that may improve prognosis. Validation in additional studies is necessary, but this potential for generalization to other countries underscores the utility of the data presented in this study. Age, EF, WBC, prior ventricular tachycardia or ventricular fibrillation, prior angina, and heart rate were the top 5 factors with a risk ratio greater than 2.00, potentially suggesting that these could be the most clinically important factors.

    Limitations

    Our study has several limitations. Our training, test, and validation samples were all drawn from 1 cohort. Although this approach has been used in many prediction studies based on a single cohort,21,32,33 the risk factors and model identified in our study should continue to be evaluated and updated as additional data become available. The risk score was calculated based on rounding each risk factor’s coefficient. Although this may result in a loss of information, it provides a simple and easy method for risk prediction in daily clinical practice. To further encourage the use of the score in clinical practice, the score could be implemented as an application for portable devices by adapting a nomogram.34

    Conclusions

    In conclusion, our 19-factor risk model had good predictive range and was able to identify high-risk patients at the time of discharge. It provides a basis for clinicians to better understand patients’ risk of long-term major cardiovascular events after an AMI and can help clinicians make better-targeted, evidence-based decisions for AMI care following discharge.

    Back to top
    Article Information

    Accepted for Publication: May 9, 2018.

    Published: August 10, 2018. doi:10.1001/jamanetworkopen.2018.1079

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2018 Wang Y et al. JAMA Network Open.

    Corresponding Author: Lixin Jiang, MD, PhD, National Clinical Research Center of Cardiovascular Diseases, State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Rd, Beijing 100037, People’s Republic of China (jiangl@fwoxford.org).

    Author Contributions: Drs Y. Wang and Hu had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Y. Wang and J. Li are co–first authors; Drs Krumholz and L. Jiang are both senior authors.

    Concept and design: Y. Wang, J. Li, Zheng, Normand, Krumholz, L. Jiang.

    Acquisition, analysis, or interpretation of data: Y. Wang, J. Li, Zheng, Z. Jiang, Hu, Wadhera, Bai, Lu, Q. Wang, Y. Li, Wu, Xing, Normand, Krumholz, L. Jiang.

    Drafting of the manuscript: Y. Wang, J. Li, Krumholz.

    Critical revision of the manuscript for important intellectual content: Y. Wang, J. Li, Zheng, Z. Jiang, Hu, Wadhera, Bai, Lu, Q. Wang, Y. Li, Wu, Xing, Normand, Krumholz, L. Jiang.

    Statistical analysis: Y. Wang, Hu, Normand.

    Administrative, technical, or material support: J. Li, Zheng, Z. Jiang, Xing.

    Supervision: Y. Wang, J. Li, Xing, Krumholz, L. Jiang.

    Conflict of Interest Disclosures: Dr J. Li reported grants from the Chinese government during the conduct of the study; in addition, Dr J. Li had a patent for a risk prediction tool. Dr Hu reported a pending patent for a long-term predictive model of patients’ prognosis in 1 year after discharge. Dr Krumholz reported grants from Medtronic, Johnson & Johnson (Janssen), and the US Food and Drug Administration; other support from the Centers for Medicare & Medicaid Services; personal fees from UnitedHealthcare, IBM Watson Health, Element Science, and Aetna; and other support from Hugo outside the submitted work. No other disclosures were reported.

    Additional Contributions: We thank the hospitals that participated in the China Patient-Centered Evaluative Assessment of Cardiac Events Prospective Study of AMI for their contributions to this work. We thank R. Matthew Ramirez, PhD (University of California, Berkeley), and Changyu Shen, PhD (Beth Israel Deaconess Medical, Harvard Medical School, Boston, Massachusetts), for their valuable comments on this study and Erica C. Leifheit, PhD (Yale School of Public Health), for her contribution to the figures. Drs Ramirez, Shen, and Leifheit were not compensated for their participation.

    References
    1.
    Tu  JV, Austin  PC, Walld  R, Roos  L, Agras  J, McDonald  KM.  Development and validation of the Ontario acute myocardial infarction mortality prediction rules.  J Am Coll Cardiol. 2001;37(4):992-997. doi:10.1016/S0735-1097(01)01109-3PubMedGoogle ScholarCrossref
    2.
    Smolderen  KG, Buchanan  DM, Gosch  K,  et al.  Depression treatment and 1-year mortality following acute myocardial infarction: insights from the TRIUMPH registry (Translational Research Investigating Underlying Disparities in Acute Myocardial Infarction Patients’ Health Status).  Circulation. 2017;135(18):1681-1689. doi:10.1161/CIRCULATIONAHA.116.025140PubMedGoogle ScholarCrossref
    3.
    Plakht  Y, Shiyovich  A, Gilutz  H.  Predictors of long-term (10-year) mortality postmyocardial infarction: age-related differences. Soroka Acute Myocardial Infarction (SAMI) Project.  J Cardiol. 2015;65(3):216-223. doi:10.1016/j.jjcc.2014.06.001PubMedGoogle ScholarCrossref
    4.
    Ketchum  ES, Dickstein  K, Kjekshus  J,  et al.  The Seattle Post Myocardial Infarction Model (SPIM): prediction of mortality after acute myocardial infarction with left ventricular dysfunction.  Eur Heart J Acute Cardiovasc Care. 2014;3(1):46-55. doi:10.1177/2048872613502283PubMedGoogle ScholarCrossref
    5.
    Chaudhry  SI, Khan  RF, Chen  J,  et al.  National trends in recurrent AMI hospitalizations 1 year after acute myocardial infarction in Medicare beneficiaries: 1999-2010.  J Am Heart Assoc. 2014;3(5):e001197. doi:10.1161/JAHA.114.001197PubMedGoogle ScholarCrossref
    6.
    Wang  Y, Lichtman  JH, Dharmarajan  K,  et al.  National trends in stroke after acute myocardial infarction among Medicare patients in the United States: 1999 to 2010.  Am Heart J. 2015;169(1):78-85.e4. doi:10.1016/j.ahj.2014.06.011PubMedGoogle ScholarCrossref
    7.
    Cahill  TJ, Kharbanda  RK.  Heart failure after myocardial infarction in the era of primary percutaneous coronary intervention: mechanisms, incidence and identification of patients at risk.  World J Cardiol. 2017;9(5):407-415. doi:10.4330/wjc.v9.i5.407PubMedGoogle ScholarCrossref
    8.
    Sampson  UK, Pfeffer  MA, McMurray  JJ, Lokhnygina  Y, White  HD, Solomon  SD; VALIANT Trial Investigators.  Predictors of stroke in high-risk patients after acute myocardial infarction: insights from the VALIANT trial.  Eur Heart J. 2007;28(6):685-691. doi:10.1093/eurheartj/ehl197PubMedGoogle ScholarCrossref
    9.
    Li  J, Dreyer  RP, Li  X,  et al; China PEACE Collaborative Group.  China patient-centered evaluative assessment of cardiac events prospective study of acute myocardial infarction: study design.  Chin Med J (Engl). 2016;129(1):72-80. doi:10.4103/0366-6999.172596PubMedGoogle ScholarCrossref
    10.
    Moons  KG, Altman  DG, Reitsma  JB,  et al.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration.  Ann Intern Med. 2015;162(1):W1-W73. doi:10.7326/M14-0698PubMedGoogle ScholarCrossref
    11.
    Sterne  JA, White  IR, Carlin  JB,  et al.  Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls.  BMJ. 2009;338:b2393. doi:10.1136/bmj.b2393PubMedGoogle ScholarCrossref
    12.
    Dellaportas  P, Forster  JJ, Ntzoufras  I.  On Bayesian model and variable selection using MCMC.  Stat Comput. 2002;12(1):27-36. doi:10.1023/A:1013164120801Google ScholarCrossref
    13.
    Lichtman  JH, Leifheit-Limson  EC, Jones  SB, Wang  Y, Goldstein  LB.  Preventable readmissions within 30 days of ischemic stroke among Medicare beneficiaries.  Stroke. 2013;44(12):3429-3435. doi:10.1161/STROKEAHA.113.003165PubMedGoogle ScholarCrossref
    14.
    Wang  Y.  A multinomial logistic regression modeling approach for anomaly intrusion detection.  Comput Secur. 2005;24(8):662-674. doi:10.1016/j.cose.2005.05.003Google ScholarCrossref
    15.
    Harrell  FE.  Regression Modeling Strategies With Applications to Linear Models, Logistic Regression, and Survival Analysis. 3rd ed. New York, NY: Springer; 2001.
    16.
    Heagerty  PJ, Lumley  T, Pepe  MS.  Time-dependent ROC curves for censored survival data and a diagnostic marker.  Biometrics. 2000;56(2):337-344. doi:10.1111/j.0006-341X.2000.00337.xPubMedGoogle ScholarCrossref
    17.
    Schemper  M, Henderson  R.  Predictive accuracy and explained variation in Cox regression.  Biometrics. 2000;56(1):249-255. doi:10.1111/j.0006-341X.2000.00249.xPubMedGoogle ScholarCrossref
    18.
    Schoenfeld  D.  Partial residuals for the proportional hazards regression model.  Biometrika. 1982;69(1):239-241. doi:10.1093/biomet/69.1.239Google ScholarCrossref
    19.
    Krumholz  HM, Wang  Y, Mattera  JA,  et al.  An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction.  Circulation. 2006;113(13):1683-1692. doi:10.1161/CIRCULATIONAHA.105.611186PubMedGoogle ScholarCrossref
    20.
    Yeh  RW, Secemsky  EA, Kereiakes  DJ,  et al; DAPT Study Investigators.  Development and validation of a prediction rule for benefit and harm of dual antiplatelet therapy beyond 1 year after percutaneous coronary intervention.  JAMA. 2016;315(16):1735-1749. doi:10.1001/jama.2016.3775PubMedGoogle ScholarCrossref
    21.
    Rassi  A  Jr, Rassi  A, Little  WC,  et al.  Development and validation of a risk score for predicting death in Chagas’ heart disease.  N Engl J Med. 2006;355(8):799-808. doi:10.1056/NEJMoa053241PubMedGoogle ScholarCrossref
    22.
    Lanza  ST, Collins  LM, Lemmon  DR, Schafer  JL.  PROC LCA: a SAS procedure for latent class analysis.  Struct Equ Modeling. 2007;14(4):671-694. doi:10.1080/10705510701575602PubMedGoogle ScholarCrossref
    23.
    Miller  CR, Sabagh  G, Dingman  HF.  Latent class analysis and differential mortality.  J Am Stat Assoc. 1962;57(298):430-438. doi:10.1080/01621459.1962.10480669Google ScholarCrossref
    24.
    Kim  M, Wall  MM, Li  G.  Applying latent class analysis to risk stratification for perioperative mortality in patients undergoing intraabdominal general surgery.  Anesth Analg. 2016;123(1):193-205. doi:10.1213/ANE.0000000000001279PubMedGoogle ScholarCrossref
    25.
    Crow  SJ, Swanson  SA, Peterson  CB, Crosby  RD, Wonderlich  SA, Mitchell  JE.  Latent class analysis of eating disorders: relationship to mortality.  J Abnorm Psychol. 2012;121(1):225-231. doi:10.1037/a0024455PubMedGoogle ScholarCrossref
    26.
    Leigh  L, Hudson  IL, Byles  JE.  Sleeping difficulty, disease and mortality in older women: a latent class analysis and distal survival analysis.  J Sleep Res. 2015;24(6):648-657. doi:10.1111/jsr.12324PubMedGoogle ScholarCrossref
    27.
    Ferrat  E, Audureau  E, Paillaud  E,  et al; ELCAPA Study Group.  Four distinct health profiles in older patients with cancer: latent class analysis of the prospective ELCAPA cohort.  J Gerontol A Biol Sci Med Sci. 2016;71(12):1653-1660. doi:10.1093/gerona/glw052PubMedGoogle ScholarCrossref
    28.
    Evenson  KR, Herring  AH, Wen  F.  Accelerometry-assessed latent class patterns of physical activity and sedentary behavior with mortality.  Am J Prev Med. 2017;52(2):135-143. doi:10.1016/j.amepre.2016.10.033PubMedGoogle ScholarCrossref
    29.
    Krumholz  HM, Radford  MJ, Wang  Y, Chen  J, Heiat  A, Marciniak  TA.  National use and effectiveness of beta-blockers for the treatment of elderly patients after acute myocardial infarction: National Cooperative Cardiovascular Project.  JAMA. 1998;280(7):623-629. doi:10.1001/jama.280.7.623PubMedGoogle ScholarCrossref
    30.
    Krumholz  HM, Radford  MJ, Ellerbeck  EF,  et al.  Aspirin in the treatment of acute myocardial infarction in elderly Medicare beneficiaries: patterns of use and outcomes.  Circulation. 1995;92(10):2841-2847. doi:10.1161/01.CIR.92.10.2841PubMedGoogle ScholarCrossref
    31.
    Barron  HV, Harr  SD, Radford  MJ, Wang  Y, Krumholz  HM.  The association between white blood cell count and acute myocardial infarction mortality in patients > or =65 years of age: findings from the cooperative cardiovascular project.  J Am Coll Cardiol. 2001;38(6):1654-1661. doi:10.1016/S0735-1097(01)01613-8PubMedGoogle ScholarCrossref
    32.
    Mehta  SK, Frutkin  AD, Lindsey  JB,  et al; National Cardiovascular Data Registry.  Bleeding in patients undergoing percutaneous coronary intervention: the development of a clinical risk algorithm from the National Cardiovascular Data Registry.  Circ Cardiovasc Interv. 2009;2(3):222-229. doi:10.1161/CIRCINTERVENTIONS.108.846741PubMedGoogle ScholarCrossref
    33.
    Rao  SV, McCoy  LA, Spertus  JA,  et al.  An updated bleeding model to predict the risk of post-procedure bleeding among patients undergoing percutaneous coronary intervention: a report using an expanded bleeding definition from the National Cardiovascular Data Registry CathPCI Registry.  JACC Cardiovasc Interv. 2013;6(9):897-904. doi:10.1016/j.jcin.2013.04.016PubMedGoogle ScholarCrossref
    34.
    Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387. doi:10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4PubMedGoogle ScholarCrossref
    ×