[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.  Incidence Rates per 1000 Person-Years for Composite Outcome and Atherosclerotic Cardiovascular Disease (ASCVD) Death
Incidence Rates per 1000 Person-Years for Composite Outcome and Atherosclerotic Cardiovascular Disease (ASCVD) Death

Whiskers indicate 95% CIs. To convert total cholesterol to millimoles per liter, multiply by 0.0259.

Figure 2.  Calibration of Pooled Cohort Equations and Cohort-Derived Model for 5-Year Prediction of Atherosclerotic Cardiovascular Disease Events
Calibration of Pooled Cohort Equations and Cohort-Derived Model for 5-Year Prediction of Atherosclerotic Cardiovascular Disease Events
Table 1.  Baseline Characteristics and Incident ASCVD Events Among 1 672 336 Veterans
Baseline Characteristics and Incident ASCVD Events Among 1 672 336 Veterans
Table 2.  Five-Year Estimation Models for Composite ASCVD Events and ASCVD Mortality, Without and With Statin Therapy
Five-Year Estimation Models for Composite ASCVD Events and ASCVD Mortality, Without and With Statin Therapy
Table 3.  Harrell C Statistics for Composite ASCVD Events
Harrell C Statistics for Composite ASCVD Events
1.
D’Agostino  RB  Sr, Grundy  S, Sullivan  LM, Wilson  P; CHD Risk Prediction Group.  Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation.   JAMA. 2001;286(2):180-187. doi:10.1001/jama.286.2.180PubMedGoogle ScholarCrossref
2.
Goff  DC  Jr, Lloyd-Jones  DM, Bennett  G,  et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.   Circulation. 2014;129(25)(suppl 2):S49-S73. doi:10.1161/01.cir.0000437741.48606.98PubMedGoogle ScholarCrossref
3.
Stone  NJ, Robinson  JG, Lichtenstein  AH,  et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.   J Am Coll Cardiol. 2014;63(25 Pt B):2889-2934. doi:10.1016/j.jacc.2013.11.002PubMedGoogle ScholarCrossref
4.
Grundy  SM, Stone  NJ, Bailey  AL,  et al.  2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.   Circulation. 2019;139(25):e1082-e1143.PubMedGoogle Scholar
5.
Salami  JA, Warraich  H, Valero-Elizondo  J,  et al.  National trends in statin use and expenditures in the US adult population from 2002 to 2013: insights from the Medical Expenditure Panel Survey.   JAMA Cardiol. 2017;2(1):56-65. doi:10.1001/jamacardio.2016.4700PubMedGoogle ScholarCrossref
6.
Pencina  MJ, Navar-Boggan  AM, D’Agostino  RB  Sr,  et al.  Application of new cholesterol guidelines to a population-based sample.   N Engl J Med. 2014;370(15):1422-1431. doi:10.1056/NEJMoa1315665PubMedGoogle ScholarCrossref
7.
Sussman  JB, Wiitala  WL, Zawistowski  M, Hofer  TP, Bentley  D, Hayward  RA.  The Veterans Affairs Cardiac Risk Score: recalibrating the atherosclerotic cardiovascular disease score for applied use.   Med Care. 2017;55(9):864-870. doi:10.1097/MLR.0000000000000781PubMedGoogle ScholarCrossref
8.
Price  LE, Shea  K, Gephart  S.  The Veterans Affairs’s Corporate Data Warehouse: uses and implications for nursing research and practice.   Nurs Adm Q. 2015;39(4):311-318. doi:10.1097/NAQ.0000000000000118PubMedGoogle ScholarCrossref
9.
US Department of Veterans Affairs. Frequently Asked Questions—2019 National Veteran Suicide Prevention Annual Report. Published September 20, 2019. Accessed June 4, 2020. https://www.mentalhealth.va.gov/docs/data-sheets/2019/FAQ_2019_Annual_Report_508.pdf
10.
US Department of Veterans Affairs. System of records notice 97VA10P1: Consolidated Data Information System-VA. Updated March 3, 2015. Accessed June 4, 2020. https://www.oprm.va.gov/docs/sorn/sorn97va10p1.pdf
11.
Vassy  JL, Ho  YL, Honerlaw  J,  et al.  Yield and bias in defining a cohort study baseline from electronic health record data.   J Biomed Inform. 2018;78:54-59. doi:10.1016/j.jbi.2017.12.017PubMedGoogle ScholarCrossref
12.
Zhang  Y, Cai  T, Yu  S,  et al.  High-throughput phenotyping with electronic medical record data using a common semi-supervised approach (PheCAP).   Nat Protoc. 2019;14(12):3426-3444. doi:10.1038/s41596-019-0227-6PubMedGoogle ScholarCrossref
13.
Raghavan  S, Vassy  JL, Ho  YL,  et al.  Diabetes mellitus-related all-cause and cardiovascular mortality in a national cohort of adults.   J Am Heart Assoc. 2019;8(4):e011295. doi:10.1161/JAHA.118.011295PubMedGoogle Scholar
14.
Song  RJ, Ho  Y-L, Nguyen  X-MT,  et al  Abstract 18809: development of an electronic health record-based algorithm for smoking status using the Million Veteran Program (MVP) cohort survey response.   Circulation. 2016;134(suppl 1):A18809-A18809. Accessed May 29, 2020. https://www.ahajournals.org/doi/abs/10.1161/circ.134.suppl_1.18809Google Scholar
15.
Pencina  MJ, D’Agostino  RB  Sr, D’Agostino  RB  Jr, Vasan  RS.  Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.   Stat Med. 2008;27(2):157-172. doi:10.1002/sim.2929PubMedGoogle ScholarCrossref
16.
Muntner  P, Colantonio  LD, Cushman  M,  et al.  Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations.   JAMA. 2014;311(14):1406-1415. doi:10.1001/jama.2014.2630PubMedGoogle ScholarCrossref
17.
VA-ASCVD risk prediction calculator. Updated May 13, 2020. Accessed May 29, 2020. http://ascvd.cardiocalcs.com
18.
Baigent  C, Blackwell  L, Emberson  J,  et al; Cholesterol Treatment Trialists’ (CTT) Collaboration.  Efficacy and safety of more intensive lowering of LDL cholesterol: a meta-analysis of data from 170,000 participants in 26 randomised trials.   Lancet. 2010;376(9753):1670-1681. doi:10.1016/S0140-6736(10)61350-5PubMedGoogle ScholarCrossref
19.
Ridker  PM, Wilson  PW.  A trial-based approach to statin guidelines.   JAMA. 2013;310(11):1123-1124. doi:10.1001/jama.2013.276529PubMedGoogle ScholarCrossref
20.
Cook  NR, Ridker  PM.  Calibration of the Pooled Cohort Equations for atherosclerotic cardiovascular disease: an update.   Ann Intern Med. 2016;165(11):786-794. doi:10.7326/M16-1739PubMedGoogle ScholarCrossref
21.
Royston  P, Altman  DG.  External validation of a Cox prognostic model: principles and methods.   BMC Med Res Methodol. 2013;13:33. doi:10.1186/1471-2288-13-33PubMedGoogle ScholarCrossref
22.
Pencina  MJ, D’Agostino  RB.  Evaluation of the Framingham risk score in the European Prospective Investigation of Cancer-Norfolk cohort—invited commentary.   Arch Intern Med. 2008;168(11):1216-1218. doi:10.1001/archinte.168.11.1216PubMedGoogle ScholarCrossref
23.
Pennells  L, Kaptoge  S, Wood  A,  et al; Emerging Risk Factors Collaboration.  Equalization of four cardiovascular risk algorithms after systematic recalibration: individual-participant meta-analysis of 86 prospective studies.   Eur Heart J. 2019;40(7):621-631. doi:10.1093/eurheartj/ehy653PubMedGoogle ScholarCrossref
24.
Emdin  CA, Khera  AV, Natarajan  P,  et al.  Evaluation of the Pooled Cohort Equations for prediction of cardiovascular risk in a contemporary prospective cohort.   Am J Cardiol. 2017;119(6):881-885. doi:10.1016/j.amjcard.2016.11.042PubMedGoogle ScholarCrossref
25.
Wolfson  J, Vock  DM, Bandyopadhyay  S,  et al.  Use and customization of risk scores for predicting cardiovascular events using electronic health record data.   J Am Heart Assoc. 2017;6(4):e003670. doi:10.1161/JAHA.116.003670PubMedGoogle Scholar
26.
Pencina  MJ, Fine  JP, D’Agostino  RB  Sr.  Discrimination slope and integrated discrimination improvement: properties, relationships and impact of calibration.   Stat Med. 2017;36(28):4482-4490. doi:10.1002/sim.7139PubMedGoogle ScholarCrossref
27.
Toll  DB, Janssen  KJ, Vergouwe  Y, Moons  KG.  Validation, updating and impact of clinical prediction rules: a review.   J Clin Epidemiol. 2008;61(11):1085-1094. doi:10.1016/j.jclinepi.2008.04.008PubMedGoogle ScholarCrossref
28.
Danesh  J, Lewington  S, Thompson  SG,  et al; Fibrinogen Studies Collaboration.  Plasma fibrinogen level and the risk of major cardiovascular diseases and nonvascular mortality: an individual participant meta-analysis.   JAMA. 2005;294(14):1799-1809.PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 1,318
    Citations 0
    Original Investigation
    Cardiology
    July 14, 2020

    Estimation of Atherosclerotic Cardiovascular Disease Risk Among Patients in the Veterans Affairs Health Care System

    Author Affiliations
    • 1Veterans Affairs Boston Healthcare System, Boston, Massachusetts
    • 2Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
    • 3Veterans Affairs Eastern Colorado Healthcare System, Aurora
    • 4Division of Hospital Medicine, University of Colorado School of Medicine, Aurora
    • 5Colorado Cardiovascular Outcomes Research Consortium, Aurora
    • 6Landmark College, Putney, Vermont
    • 7Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts
    • 8Atlanta Veterans Affairs Medical Center, Decatur, Georgia
    • 9Division of Cardiology, Emory University School of Medicine, Atlanta, Georgia
    • 10Rollins School of Public Health, Department of Epidemiology, Emory University, Atlanta, Georgia
    JAMA Netw Open. 2020;3(7):e208236. doi:10.1001/jamanetworkopen.2020.8236
    Key Points español 中文 (chinese)

    Question  Are current risk prediction models accurate at estimating risk of initial atherosclerotic cardiovascular (ASCVD) events in veterans?

    Findings  In this cohort study of 1 672 336 veterans with an outpatient visit between 2002 and 2007, the 2013 American College of Cardiology/American Heart Association model overestimated absolute risk of ASCVD during 5 years of follow-up. Statin use was associated with 7% lower relative risk of ASCVD and 25% lower relative risk of ASCVD mortality.

    Meaning  The findings of this study suggest that reestimation and the inclusion of statin use in ASCVD prediction models might be needed for their appropriate use in a health care system.

    Abstract

    Importance  Current guidelines recommend statin therapy for millions of US residents for the primary prevention of atherosclerotic cardiovascular disease (ASCVD). It is unclear whether traditional prediction models that do not account for current widespread statin use are sufficient for risk assessment.

    Objectives  To examine the performance of the Pooled Cohort Equations (PCE) for 5-year ASCVD risk estimation in a contemporary cohort and to test the hypothesis that inclusion of statin therapy improves model performance.

    Design, Setting, and Participants  This cohort study included adult patients in the Veterans Affairs health care system without baseline ASCVD. Using national electronic health record data, 3 Cox proportional hazards models were developed to estimate 5-year ASCVD risk, as follows: the variables and published β coefficients from the PCE (model 1), the PCE variables with cohort-derived β coefficients (model 2), and model 2 plus baseline statin use (model 3). Data were collected from January 2002 to December 2012 and analyzed from June 2016 to March 2020.

    Exposures  Traditional ASCVD risk factors from the PCE plus baseline statin use.

    Main Outcomes and Measures  Incident ASCVD and ASCVD mortality.

    Results  Of 1 672 336 patients in the cohort (mean [SD] baseline age 58.0 [13.8] years, 1 575 163 [94.2%] men, 1 383 993 [82.8%] white), 312 155 (18.7%) were receiving statin therapy at baseline. During 5 years of follow-up, 66 605 (4.0%) experienced an ASCVD event, and 31 878 (1.9%) experienced ASCVD death. Compared with the original PCE, the cohort-derived model did not improve model discrimination in any of the 4 age-sex strata but did improve model calibration. The PCE overestimated ASCVD risk compared with the cohort-derived model; 211 237 of 1 136 161 white men (18.6%), 29 634 of 218 463 black men (13.6%), 1741 of 44 399 white women (3.9%), and 836 of 16 034 black women (5.2%) would be potentially eligible for statin therapy under the PCE but not the cohort-derived model. When added to the cohort-derived model, baseline statin therapy was associated with a 7% (95% CI, 5%-9%) lower relative risk of ASCVD and a 25% (95% CI, 23%-28%) lower relative risk for ASCVD death.

    Conclusions and Relevance  In this study, lower than expected rates of incident ASCVD events in a contemporary national cohort were observed. The PCE overestimated ASCVD risk, and more than 15% of patients would be potentially eligible for statin therapy based on the PCE but not on a cohort-derived model. In the statin era, health care professionals and systems should base ASCVD risk assessment on models calibrated to their patient populations.

    Introduction

    Atherosclerotic cardiovascular disease (ASCVD) continues to be the leading cause of death and disability in the United States, and predicting and reducing ASCVD risk remains a paramount priority for clinical medicine and public health. In 2001, a National Heart, Lung, and Blood Institute workshop concluded that Framingham Risk Score variables—age, male sex, smoking, systolic blood pressure, antihypertensive therapy, and total and high-density lipoprotein (HDL) cholesterol—were key risk factors that could guide clinical decision-making about lipid-lowering therapy for primary ASCVD prevention.1 In 2013, the Pooled Cohort Equations (PCE) used data from more diverse US populations and extended ASCVD risk algorithms to men and women of white and black race.2 Subsequentlyy, the 2013 American College of Cardiology/American Heart Association (ACC/AHA) guidelines recommended using PCE risk categories to guide the intensity of statin therapy.3 The 2018 update to these guidelines continued to recommend the use of the PCE in risk stratification but added other risk enhancing factors, such as chronic kidney or inflammatory diseases, to consider when making decisions about statin therapy for ASCVD prevention.4

    More than 140 million US residents are now prescribed statin therapy,5 and almost half of US adults are potentially eligible to receive statin therapy for ASCVD prevention according to ACC/AHA guidelines.2,6 As the use of statins in the contemporary practice of medicine has become more widespread, it is unclear whether traditional prediction models that do not account for statin use are sufficiently informative for risk assessment. A prior report of national data from the Veterans Health Administration (VA)7 found that the PCE significantly overestimated ASCVD risk, but it did not derive new effect estimates for PCE terms or evaluate the inclusion of statin therapy in risk estimation. Here, we used national VA electronic health record data to examine the performance of the PCE for 5-year ASCVD risk estimation in a contemporary patient cohort and tested the hypothesis that the addition of statin therapy improves model performance.

    Methods
    Research Ethics Statement

    This study was reviewed and approved by the institutional review boards of Emory University and VA Boston Healthcare System. This study was restricted to secondary data analysis, and thus, the requirement for informed consent from participants was waived. This manuscript follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cohort studies.

    Data Sources

    The national VA Corporate Data Warehouse8 was used to create a virtual study cohort of veterans who are regular users of the VA health care system. Data from the VA Corporate Data Warehouse were linked to data from the Centers for Medicaid & Medicare Services and National Death Index databases using scrambled social security numbers.9,10

    Population

    We created a prospective cohort study of incident ASCVD events from national VA data, using previously described methods.11 In brief, this cohort included all VA patients aged at least 18 years with at least 1 primary care visit at a VA facility who had at least 1 outpatient lipid result between 2002 and 2007 and a blood pressure measurement within 30 days of this index lipid testing, a criterion based on our prior work demonstrating that such a restriction limits potential bias within this cohort.11 The date of index lipid determination served as the baseline date of entry to the cohort.

    Individuals were excluded from the cohort if they had a history of HIV infection, cancer, significant kidney or liver disease, schizophrenia, or dementia at baseline (eTable 1 in the Supplement). Given the focus on the performance of the 2013 ACC/AHA PCE, which consists of 4 separate risk models for white and black men and women, we excluded any individual with a different race or ethnicity. In the present analysis of primary ASCVD events, individuals were also excluded if they had baseline history of cardiovascular disease, defined as the presence of a code for any of the following: (1) stroke or cerebrovascular disease, including hemorrhagic stroke, ischemic stroke, and transient ischemic attack (International Classification of Diseases, Ninth Revision [ICD-9] codes 430-438 or meeting the ischemic stroke algorithm described below); (2) coronary heart disease or coronary artery disease, including myocardial infarction (MI), angina, and coronary insufficiency (ICD-9 codes 410-414 or 429.2 or meeting the MI algorithm described below); (3) peripheral vascular disease (ICD-9 codes 440.20-440.4 or 443.9 or history of amputation [eTable 1 in the Supplement]); and (4) congestive heart failure (ICD-9 code 428).

    The eFigure in the Supplement illustrates the cohort creation. Of 8 401 688 patients who accessed VA care from 2002 to 2007, 1 672 336 (19.9%) met the cohort inclusion criteria. The most common reasons for exclusion were absence of a lipid result from 2000 to 2007 (3 713 614 patients [44.2%]) and significant comorbidities, including prevalent CVD (2 349 064 patients [28.0%]).

    Outcomes

    The primary outcomes included time to incident ASCVD event and ASCVD mortality. Incident ASCVD events were defined as a composite outcome of ischemic stroke, MI, and coronary death. To identify stroke and MI, we adapted a phenotyping algorithm designed to identify prevalent cases, using a combination of ICD codes from both VA and Center for Medicare & Medicaid Services data sources, natural language processing, and medical record review labels.12 These phenotyping methods resulted in probability of ischemic stroke and MI for each participant. A predicted probability of ischemic stroke of at least 0.770 and of MI of least 0.765 was defined as a definite case to ensure positive predictive values of at least 90% compared with expert clinician medical record review. Vital status and date of death, if applicable, were derived from the National Death Index. We defined ASCVD mortality as having any of the following ICD-10-CM diagnosis codes as the National Death Index primary cause of death: I10, I11, I13, I16, I20 to I25, I46, I63, I67, I70, I74, I75, and G45.

    Risk Factors for ASCVD

    Guided by the PCE, we considered the following traditional risk factors: age, sex, race, total cholesterol, HDL cholesterol, diabetes, systolic blood pressure, blood pressure treatment, and smoking history, all assessed at the index date. Demographic variables, such as age (difference between the index date and date of birth), race (categorized as black or white), and sex, were derived from administrative data sources within the Corporate Data Warehouse. Diabetes was defined as the use of a diabetes medication before the baseline date (eTable 1 in the Supplement) plus either 2 ICD-9-CM diagnosis codes 250.xx or the use of 1 or more 250.xx codes in combination with a VA primary care visit.13 Blood pressure treatment was defined as an active prescription on the baseline date for at least 1 of the following medication classes, taken as a single agent or in combination formulations: diuretics, angiotensin-converting enzyme inhibitors, angiotensin II receptor inhibitors, α-blockers, β-blockers, and calcium channel blockers. Consistent with the PCE, smoking history was categorized as current or not current (a composite of never and former smoking) with an algorithm developed and validated within the VA medical record data.14 We also assessed the role of baseline statin therapy, defined by an active statin prescription at the time of baseline lipid measurement (eTable 1 in the Supplement).

    Statistical Analysis

    Analysis was performed from June 2016 to March 2020. Baseline descriptive statistics of the study sample, such as the minimum, maximum, median, and mean for each continuous variable and frequency table for each categorical variable, were initially analyzed to summarize the data and detect outliers and missing values. Missing data were uncommon, and individuals with missing data for risk factors were excluded from analysis. We then fitted Cox proportional hazard models to estimate the effect of each PCE risk factor on 5-year and 10-year ASCVD events and ASCVD mortality, with and without baseline statin treatment. Individuals without the primary outcome (composite ASCVD event or ASCVD mortality) were censored 5 or 10 years after baseline in the corresponding analysis. We used a 5-year estimation timeline as our primary analysis because of the large number of ASCVD events observed during that interval and to limit potential misclassification of statin and blood pressure medication treatment status for the participants after the baseline date. We present 10-year models as supplementary analyses. In all models, we categorized total cholesterol in 5 separate clinically relevant strata (50-150 mg/dL, 151-200 mg/dL, 201-250 mg/dL, and >250 mg/dL [to convert to millimoles per liter, multiply by 0.0259]) because of its U-shaped association with mortality observed in this population (Figure 1).

    To evaluate the performance of the PCE in this cohort, we developed separate sex- and race-specific Cox proportional hazards models estimating the 5-year risk of ASCVD and ASCVD mortality. We developed 3 successive models, as follows: (1) the published 2013 ACC/AHA variables (including log transformations and interactions terms) and β coefficients from the 2013 ACC/AHA prediction equations (model 1)2; (2) these variables fit to cohort-specific Cox regression models for white men, black men, white women, and black women separately (model 2); and (3) model 2 plus baseline statin use (model 3). Harrell C statistics and their SDs were calculated to evaluate the ability of the models to separate those who experienced events from those who did not (ie, discrimination).15 To visualize model calibration, we plotted observed 5-year Kaplan-Meier rates according to the deciles of model estimated rates, using previously reported 5-year survival estimates from the original development of the PCE.2,16 Finally, we examined the distribution of patients aged 40 to 79 years in clinically meaningful 5-year ASCVD risk categories (ie, <2.5%, 2.5%-3.74%, 3.75%-10%, >10%) according to the original PCE (model 1) and the cohort-specific model (model 2).

    We used SAS enterprise guide version 7.1 and SAS version 9.4 software (SAS Institute) for these analyses. Statistical significance was set at α = .05. All tests of statistical significance were 2-tailed.

    Results

    In the total eligible cohort of 1 672 336 patients, the mean (SD) baseline age was 58.0 (13.8) years, 1 575 163 patients (94.2%) were men, and 1 383 993 (82.8%) were white individuals. Table 1 shows the baseline characteristics of this cohort, additionally stratified by incident ASCVD event and death. Overall, 331 925 patients (19.9%) were currently smoking at baseline, and 290 092 (17.4%) had diabetes. At baseline, 312 155 (18.7%) had a prescription for statin therapy. The number of participants with a statin prescription at some time during 5 and 10 years of follow-up increased to 899 464 (53.8%) and 1 048 079 (62.7%), respectively. During a mean (SD) follow-up of 4.6 (1.1) years, 66 605 (4.0%) experienced an ASCVD event; during a mean (SD) follow-up of 4.8 (0.7) years, 31 878 (1.9%) experienced ASCVD death. The characteristics of the 1 663 422 participants included in the supplementary analyses of 10-year events, among whom 121 893 (7.3%) experienced an ASCVD event and 74 230 (4.5%) experienced an ASCVD death, appear in eTable 2 in the Supplement.

    Figure 1 shows the unadjusted incidence rates of ASCVD events and ASCVD death in the cohort overall and separately for men and women. Incidence of ASCVD events was 9.02 (95% CI, 8.05-9.09) events per 1000 person-years in men, 3.22 (95% CI, 3.06-3.39) events per 1000 person-years in women, and 8.67 (95% CI, 8.61-8.74) per 1000 person-years in the overall cohort. Incidence of ASCVD death was 4.10 (95% CI, 4.06-4.15) events per 1000 person-years in men, 1.29 (95% CI, 1.19-1.40) events per 1000 person-years in women, and 3.93 (95% CI, 3.89-3.98) per 1000 person-years in the overall cohort. As shown in Table 2, risk factors for 5-year ASCVD event included increasing age (without statin: hazard ratio [HR], 1.23; 95% CI, 1.22-1.23; P < .001; with statin: HR, 1.23; 95% CI, 1.22-1.23; P < .001), male sex (women without statin: HR, 0.63; 95% CI, 0.60-0.66; P < .001; women with statin: HR, 0.63; 95% CI, 0.60-0.66; P < .001), black race (without statin: HR, 1.20; 95% CI, 1.18-1.23; P < .001; with statin: HR, 1.19; 95% CI, 1.17-1.22; P < .001), diabetes (without statin: HR, 1.61; 95% CI, 1.58-1.63; P < .001; with statin: HR, 1.62; 95% CI, 1.59-1.65; P < .001), current smoking (without statin: HR, 1.52; 95% CI, 1.49-1.55; P < .001; with statin: HR, 1.52; 95% CI, 1.49-1.55; P < .001), and antihypertensive therapy (without statin: HR, 1.35; 95% CI, 1.33-1.39; P < .001; with statin: HR, 1.37; 95% CI, 1.35-1.39; P < .001). Each 10–mm Hg increment in systolic blood pressure was associated with a 9% (95% CI, 9%-10%) greater relative risk for ASCVD (P < .001). Compared with the reference range for total cholesterol level of 50 to 150 mg/dL, a level greater than 250 mg/dL was associated with 59% (95% CI, 54%-65%) greater relative risk for ASCVD event in a model not accounting for statin therapy (P < .001). When added to the model, statin therapy was associated with 7% (95% CI, 5%-9%) lower ASCVD relative risk (P < .001).

    Table 2 also shows the results of 5-year prediction models for ASCVD death. Again, risk factors included increasing age (without statin: HR, 1.51; 95% CI, 1.50-1.52; P < .001; with statin: HR, 1.51; 95% CI, 1.50-1.52; P < .001), male sex (women without statin: HR, 0.58; 95% CI, 0.54-0.63; P < .001; with statin: HR, 0.58; 95% CI, 0.54-0.63; P < .001), black race (without statin: HR, 1.20; 95% CI, 1.16-1.24; P < .001; with statin: HR, 1.17; 95% CI, 1.13-1.21; P < .001), diabetes (without statin: HR, 1.32; 95% CI, 1.28-1.35; P < .001; with statin: HR, 1.36; 95% CI, 1.33-1.40; P < .001), current smoking (without statin: HR, 1.24; 95% CI, 1.19-1.28; P < .001; with statin: HR, 1.22; 95% CI, 1.18-1.26; P < .001), and antihypertensive therapy (without statin: HR, 1.39; 95% CI, 1.35-1.41; P < .001; with statin: HR, 1.45; 95% CI, 1.41-1.49; P < .001). Statin therapy at baseline was associated with a 25% (95% CI, 23%-28%) lower relative risk for ASCVD death (P < .001).

    The directions and magnitudes of risk factor effects were similar in stratified 5-year analyses of patients younger than 65 years (eTable 3 in the Supplement) and those aged 65 years or older (eTable 4 in the Supplement). Statin therapy was associated with lower relative risk of ASCVD death in both age strata but was only associated with lower relative risk of composite ASCVD events among the older stratum (<65 years: HR, 1.00; 95% CI, 0.97-1.00; P = .89; ≥65 years: HR, 0.87; 95% CI, 0.84-0.89; P < .001).

    The HRs for the risk factors were similar in the 10-year models for ASCVD events and mortality (eTable 5 in the Supplement) compared with the 5-year models (Table 2). A notable exception was smoking status, which was associated with a higher risk of ASCVD events (HR, 1.68; 95% CI, 1.65-1.70; P < .001) and mortality (HR, 1.74; 95% CI, 1.70-1.77; P < .001) during a 10-year period than during a 5-year period (HR, 1.52; 95% CI, 1.49-1.55; P < .001 and HR, 1.24; 95% CI, 1.19-1.28; P < .001, respectively).

    As shown in Table 3, the use of cohort-specific effect estimates (model 2) and the addition of statin therapy (model 3) did not meaningfully improve discrimination for risk of ASCVD event in any of the 4 sex-race groups examined compared to the PCE model (model 1) (eg, Harrell C statistic [SD] among white men, model 1: 0.66 [0.004]; model 2, 0.67 [0.004]; model 3: 0.67 [0.004]). All models had better discrimination among women than among men (eg, Harrell C statistic [SD] among black women vs black men, model 1: 0.79 [0.036] vs 0.72 [0.007]; model 2: 0.80 [0.030] vs 0.76 [0.007]; model 3: 0.80 [0.029] vs 0.72 [0.007]). These patterns were observed in the overall cohort and the subset aged 40 to 79 years (Table 3).

    Figure 2 illustrates the calibration of models 1 and 2 for 5-year ASCVD estimation among the 4 race-sex strata. In all strata, the PCE model (model 1) overestimated 5-year event rates, with all slopes of observed vs expected event rates deviating considerably from 1. The use of cohort-derived effect estimates (model 2) improved calibration. eTable 6 in the Supplement shows the distribution of patients aged 40 to 79 years in clinically relevant categories of 5-year ASCVD risk by the PCE (model 1) and the cohort-specific model (model 2). In the cohort overall, 211 237 of 1 136 161 white men (18.6%) and 29 634 of 218 463 black men (13.6%) would have an estimated 5-year ASCVD risk of at least 3.75% according to the PCE and thus be potentially eligible for statin therapy when the cohort-specific model categorized their 5-year risk as less than 3.75%. Similarly, 1741 of 44 399 white women (3.9%) and 836 of 16 034 black women (5.2%) were assigned higher risk categories by the PCE compared with the cohort-specific model. The risk calculator for the VA cohort-specific model is publicly avaialble.17

    Discussion

    We have shown that the 2013 PCE overestimated absolute ASCVD risk in a contemporary patient cohort receiving real-world care in a large national health care system and that the use of cohort-specific estimates for PCE risk factors improved model calibration and assignment to clinically meaningful risk categories. Baseline statin therapy was associated with an 8% lower 5-year ASCVD relative risk and 26% lower 5-year relative risk of ASCVD mortality, but the addition of statin therapy to models did not improve discrimination. These results call into question the validity of applying ASCVD prediction models developed in the prestatin era to contemporary ASCVD risk estimation, but they do not necessitate the addition of statin therapy to these models.

    Risk stratification is a critical first step for implementing primary ASCVD prevention, and key interventions include lifestyle modification and pharmacotherapy with statins and aspirin. Statins have demonstrated conclusive benefits for reducing ASCVD events and mortality in randomized clinical trials.18,19 The 2013 Cardiovascular Risk Estimation Guidelines estimated that 10-year ASCVD risk exceeded 7.5% for approximately one-third of US adults and designated such individuals as definite candidates for statin therapy. As a result, more US residents now receive statin therapy.5 Given that the PCE were developed in cohorts from the late 1960s through the early 1990s,20 it is reasonable to evaluate whether the PCE and other similarly dated models remain clinically valid for contemporary risk estimation among patient populations heavily enriched with statin users. The present analyses demonstrated overestimation of ASCVD risk based on PCE. Such an approach in the modern era may lead to inappropriate ASCVD risk estimation and might affect how preventive therapy is instituted.

    The inadequacy of the PCE for ASCVD risk estimation in the present cohort is a matter of poor model calibration, a measure of how closely the absolute event rates estimated by a model match the rates actually observed.21 Since current primary prevention guidelines use categories of absolute ASCVD risk (eg, 7.5%-19.9% 10-year risk) to guide recommendations for statin therapy,22 calibration is an important metric to inform the ASCVD prevention programs in a health care system. We found that the published PCE would significantly overestimate ASCVD risk in the national VA patient population. Approximately 1 in 6 men in this cohort had PCE-estimated ASCVD risk that would make them potentially eligible for statin treatment and cohort-estimated risk that would not. Given this overestimation, we tested the hypothesis and demonstrated that deriving new cohort-specific effect estimates for the original PCE risk factors improved model calibration, correcting this overestimation.

    The present findings from a health care system–derived cohort are consistent with a meta-analysis of data from 360 737 participants in 86 prospective studies in the Emerging Risk Factors Collaboration,23 in which the PCE overestimated ASCVD relative risk by a mean of 41%. Indeed, the PCE overestimated ASCVD in the cohorts used for the original PCE model validation.2,20 Our national results are also consistent with those from more limited health care patient populations from Florida, Illinois, and Minnesota24,25 and from a prior report of national VA data, which found that the PCE estimated 63% more ASCVD events than were observed.7 The present study extends this latter work by using a novel high-throughput phenotyping algorithm to identify ASCVD events, deriving new effect estimates for PCE terms, and evaluating the inclusion of statin therapy in risk estimation. Our demonstration that recalculating cohort-derived β coefficients for PCE terms improves PCE model performance in a specific population is consistent with an analysis of electronic health record data from more than 84 000 adults receiving care at a large health care delivery and insurance organization in Minnesota between 2001 and 2011.25 Another approach to correcting the risk overestimation by the PCE is model recalibration, whereby observed risk in a target population is used to recalculate the model’s overall slope, intercept, or both.20,26,27 Recalibration of the PCE successfully corrected the observed risk overestimation in the Emerging Risk Factors Collaboration and the prior VA study.7,23 However, recalibration may only be appropriate when risk factors have consistent directions of effect between cohorts. The nonmonotonic association we observed between total cholesterol and ASCVD outcomes in this VA cohort favors our approach of reestimating the effects.

    Beyond the derivation of new cohort-specific effect estimates for risk factors in the PCE, it is reasonable to ask whether statin therapy itself should be included in ASCVD models in the modern era. In this contemporary national cohort receiving real-world clinical care, baseline statin therapy in veterans was associated with a clinically significant reduction in 5-year ASCVD events and mortality, consistent with evidence from randomized clinical trials.18,28 We limited our analyses to a 5-year period instead of the 10 years used by the PCE to reduce the potential consequences of incident (ie, drop-in) statin use after baseline, which occurred for 54% of participants within 5 years and for 63% of participants within 10 years. Nevertheless, we observed lower-than-anticipated ASCVD event rates in a contemporary cohort enriched for ASCVD risk factors. Prevalent and drop-in statin use during follow-up in this cohort is likely to be a major determinant of these lower rates and suggests the need for policy makers and clinicians to consider shortening the time horizon of the models used to inform ASCVD prevention. Given that more than half of the participants were prescribed statin therapy within 5 years of baseline, whether a 5-year or even shorter period is the optimal interval remains an open question.

    This research has clinical implications for ASCVD prevention in modern medicine. Before the widespread implementation of electronic health records and clinical databases, clinicians had to rely on ASCVD risk models from external cohort studies to estimate the risk for the patients under their care. It is now becoming increasingly feasible for clinicians practicing in large health care systems to use patient data from their specific settings to estimate ASCVD risk more accurately.25 The present work demonstrates the potential advantage of using risk estimation models derived from one’s own practice setting to guide ASCVD prevention, even in a large diverse health care system spanning all regions of the United States. Our investigation offers insight into how the VA health care system and other large integrated systems might approach precision ASCVD prevention for their patient populations.

    Limitations

    Our work has some limitations to note. First, only 6% of the cohort were women, although this proportion represents almost 100 000 individuals and 1500 ASCVD events. Our models demonstrated better discrimination among women than men, although the 95% CIs around these estimates indicated lower precision. Second, our analysis did not incorporate certain key patient characteristics that define statin eligibility by current guidelines, such as risk-enhancing factors like coronary artery calcium score or shared decision-making.4 Third, we limited our analysis to an examination of the PCE and its risk factors and did not undertake the development and validation of a novel risk prediction model.

    Conclusions

    In this study, we observed lower rates of incident ASCVD events than expected in a contemporary national cohort. The PCE overestimated ASCVD risk, and more than 20% of patients would potentially be eligible for statin therapy based on the PCE but not on a cohort-derived model. In the statin era, health care professionals and systems should base ASCVD risk assessment on models calibrated to their patient populations.

    Back to top
    Article Information

    Accepted for Publication: March 24, 2020.

    Published: July 14, 2020. doi:10.1001/jamanetworkopen.2020.8236

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Vassy JL et al. JAMA Network Open.

    Corresponding Author: Jason L. Vassy, MD, MPH, SM, Veterans Affairs Boston Healthcare System, 150 S Huntington Ave, 152-G, Boston, MA 02130 (jvassy@partners.org).

    Author Contributions: Drs Vassy and Wilson had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Vassy, Ho, Gaziano, Gagnon, Cho, Wilson.

    Acquisition, analysis, or interpretation of data: Vassy, Lu, Ho, Galloway, Raghavan, Honerlaw, Tarko, Russo, Qazi, Orkaby, Tanukonda, Djousse, Gagnon, Cho, Wilson.

    Drafting of the manuscript: Vassy, Lu, Galloway, Cho.

    Critical revision of the manuscript for important intellectual content: Vassy, Ho, Galloway, Raghavan, Honerlaw, Tarko, Russo, Qazi, Orkaby, Tanukonda, Djousse, Gaziano, Gagnon, Cho, Wilson.

    Statistical analysis: Vassy, Lu, Ho, Tarko, Russo, Gagnon.

    Obtained funding: Galloway, Gaziano, Cho, Wilson.

    Administrative, technical, or material support: Vassy, Galloway, Raghavan, Honerlaw, Tanukonda, Cho, Wilson.

    Supervision: Vassy, Gaziano, Cho, Wilson.

    Conflict of Interest Disclosures: Drs Vassy, Ho, Cho, and Wilson reported having a patent pending for a shrinkage method for down-selecting variables for inclusion in risk estimation models. Dr Raghavan reported receiving grant funding from the US Department of Veterans Affairs during the conduct of the study. Dr Orkaby reported receiving grants from the National Institutes of Health and grants from the US Department of Veterans Affairs outside the submitted work. No other disclosures were reported.

    Funding/Support: This research was supported by the US Department of Veterans Affairs Merit Award I01 CX001025 and Career Development Award IK2 CX001262. Support for Veterans Affairs/Centers for Medicare & Medicaid Services data provided by the Department of Veterans Affairs, Veterans Affairs Health Services Research, and Development Service, VA Information Resource Center under project numbers SDR 02-237 and 98-004.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: This article does not represent the views of the US Department of Veterans Affairs or the US government.

    References
    1.
    D’Agostino  RB  Sr, Grundy  S, Sullivan  LM, Wilson  P; CHD Risk Prediction Group.  Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation.   JAMA. 2001;286(2):180-187. doi:10.1001/jama.286.2.180PubMedGoogle ScholarCrossref
    2.
    Goff  DC  Jr, Lloyd-Jones  DM, Bennett  G,  et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.   Circulation. 2014;129(25)(suppl 2):S49-S73. doi:10.1161/01.cir.0000437741.48606.98PubMedGoogle ScholarCrossref
    3.
    Stone  NJ, Robinson  JG, Lichtenstein  AH,  et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.   J Am Coll Cardiol. 2014;63(25 Pt B):2889-2934. doi:10.1016/j.jacc.2013.11.002PubMedGoogle ScholarCrossref
    4.
    Grundy  SM, Stone  NJ, Bailey  AL,  et al.  2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.   Circulation. 2019;139(25):e1082-e1143.PubMedGoogle Scholar
    5.
    Salami  JA, Warraich  H, Valero-Elizondo  J,  et al.  National trends in statin use and expenditures in the US adult population from 2002 to 2013: insights from the Medical Expenditure Panel Survey.   JAMA Cardiol. 2017;2(1):56-65. doi:10.1001/jamacardio.2016.4700PubMedGoogle ScholarCrossref
    6.
    Pencina  MJ, Navar-Boggan  AM, D’Agostino  RB  Sr,  et al.  Application of new cholesterol guidelines to a population-based sample.   N Engl J Med. 2014;370(15):1422-1431. doi:10.1056/NEJMoa1315665PubMedGoogle ScholarCrossref
    7.
    Sussman  JB, Wiitala  WL, Zawistowski  M, Hofer  TP, Bentley  D, Hayward  RA.  The Veterans Affairs Cardiac Risk Score: recalibrating the atherosclerotic cardiovascular disease score for applied use.   Med Care. 2017;55(9):864-870. doi:10.1097/MLR.0000000000000781PubMedGoogle ScholarCrossref
    8.
    Price  LE, Shea  K, Gephart  S.  The Veterans Affairs’s Corporate Data Warehouse: uses and implications for nursing research and practice.   Nurs Adm Q. 2015;39(4):311-318. doi:10.1097/NAQ.0000000000000118PubMedGoogle ScholarCrossref
    9.
    US Department of Veterans Affairs. Frequently Asked Questions—2019 National Veteran Suicide Prevention Annual Report. Published September 20, 2019. Accessed June 4, 2020. https://www.mentalhealth.va.gov/docs/data-sheets/2019/FAQ_2019_Annual_Report_508.pdf
    10.
    US Department of Veterans Affairs. System of records notice 97VA10P1: Consolidated Data Information System-VA. Updated March 3, 2015. Accessed June 4, 2020. https://www.oprm.va.gov/docs/sorn/sorn97va10p1.pdf
    11.
    Vassy  JL, Ho  YL, Honerlaw  J,  et al.  Yield and bias in defining a cohort study baseline from electronic health record data.   J Biomed Inform. 2018;78:54-59. doi:10.1016/j.jbi.2017.12.017PubMedGoogle ScholarCrossref
    12.
    Zhang  Y, Cai  T, Yu  S,  et al.  High-throughput phenotyping with electronic medical record data using a common semi-supervised approach (PheCAP).   Nat Protoc. 2019;14(12):3426-3444. doi:10.1038/s41596-019-0227-6PubMedGoogle ScholarCrossref
    13.
    Raghavan  S, Vassy  JL, Ho  YL,  et al.  Diabetes mellitus-related all-cause and cardiovascular mortality in a national cohort of adults.   J Am Heart Assoc. 2019;8(4):e011295. doi:10.1161/JAHA.118.011295PubMedGoogle Scholar
    14.
    Song  RJ, Ho  Y-L, Nguyen  X-MT,  et al  Abstract 18809: development of an electronic health record-based algorithm for smoking status using the Million Veteran Program (MVP) cohort survey response.   Circulation. 2016;134(suppl 1):A18809-A18809. Accessed May 29, 2020. https://www.ahajournals.org/doi/abs/10.1161/circ.134.suppl_1.18809Google Scholar
    15.
    Pencina  MJ, D’Agostino  RB  Sr, D’Agostino  RB  Jr, Vasan  RS.  Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.   Stat Med. 2008;27(2):157-172. doi:10.1002/sim.2929PubMedGoogle ScholarCrossref
    16.
    Muntner  P, Colantonio  LD, Cushman  M,  et al.  Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations.   JAMA. 2014;311(14):1406-1415. doi:10.1001/jama.2014.2630PubMedGoogle ScholarCrossref
    17.
    VA-ASCVD risk prediction calculator. Updated May 13, 2020. Accessed May 29, 2020. http://ascvd.cardiocalcs.com
    18.
    Baigent  C, Blackwell  L, Emberson  J,  et al; Cholesterol Treatment Trialists’ (CTT) Collaboration.  Efficacy and safety of more intensive lowering of LDL cholesterol: a meta-analysis of data from 170,000 participants in 26 randomised trials.   Lancet. 2010;376(9753):1670-1681. doi:10.1016/S0140-6736(10)61350-5PubMedGoogle ScholarCrossref
    19.
    Ridker  PM, Wilson  PW.  A trial-based approach to statin guidelines.   JAMA. 2013;310(11):1123-1124. doi:10.1001/jama.2013.276529PubMedGoogle ScholarCrossref
    20.
    Cook  NR, Ridker  PM.  Calibration of the Pooled Cohort Equations for atherosclerotic cardiovascular disease: an update.   Ann Intern Med. 2016;165(11):786-794. doi:10.7326/M16-1739PubMedGoogle ScholarCrossref
    21.
    Royston  P, Altman  DG.  External validation of a Cox prognostic model: principles and methods.   BMC Med Res Methodol. 2013;13:33. doi:10.1186/1471-2288-13-33PubMedGoogle ScholarCrossref
    22.
    Pencina  MJ, D’Agostino  RB.  Evaluation of the Framingham risk score in the European Prospective Investigation of Cancer-Norfolk cohort—invited commentary.   Arch Intern Med. 2008;168(11):1216-1218. doi:10.1001/archinte.168.11.1216PubMedGoogle ScholarCrossref
    23.
    Pennells  L, Kaptoge  S, Wood  A,  et al; Emerging Risk Factors Collaboration.  Equalization of four cardiovascular risk algorithms after systematic recalibration: individual-participant meta-analysis of 86 prospective studies.   Eur Heart J. 2019;40(7):621-631. doi:10.1093/eurheartj/ehy653PubMedGoogle ScholarCrossref
    24.
    Emdin  CA, Khera  AV, Natarajan  P,  et al.  Evaluation of the Pooled Cohort Equations for prediction of cardiovascular risk in a contemporary prospective cohort.   Am J Cardiol. 2017;119(6):881-885. doi:10.1016/j.amjcard.2016.11.042PubMedGoogle ScholarCrossref
    25.
    Wolfson  J, Vock  DM, Bandyopadhyay  S,  et al.  Use and customization of risk scores for predicting cardiovascular events using electronic health record data.   J Am Heart Assoc. 2017;6(4):e003670. doi:10.1161/JAHA.116.003670PubMedGoogle Scholar
    26.
    Pencina  MJ, Fine  JP, D’Agostino  RB  Sr.  Discrimination slope and integrated discrimination improvement: properties, relationships and impact of calibration.   Stat Med. 2017;36(28):4482-4490. doi:10.1002/sim.7139PubMedGoogle ScholarCrossref
    27.
    Toll  DB, Janssen  KJ, Vergouwe  Y, Moons  KG.  Validation, updating and impact of clinical prediction rules: a review.   J Clin Epidemiol. 2008;61(11):1085-1094. doi:10.1016/j.jclinepi.2008.04.008PubMedGoogle ScholarCrossref
    28.
    Danesh  J, Lewington  S, Thompson  SG,  et al; Fibrinogen Studies Collaboration.  Plasma fibrinogen level and the risk of major cardiovascular diseases and nonvascular mortality: an individual participant meta-analysis.   JAMA. 2005;294(14):1799-1809.PubMedGoogle Scholar
    ×