Evaluation of the Pooled Cohort Risk Equations for Cardiovascular Risk Prediction in a Multiethnic Cohort From the Women’s Health Initiative | Cardiology | JAMA Internal Medicine | JAMA Network
[Skip to Content]
[Skip to Content Landing]
Figure 1.  Observed vs Predicted Risk by Baseline 10-Year Risk Categories
Observed vs Predicted Risk by Baseline 10-Year Risk Categories

Kaplan-Meier observed unadjusted (dark blue bars) and predicted (light blue bars) 10-year risks of atherosclerotic cardiovascular disease events by baseline 10-year risk categories of less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more before (A) and after (B) weighting to the age and racial/ethnic distribution of the overall WHI cohort. Observed events were WHI-adjudicated events.

Figure 2.  Observed vs Predicted Risk by Baseline 10-Year Risk Categories, Weighted to the Age and Racial Distribution of the Overall Women's Health Initiative (WHI) Cohort
Observed vs Predicted Risk by Baseline 10-Year Risk Categories, Weighted to the Age and Racial Distribution of the Overall Women's Health Initiative (WHI) Cohort

Kaplan-Meier observed unadjusted (blue bars), observed adjusted for statin and aspirin use during follow-up (light blue bars), and predicted (white bars) 10-year risks of atherosclerotic cardiovascular disease events by baseline 10-year risk categories of less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more, weighted to the age and racial/ethnic distribution of the overall WHI cohort. Panel A is for all participants and panel B is after excluding active hormone arm. Observed events were WHI-adjudicated events.

Figure 3.  Observed vs Predicted Risk by Baseline 10-Year Risk Categories in Women 65 Years and Older Enrolled in Medicare A or A/B (n = 6071)
Observed vs Predicted Risk by Baseline 10-Year Risk Categories in Women 65 Years and Older Enrolled in Medicare A or A/B (n = 6071)

In women 65 years and older enrolled in Medicare A or A/B, Kaplan-Meier observed unadjusted (dark blue bars), observed adjusted for statin and aspirin use during follow-up (light blue bars), and predicted (white bars) 10-year risks of atherosclerotic cardiovascular disease events by baseline 10-year risk categories of less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more. Panels A and B are before including Centers for Medicare and Medicaid (CMS) events (A, participants enrolled in Medicare A or A/B fee for service; and B after excluding active hormone therapy arm); panels C and D also excluded active hormone therapy arm but included events ascertained from the CMS, assuming a positive predictive value (ie, proportion of true cases) of 60% for principal discharge diagnosis (C) or 85% for principal or secondary diagnosis (D).

aExcludes hormone therapy.

Table 1.  2013 ACC/AHA Pooled Cohort (n = 19 995) Equations Predicted and Observed Risks, Before and After Weighting to the Overall WHI Cohort (N = 161 808)a
2013 ACC/AHA Pooled Cohort (n = 19 995) Equations Predicted and Observed Risks, Before and After Weighting to the Overall WHI Cohort (N = 161 808)a
Table 2.  Predicted and Observed Risk in Women 65 Years and Older Enrolled in CMS by Racial/Ethnic and Age Subgroups Before and After Accounting for CMS Eventsa
Predicted and Observed Risk in Women 65 Years and Older Enrolled in CMS by Racial/Ethnic and Age Subgroups Before and After Accounting for CMS Eventsa
1.
Goff  DC  Jr, Lloyd-Jones  DM, Bennett  G,  et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  Circulation. 2014;129(25)(suppl 2):S49-S73.PubMedGoogle ScholarCrossref
2.
Stone  NJ, Robinson  JG, Lichtenstein  AH,  et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  Circulation. 2014;129(25)(suppl 2):S1-S45.PubMedGoogle ScholarCrossref
3.
Kavousi  M, Leening  MJ, Nanchen  D,  et al.  Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort.  JAMA. 2014;311(14):1416-1423.PubMedGoogle ScholarCrossref
4.
Cook  NR, Ridker  PM.  Further insight into the cardiovascular risk calculator: the roles of statins, revascularizations, and underascertainment in the Women’s Health Study.  JAMA Intern Med. 2014;174(12):1964-1971.PubMedGoogle ScholarCrossref
5.
DeFilippis  AP, Young  R, Carrubba  CJ,  et al.  An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort.  Ann Intern Med. 2015;162(4):266-275.PubMedGoogle ScholarCrossref
6.
Rana  JS, Tabada  GH, Solomon  MD,  et al.  Accuracy of the atherosclerotic cardiovascular risk equation in a large contemporary, multiethnic population.  J Am Coll Cardiol. 2016;67(18):2118-2130.PubMedGoogle ScholarCrossref
7.
Cook  NR, Ridker  PM.  Calibration of the pooled cohort equations for atherosclerotic cardiovascular disease: An update.  Ann Intern Med. 2016;165(11):786-794.PubMedGoogle ScholarCrossref
8.
The Women’s Health Initiative Study Group.  Design of the Women’s Health Initiative clinical trial and observational study.  Control Clin Trials. 1998;19(1):61-109.PubMedGoogle ScholarCrossref
9.
Manson  JE, Chlebowski  RT, Stefanick  ML,  et al.  Menopausal hormone therapy and health outcomes during the intervention and extended poststopping phases of the Women’s Health Initiative randomized trials.  JAMA. 2013;310(13):1353-1368.PubMedGoogle ScholarCrossref
10.
Hays  J, Hunt  JR, Hubbell  FA,  et al.  The Women’s Health Initiative recruitment methods and results.  Ann Epidemiol. 2003;13(9)(suppl):S18-S77.PubMedGoogle ScholarCrossref
11.
Cook  NR, Ridker  PM.  Response to Comment on the reports of over-estimation of ASCVD risk using the 2013 AHA/ACC risk equation.  Circulation. 2014;129(2):268-269.PubMedGoogle ScholarCrossref
12.
Curb  JD, McTiernan  A, Heckbert  SR,  et al; WHI Morbidity and Mortality Committee.  Outcomes ascertainment and adjudication methods in the Women’s Health Initiative.  Ann Epidemiol. 2003;13(9)(suppl):S122-S128.PubMedGoogle ScholarCrossref
13.
Lakshminarayan  K, Larson  JC, Virnig  B,  et al.  Comparison of Medicare claims versus physician adjudication for identifying stroke outcomes in the Women’s Health Initiative.  Stroke. 2014;45(3):815-821.PubMedGoogle ScholarCrossref
14.
Hlatky  MA, Ray  RM, Burwen  DR,  et al.  Use of Medicare data to identify coronary heart disease outcomes in the Women’s Health Initiative.  Circ Cardiovasc Qual Outcomes. 2014;7(1):157-162.PubMedGoogle ScholarCrossref
15.
Lemeshow  S, Hosmer  DW  Jr.  A review of goodness of fit statistics for use in the development of logistic regression models.  Am J Epidemiol. 1982;115(1):92-106.PubMedGoogle ScholarCrossref
16.
Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387.PubMedGoogle ScholarCrossref
17.
Ridker  PM, Cook  NR.  Statins: new American guidelines for prevention of cardiovascular disease.  Lancet. 2013;382(9907):1762-1765.PubMedGoogle ScholarCrossref
18.
Poses  RM, Cebul  RD, Collins  M, Fager  SS.  The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis.  Ann Intern Med. 1986;105(4):586-591.PubMedGoogle ScholarCrossref
19.
Muntner  P, Colantonio  LD, Cushman  M,  et al.  Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations.  JAMA. 2014;311(14):1406-1415.PubMedGoogle ScholarCrossref
20.
Ridker  PM, Buring  JE, Rifai  N, Cook  NR.  Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score.  JAMA. 2007;297(6):611-619.PubMedGoogle ScholarCrossref
21.
D’Agostino  RB  Sr, Vasan  RS, Pencina  MJ,  et al.  General cardiovascular risk profile for use in primary care: the Framingham Heart Study.  Circulation. 2008;117(6):743-753.PubMedGoogle ScholarCrossref
22.
D’Agostino  RB  Sr, Grundy  S, Sullivan  LM, Wilson  P; CHD Risk Prediction Group.  Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation.  JAMA. 2001;286(2):180-187.PubMedGoogle ScholarCrossref
23.
Steyerberg  EW, Borsboom  GJ, van Houwelingen  HC, Eijkemans  MJ, Habbema  JD.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage.  Stat Med. 2004;23(16):2567-2586.PubMedGoogle ScholarCrossref
24.
Psaty  BM, Delaney  JA, Arnold  AM,  et al.  Study of cardiovascular health outcomes in the era of claims data: The Cardiovascular Health Study.  Circulation. 2016;133(2):156-164.PubMedGoogle ScholarCrossref
25.
White  AD, Folsom  AR, Chambless  LE,  et al.  Community surveillance of coronary heart disease in the Atherosclerosis Risk in Communities (ARIC) Study: methods and initial two years’ experience.  J Clin Epidemiol. 1996;49(2):223-233.PubMedGoogle ScholarCrossref
26.
Heckbert  SR, Kooperberg  C, Safford  MM,  et al.  Comparison of self-report, hospital discharge codes, and adjudication of cardiovascular events in the Women’s Health Initiative.  Am J Epidemiol. 2004;160(12):1152-1158.PubMedGoogle ScholarCrossref
27.
Yasaitis  LC, Berkman  LF, Chandra  A.  Comparison of self-reported and Medicare claims-identified acute myocardial infarction.  Circulation. 2015;131(17):1477-1485.PubMedGoogle ScholarCrossref
28.
Ness  RB; Joint Policy Committee, Societies of Epidemiology.  Influence of the HIPAA Privacy Rule on health research.  JAMA. 2007;298(18):2164-2170.PubMedGoogle ScholarCrossref
29.
Houser  SH, Howard  VJ, Hovater  MK, Safford  MM.  Obtaining medical records from healthcare facilities under the HIPAA Privacy Rule: the experience of a national longitudinal cohort study.  Neuroepidemiology. 2007;28(3):162-168.PubMedGoogle ScholarCrossref
30.
Colantonio  LD, Richman  JS, Carson  AP,  et al.  Performance of the atherosclerotic cardiovascular disease Pooled Cohort risk equations by social deprivation status.  J Am Heart Assoc. 2017;6(3):e005676. doi:10.1161/JAHA.117.005676PubMedGoogle ScholarCrossref
31.
Yadlowsky  S, Hayward  RA, Sussman  JB,  et al.  Clinical implications of revised pooled cohort equations for estimating atherosclerotic cardiovascular disease risk [published online June 5, 2018].  Ann Intern Med. doi:10.7326/M17-3011.Google Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    September 2018

    Evaluation of the Pooled Cohort Risk Equations for Cardiovascular Risk Prediction in a Multiethnic Cohort From the Women’s Health Initiative

    Author Affiliations
    • 1Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
    • 2Emory University School of Medicine, Atlanta, Georgia
    • 3Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
    • 4Fred Hutchinson Cancer Research Center, Seattle, Washington
    • 5MedStar Health Research Institute, Hyattsville, Maryland
    • 6University of Florida College of Medicine, Gainesville, Gainesville
    • 7Brown University, Providence, Rhode Island
    • 8HealthPartners Institute, Minneapolis, Minnesota
    • 9George Washington University, Washington, DC
    • 10University of Iowa, Iowa City
    • 11National Heart, Lung, and Blood Institute, Bethesda, Maryland
    • 12Weill Cornell Medicine, New York, New York
    JAMA Intern Med. 2018;178(9):1231-1240. doi:10.1001/jamainternmed.2018.2875
    Key Points

    Question  Do the pooled cohort equations (PCEs) accurately predict 10-year risk of atherosclerotic cardiovascular disease (ASCVD) among participants in the Women’s Health Initiative?

    Findings  In this cohort study of 19 995 women, observed risks were generally lower than predicted by PCE; adjustment for changes in statin and aspirin use resulted in small increases in observed risks. However, there was a considerable impact of adding events identified by Centers for Medicare and Medicaid Services (CMS) surveillance but not self-reported, and the PCE models discriminated risk well.

    Meaning  Observed and predicted ASCVD risks were better aligned after including CMS events.

    Abstract

    Importance  Atherosclerotic cardiovascular disease (ASCVD) kills approximately 1 in every 3 US women. Current cholesterol, hypertension, and aspirin guidelines recommend calculating 10-year risk of ASCVD using the 2013 Pooled Cohort Equations (PCE). However, numerous studies have reported apparent overestimation of risk with the PCE, and reasons for overestimation are unclear.

    Objective  We evaluated the predictive accuracy of the PCE in the Women’s Health Initiative (WHI), a multiethnic cohort of contemporary US postmenopausal women. We evaluated the effects of time-varying treatments such as aspirin and statins, and ascertainment of additional ASCVD events by linkage with the Centers for Medicare and Medicaid Services (CMS) claims.

    Design, Setting, and Participants  The WHI recruited the largest number of US women (n = 161 808) with the racial/ethnic, geographic, and age diversity of the general population (1993-1998). For this study, we included women aged 50 to 79 (n = 19 995) participating in the WHI with data on the risk equation variables at baseline and who met the guideline inclusion and exclusion criteria. Median follow-up was 10 years.

    Main Outcomes and Measures  For this study, ASCVD was defined as myocardial infarction, stroke, or cardiovascular death.

    Results  Among the 19 995 women (mean [SD] age, 64 [7.3] years; 8305 [41.5%] white, 7688 [38.5%] black, 3491 [17.5%] Hispanic, 103 [0.5%] American Indian, 321 [1.6%] Asian/Pacific Islander, and 87 [0.4%] other/unknown), a total of 1236 ASCVD events occurred in 10 years and were adjudicated through medical record review by WHI investigators. The WHI-adjudicated observed risks were lower than predicted. The observed (predicted) risks for baseline 10-year risk categories less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more were 1.7 (2.8), 4.4 (6.2), 5.3 (8.7), and 12.4 (18.2), respectively. Small changes were noted after adjusting for time-dependent changes in statin and aspirin use. Among women 65 years or older enrolled in Medicare, WHI-adjudicated risks were also lower than predicted, but observed (predicted) risks became aligned after including events ascertained by linkage with CMS for additional surveillance for events: 3.8 (4.3), 7.1 (6.4), 8.3 (8.7), and 18.9 (18.7), respectively. Similar results were seen across ethnic/racial groups. Overall, the equations discriminated risk well (C statistic, 0.726; 95% CI, 0.714-0.738).

    Conclusions and Relevance  Without including surveillance for ASCVD events using CMS, observed risks in the WHI were lower than predicted by PCE as noted in several other US cohorts, but risks were better aligned after including CMS events.

    Trial Registration  ClinicalTrials.gov identifier: NCT00000611

    Introduction

    Atherosclerotic cardiovascular disease (ASCVD) kills approximately 1 in every 3 US women, exceeding deaths from all forms of cancer combined.1 Guidelines recommend assessing ASCVD risk, and targeting the intensity of preventive interventions to the risk level.2 In 2013, the American College of Cardiology (ACC) and American Heart Association (AHA) risk assessment guidelines developed new Pooled Cohort Equations (PCE) that estimate 10-year risk of ASCVD, with 4 separate equations for African American and white women and men.1 The PCE were derived from the Atherosclerosis Risk in Communities (ARIC) study, the Cardiovascular Health Study (CHS), the Coronary Artery Risk Development in Young Adults (CARDIA) study, and the Framingham Original and Offspring cohorts.1 The 2013 ACC/AHA cholesterol guidelines recommended using these equations to inform discussions about initiating statin treatment for adults without clinical ASCVD,2 and more recently the 2017 ACC/AHA hypertension guidelines necessitate them for blood pressure targets and treatment recommendations. Since the publication of the PCE, numerous studies have reported apparent overestimation of risk in both US and European populations.3-7 Critics have attributed risk overestimation in these validation studies to the study populations being highly selected and nongeneralizable, increasing use of concomitant preventive therapies such as aspirin and statins, decreasing ASCVD event rates, and underascertainment of event rates in more recent cohorts, which usually rely on participant self-report for identifying potential events.

    To date, the Women’s Health Initiative (WHI) has recruited the largest number of US women (n = 161 808) with the racial/ethnic, geographic, and age diversity of the general population, with the caveat that women had to meet the study eligibility criteria and be willing to participate in a long-term study. Therefore, we evaluated the accuracy of the PCE for predicting 10-year risk of ASCVD in 19 995 women from the WHI who had complete data on the risk equation variables, and weighted results to the age and race/ethnicity distribution of the overall WHI (n = 161 808).8 We evaluated the effects of time-varying treatments such as aspirin and statins, and ascertainment of additional ASCVD events by linkage with the Centers for Medicare and Medicaid Services (CMS) claims.

    Methods
    Study Design

    Study participants were drawn from the WHI that included postmenopausal women aged 50 to 79 years who were enrolled (1993-1998) at 1 of 40 WHI clinical centers nationwide into either a clinical trial (CT; n = 68 132) or an observational study (OS; n = 93 676).8,9 The WHI enrolled racial/ethnic minority groups proportionate to the total general US population in these age groups. Recruitment and baseline data collection have been previously reported (eMethods in the Supplement).8,10 Institutional review board approval was obtained at each center and all participants provided written informed consent.

    Statin and aspirin use was collected at baseline and year 3 for OS participants, and at baseline, years 1, 3, 6, and 9 for CT participants, and at study close-out for all participants.11 Of the total 161 808 women in WHI, a subset of 23 146 had baseline lipid measurements (eFigures 1 and 2 in the Supplement). Of these women, a total of 20 431 women had complete information on the components of the PCE. Because the exclusion/inclusion criteria that were used to develop the PCE (ACC/AHA risk assessment guidelines1) differed from the criteria used to determine statin eligibility (ACC/AHA cholesterol guidelines2), we identified a total of 19 995 WHI participants in 2 subcohorts (eFigure 1 in the Supplement) based on the criteria from these 2 guidelines1,2: (1) WHI risk validation subcohort (n = 19 581) that met the criteria for the risk assessment guidelines,1 and (2) WHI treatment eligibility subcohort (n = 13 439) that met the criteria for the cholesterol guidelines.2 As per the guidelines, the risk validation subcohort excluded women with a medical history of heart failure or atrial fibrillation,1 and the treatment eligibility subcohort excluded women with baseline statin use, LDL cholesterol levels less than 70 mg/dL, or those for whom statins were recommended2: clinical ASCVD (history of MI, stroke, peripheral arterial disease, arterial revascularization, angina, transient ischemic attack), LDL cholesterol levels of 190 mg/dL or higher, or diabetes (to convert LDL cholesterol measure to millimoles per liter, multiply by 0.0259). Women older than 75 years were excluded from the treatment eligibility cohort to be consistent with the cholesterol guidelines.2 All analyses were run separately in each subcohort.

    Outcomes

    The first ASCVD event was defined as the first occurrence of nonfatal myocardial infarction (MI), coronary heart disease (CHD) death, or fatal or nonfatal stroke.

    WHI-Adjudicated Events

    Annual questionnaires were completed by participants or their proxies who were asked if they were hospitalized since their last report, and medical records were obtained and adjudicated by trained physicians using standard criteria as described previously.12,13 In addition, even if an MI or stroke was not specifically reported or if the hospitalization was for a noncardiovascular reason, WHI inquired about and reviewed all hospitalization records for potential cardiovascular-related outcomes (eMethods in the Supplement). The WHI also ascertained deaths obtained from periodic searches of the National Death Index (NDI), to complement routine follow-up of reports of deaths by next of kin and postal authorities.

    Claims-Based Events

    Data from WHI have been linked to Medicare files from the CMS, expanding the data available on WHI participants 65 years and older, and this approach has been previously used for MI14 and stroke13 in the WHI. These claims-based events were included in the specified analyses for women 65 years or older enrolled in Medicare A or A/B (eMethods in the Supplement).

    Statistical Analyses

    Predicted rates were calculated based on the mean predicted ASCVD rates from the PCE, using the recommended race-specific equations.1 Participants were censored at the time of first occurrence of ASCVD, death, time of last follow-up, or at 12 years of follow-up, unless otherwise specified. Observed rates of ASCVD were adjusted for variable follow-up time using Kaplan-Meier survival analysis. We also plotted the average predicted risk within the clinical risk categories based on cut points of 5.0%, 7.5%, and 10%. Results were reported first as unweighted risks, and to reflect the overall WHI population, results were also weighted using inverse probability weighting to the racial/ethnic and age distribution of the overall WHI cohort (n = 161 808; eTable 1A in the Supplement). Model calibration and discrimination were assessed (eMethods in the Supplement).15,16 We assessed the impact of time-dependent statin and aspirin use during follow-up (eMethods in the Supplement).4

    Furthermore, as a subset of the CT participants were randomly allocated to the active hormone therapy arm (conjugated equine estrogens [CEE] 0.625 mg/d alone in women with prior hysterectomy for a median intervention phase of 7.2 years, or CEE plus medroxyprogesterone acetate 2.5 mg/d in women with intact uterus for a median intervention phase of 5.6 years) after the baseline assessment of lipids and other risk factors, we prespecified conducting analyses that excluded them owing to potential alterations in lipids and risk factors following initiation of hormone therapy.

    Finally, to assess the impact of more complete outcome ascertainment on the observed risks, we also conducted prespecified analyses that incorporated the CMS claims outcomes data available on participants 65 years or older at study entry who were enrolled in traditional fee-for-service Medicare Part A or A/B (eMethods in the Supplement).13,14

    Results

    Compared with the overall WHI, current study participants were of similar age, with more than 1 in 5 women 70 years or older, but were more racially diverse owing to oversampling by design of nonwhite participants (eTable 1B in the Supplement). Among the 19 995 women in the analytic sample (either the risk validation or treatment eligibility subcohorts), a total of 1236 WHI-adjudicated first ASCVD events (including 35 NDI deaths) occurred in 10 years (eMethods in the Supplement).

    In unweighted analyses, the average 10-year WHI-adjudicated observed risks were lower than the predicted risks (Table 1) (Figure 1A) (eTable 2 in the Supplement). The observed (predicted) incident rates of events for women in the risk validation subcohort with baseline 10-year predicted risk less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or higher were: 1.7 (2.8), 4.4 (6.2), 5.3 (8.7), and 12.4 (18.2), respectively. The overall C statistic was 0.726 (95% CI, 0.714-0.738), Hosmer-Lemeshow calibration χ2 was 250.42 (P < .001). After weighting to the age and racial/ethnic distribution of the overall WHI, predicted rates remained higher than observed (Figure 1B): 1.4 (2.7), 4.7 (6.2), 5.5 (8.7), and 12.7 (17.4), respectively. Generally similar results were obtained among the treatment eligibility subcohort (Table 1) (eTable 2 in the Supplement).

    At baseline, statin and aspirin use in the WHI risk validation subcohort was low (6.7% and 16.6%, respectively), differed by baseline risk category, and increased to 20% to greater than 40% during follow-up among the various risk categories (eTable 1B and eFigure 3 in the Supplement). Adjusting for the time-varying change in statin and aspirin use resulted in a slight increase in observed risks (Figure 2A) (eTable 3 in the Supplement). In another prespecified analysis, we excluded women who were assigned to the active hormone therapy arm, which only minimally lowered the observed risks (Figure 2B) (eTable 3 in the Supplement).

    Among women 65 years or older who were enrolled in Medicare A or A/B, CMS claims data identified an additional 64 cases of MI and 99 cases of stroke that were not identified by self-report (eMethods in the Supplement). After including 85% of the additional CMS cases that were missed by self-report (eMethods in the Supplement), predicted risks became closely aligned with observed risks (Figure 3) (eTable 4A in the Supplement). For example, among the risk validation subcohort with the additional CMS events, the observed (predicted) risks excluding active hormone arm and adjusting for statin/aspirin use were: 3.8 (4.3), 7.1 (6.4), 8.3 (8.7), and 18.9 (18.7). Results differed minimally in a sensitivity analysis assuming 60% positive predictive value and only the primary CMS position, or when restricting analyses to the subgroup of women randomized to placebo in the hormone therapy CT (eTable 4B in the Supplement).

    Adding the CMS events also improved both model calibrations: Hosmer-Lemeshow calibration χ2 improved from 91.870 (P < .001 [not calibrated]) to 3.32 (P = .19 [well calibrated]) (eTable 5 in the Supplement). Similar findings were observed for the treatment eligibility subcohort (Table 2).

    Among these women, we also analyzed risks by racial/ethnic groups (Table 2) (eFigure 4 in the Supplement) and age categories (Table 2). In all subgroups, predicted risks were initially higher than observed when excluding CMS events, but predicted risks became aligned with observed risks among non-Hispanic whites after also including the CMS events. The effects of adding CMS events were more variable among minorities.

    eTable 5 in the Supplement shows the proportion of WHI women by racial/ethnic groups who would be potentially eligible for statin treatment based on the 2013 ACC/AHA cholesterol guidelines.2

    Discussion

    Without including surveillance for ASCVD events using CMS linkage, observed risks of ASCVD were lower than predicted in the WHI, as noted in several other US cohorts. Adjustment for time-dependent changes in statin or aspirin treatment during the study follow-up resulted in small increases in observed risks. In contrast, there was a considerable impact of adding events that were identified by CMS surveillance but not self-reported, and the observed risks became better aligned with predicted risks after including CMS events. The PCE models discriminated risk well.

    The external validation studies that evaluated the PCE have generated considerable controversy around the predictive accuracy of these equations in contemporary cohorts. At the same time, these recent studies have been criticized for having overall event rates that were much lower than noted in the original PCE cohorts. Overestimation of risk (ie, predicted > observed risk) was noted in studies that enrolled lower-risk health care professionals in clinical trials of aspirin and did not include CMS events.17 Overestimation of risk was also noted in the Rotterdam study,3 the Multi-Ethnic Study of Atherosclerosis,5 and a selected subpopulation of Kaiser Permanente,6 among others.7 How much of the discrepancy can be attributed to changes in ASCVD rates over time (eg, the decreasing ASCVD death rates since the 1970s) vs other factors discussed below has been debated.7

    Risk scores may not perform as well in a new setting as in the original setting, which may be owing to differences in disease prevalence, or to the risk equation not reliably discriminating cases from noncases.18 We found reasonably good discrimination with the PCE in the WHI (C statistic, 0.726), consistent with other studies (C statistics of 0.71-0.82 in the original Pooled Cohorts,1 0.68-0.74 in the Contemporary Pooled Cohorts,10 0.71-0.72 in the Reasons for Geographic and Racial Differences in Stroke [REGARDS] study,19 and 0.71 in the MESA study5), and comparable to other scores.20-22

    Because discrimination was not the issue affecting the performance of the equations, the differences in the predicted:observed rates may result from apparent or real differences in disease prevalence.18 This issue is well-recognized in the literature, where it is recommended that scores should be recalibrated if incidence rates differ substantially in the new settings.23 It is well-known that CVD incidence rates have been declining, which suggests that it would be appropriate to align the predicted rates with more contemporary cohorts as long as they have comprehensive surveillance for events. In the WHI, we found that the predicted:observed rates were generally well-aligned after accounting for outcomes ascertained by linkage with CMS for capture of additional events. Similarly, in the REGARDS study, the PCE were well calibrated in the group of participants for whom use of the PCE are recommended when CMS events were included.19 Neither the WHI nor REGARDS studies sought records for events identified only through CMS.

    The intensity of surveillance for events directly determines observed event rates.24 Data from studies with comprehensive surveillance methods were used to create the PCE, as several PCE cohorts were designed to monitor incidence rates over time. The original PCE cohorts used self-reporting, but also actively searched for events using multiple labor-intensive methods to detect events that were not self-reported. These methods included actively searching hospitalization and emergency department records of hospitals in their communities (eg, Framingham, ARIC), linkage with CMS (eg, CHS), and additional surveillance methods (eg, familial and social contacts, obituary surveillance, review of emergency department records, hospital death records, and local field center investigations). Event ascertainment in these studies also included a review of medical records for these events that were identified through other surveillance mechanisms. For example, ARIC implemented a community surveillance component that had near complete (almost 100%) event ascertainment.25 Studies have found that self-report of ASCVD captures approximately 60% to 75% of events.24,26,27 Hence, relying on self-report alone without more complete surveillance for events could miss a sizeable proportion of events.

    Furthermore, the Privacy Rule of the Health Insurance Portability and Accountability Act (HIPAA), which was passed in 1996 and implemented in 2003 (and hence would not have affected the original PCE cohorts), would have impacted more contemporary studies.28 Post-HIPAA, hospitals and institutions have many deterrents to providing medical records to research studies (eg, fear of litigation, extra labor involved for retrieval of medical records, greater number of research studies). Future research and policy efforts should address HIPAA implications on contemporaneous event ascertainment and cost of conducting research.29

    Volunteers for prevention or screening trials are healthier and have lower mortality than the general population. Hence it is important that clinicians are aware of the baseline prevalence of ASCVD in their population and other relevant characteristics that may influence the pretest probability (eg, socioeconomic status, lifestyle). Overprediction of risk may be more pronounced in income strata such as individuals with higher socioeconomic status.30 Much of the difference by cohort should be captured in the risk factors included in the equations; if not,18 this suggests that the equations may be improved by including additional factors. A recently proposed revision of the pooled cohort equations using more contemporary cohort data and new derivation methods appeared to improve the accuracy of ASCVD risk estimates31 but will require additional validation, including assessment of accuracy with and without inclusion of CMS-identified events.

    We also assessed the influence of therapies, in particular statin and aspirin use, on the predicted:observed rates, as more widespread use of these therapies in more contemporary populations has been proposed as a possible explanation for the overestimation of risk noted in other studies. However, we did not find a considerable impact of statin or aspirin use, consistent with prior reports.4

    Strengths and Limitations

    Strengths of the study include the large sample size and number of ASCVD events that occurred in this racially and ethnically diverse population that was recruited to represent the racial/ethnic distribution of US postmenopausal women. Whereas the PCE cohorts included a total of 13 881 women (1192 ASCVD),1 the current study included a greater number (19 995; 1236 events), with more ethnic and geographic diversity. Additional strengths include the long follow-up and the well-characterized risk factor data, including clinical measurements of blood pressure and detailed information regarding medication use. The complementary surveillance of MI and stroke events from CMS claims allowed for more complete ascertainment of events that were not self-reported. For example, this analysis differed from the previously reported results from the WHI Observational Study17 and the other studies mentioned herein by including a larger number of women, incorporating CMS events, and using the PCE inclusion/exclusion criteria. The CMS analyses depend on assumptions about the reliability of CMS reported endpoints, especially among those without matching medical records. Whereas predictive values may be reasonable overall, they may be less or more reliable for CMS events that were not self reported or lacked medical records sufficient for adjudication.

    The current results may not be generalizable to men or other patient groups. In the risk validation subcohort, 29% of women were randomized to the active hormone arm, which would not have been reflected in their baseline risk factor measurements. However, our prespecified analyses that excluded these women showed no significant impact on the results. We cannot rule out a healthy volunteer effect, though this holds for other cohorts.

    Conclusions

    Without including surveillance for ASCVD events using CMS, observed risks of ASCVD were generally lower than predicted by the PCE in this large multiethnic cohort, but observed and predicted risks were better aligned after including CMS outcomes.

    Back to top
    Article Information

    Corresponding Author: Samia Mora, MD, Center for Lipid Metabolomics, Divisions of Preventive and Cardiovascular Medicine, Brigham and Women’s Hospital, 900 Commonwealth Ave East, Boston, MA 02215 (smora@bwh.harvard.edu).

    Accepted for Publication: May 5, 2018.

    Published Online: July 23, 2018. doi:10.1001/jamainternmed.2018.2875

    Author Contributions: Drs Mora and Liu had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Study concept and design: Mora, Wenger, S. Liu, Rossouw, Safford.

    Acquisition, analysis, or interpretation of data: Mora, Wenger, Cook, J. Liu, Howard, Limacher, S. Liu, Margolis, Martin, Paynter, Ridker, Robinson, Safford.

    Drafting of the manuscript: Mora.

    Critical revision of the manuscript for important intellectual content: Mora, Wenger, Cook, J. Liu, Howard, Limacher, S. Liu, Margolis, Martin, Paynter, Ridker, Robinson, Rossouw, Safford.

    Statistical analysis: Mora, Cook, J. Liu, S. Liu, Paynter.

    Obtained funding: Howard, Rossouw.

    Administrative, technical, or material support: Mora, S. Liu, Safford.

    Study supervision: Mora, Wenger, Howard, Safford.

    Conflict of Interest Disclosures: Dr Mora has received research grant support from the National Heart, Lung, and Blood Institute (NHLBI) (HL136852 and HL134811), and Atherotech Diagnostics for research outside the current work, and received personal fees for scientific advisory board participation for Amgen, Lilly, Pfizer, and Quest Diagnostics. Dr Wenger has received research grants/contracts/trial steering from Alnylam Pharmaceuticals, Gilead Sciences, and NHLBI, served on committee/trial data safety and monitoring board for Pfizer and the Society for Women’s Health Research, and as a consultant to Amgen, AstraZeneca, Gilead Sciences, and Merck. Dr Limacher has National Institutes of Health grant support for the Women’s Health Institute Regional Centers N01 WH04354 and R18 HL112720. Dr Ridker received research grant support from AstraZeneca, Kowa, Novartis, Amgen, Pfizer, and NHLBI, and is listed as a coinventor on patents held by the Brigham and Women's Hospital related to the use of inflammatory biomarkers in CVD (licensed to AstraZeneca and Siemens). Dr Robinson has received institutional research grants from Amarin, Amgen, Astra-Zeneca, Eli Lilly, Esai, Esperion, Glaxo-Smith Kline, Merck, Pfizer, Regeneron/Sanofi, and Takeda, and served as consultant to Akcea/Ionis, Amgen, Dr Reddy, Eli Lilly, Esperion, Merck, Pfizer, and Regeneron/Sanofi. The remaining authors have no disclosures.

    Funding/Support: The WHI Program is funded by the NHLBI, National Institutes of Health, Department of Health and Human Services through contracts N01WH22110, 24152, 32100-2, 32105-6, 32108-9, 32111-13, 32115, 32118-32119, 32122, 42107-26, 42129-32, 44221,HHSN268201600018C, HHSN268201600001C, HHSN268201600002C, HHSN268201600003C, and HHSN268201600004C. Dr Mora received funding through the NHLBI K24 HL136852 and HL134811.

    Role of the Funder/Sponsor: The Women’s Health Institute (WHI) Project Office at the National Heart, Lung, and Blood Institute (NHLBI) had no role in the design and conduct of this study or the interpretation of the data. The opinions expressed in the manuscript are those of the study authors and do not necessarily represent the views of the Department of Health and Human Services/National Institutes of Health.

    Additional Contributions: We acknowledge the dedicated efforts of investigators and staff at the WHI clinical centers, the WHI Clinical Coordinating Center, and the National Heart, Lung and Blood (NHLBI) program office (listing available at http://www.whi.org). We also thank the WHI participants for their extraordinary commitment to the WHI program.

    References
    1.
    Goff  DC  Jr, Lloyd-Jones  DM, Bennett  G,  et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  Circulation. 2014;129(25)(suppl 2):S49-S73.PubMedGoogle ScholarCrossref
    2.
    Stone  NJ, Robinson  JG, Lichtenstein  AH,  et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  Circulation. 2014;129(25)(suppl 2):S1-S45.PubMedGoogle ScholarCrossref
    3.
    Kavousi  M, Leening  MJ, Nanchen  D,  et al.  Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort.  JAMA. 2014;311(14):1416-1423.PubMedGoogle ScholarCrossref
    4.
    Cook  NR, Ridker  PM.  Further insight into the cardiovascular risk calculator: the roles of statins, revascularizations, and underascertainment in the Women’s Health Study.  JAMA Intern Med. 2014;174(12):1964-1971.PubMedGoogle ScholarCrossref
    5.
    DeFilippis  AP, Young  R, Carrubba  CJ,  et al.  An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort.  Ann Intern Med. 2015;162(4):266-275.PubMedGoogle ScholarCrossref
    6.
    Rana  JS, Tabada  GH, Solomon  MD,  et al.  Accuracy of the atherosclerotic cardiovascular risk equation in a large contemporary, multiethnic population.  J Am Coll Cardiol. 2016;67(18):2118-2130.PubMedGoogle ScholarCrossref
    7.
    Cook  NR, Ridker  PM.  Calibration of the pooled cohort equations for atherosclerotic cardiovascular disease: An update.  Ann Intern Med. 2016;165(11):786-794.PubMedGoogle ScholarCrossref
    8.
    The Women’s Health Initiative Study Group.  Design of the Women’s Health Initiative clinical trial and observational study.  Control Clin Trials. 1998;19(1):61-109.PubMedGoogle ScholarCrossref
    9.
    Manson  JE, Chlebowski  RT, Stefanick  ML,  et al.  Menopausal hormone therapy and health outcomes during the intervention and extended poststopping phases of the Women’s Health Initiative randomized trials.  JAMA. 2013;310(13):1353-1368.PubMedGoogle ScholarCrossref
    10.
    Hays  J, Hunt  JR, Hubbell  FA,  et al.  The Women’s Health Initiative recruitment methods and results.  Ann Epidemiol. 2003;13(9)(suppl):S18-S77.PubMedGoogle ScholarCrossref
    11.
    Cook  NR, Ridker  PM.  Response to Comment on the reports of over-estimation of ASCVD risk using the 2013 AHA/ACC risk equation.  Circulation. 2014;129(2):268-269.PubMedGoogle ScholarCrossref
    12.
    Curb  JD, McTiernan  A, Heckbert  SR,  et al; WHI Morbidity and Mortality Committee.  Outcomes ascertainment and adjudication methods in the Women’s Health Initiative.  Ann Epidemiol. 2003;13(9)(suppl):S122-S128.PubMedGoogle ScholarCrossref
    13.
    Lakshminarayan  K, Larson  JC, Virnig  B,  et al.  Comparison of Medicare claims versus physician adjudication for identifying stroke outcomes in the Women’s Health Initiative.  Stroke. 2014;45(3):815-821.PubMedGoogle ScholarCrossref
    14.
    Hlatky  MA, Ray  RM, Burwen  DR,  et al.  Use of Medicare data to identify coronary heart disease outcomes in the Women’s Health Initiative.  Circ Cardiovasc Qual Outcomes. 2014;7(1):157-162.PubMedGoogle ScholarCrossref
    15.
    Lemeshow  S, Hosmer  DW  Jr.  A review of goodness of fit statistics for use in the development of logistic regression models.  Am J Epidemiol. 1982;115(1):92-106.PubMedGoogle ScholarCrossref
    16.
    Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387.PubMedGoogle ScholarCrossref
    17.
    Ridker  PM, Cook  NR.  Statins: new American guidelines for prevention of cardiovascular disease.  Lancet. 2013;382(9907):1762-1765.PubMedGoogle ScholarCrossref
    18.
    Poses  RM, Cebul  RD, Collins  M, Fager  SS.  The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis.  Ann Intern Med. 1986;105(4):586-591.PubMedGoogle ScholarCrossref
    19.
    Muntner  P, Colantonio  LD, Cushman  M,  et al.  Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations.  JAMA. 2014;311(14):1406-1415.PubMedGoogle ScholarCrossref
    20.
    Ridker  PM, Buring  JE, Rifai  N, Cook  NR.  Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score.  JAMA. 2007;297(6):611-619.PubMedGoogle ScholarCrossref
    21.
    D’Agostino  RB  Sr, Vasan  RS, Pencina  MJ,  et al.  General cardiovascular risk profile for use in primary care: the Framingham Heart Study.  Circulation. 2008;117(6):743-753.PubMedGoogle ScholarCrossref
    22.
    D’Agostino  RB  Sr, Grundy  S, Sullivan  LM, Wilson  P; CHD Risk Prediction Group.  Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation.  JAMA. 2001;286(2):180-187.PubMedGoogle ScholarCrossref
    23.
    Steyerberg  EW, Borsboom  GJ, van Houwelingen  HC, Eijkemans  MJ, Habbema  JD.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage.  Stat Med. 2004;23(16):2567-2586.PubMedGoogle ScholarCrossref
    24.
    Psaty  BM, Delaney  JA, Arnold  AM,  et al.  Study of cardiovascular health outcomes in the era of claims data: The Cardiovascular Health Study.  Circulation. 2016;133(2):156-164.PubMedGoogle ScholarCrossref
    25.
    White  AD, Folsom  AR, Chambless  LE,  et al.  Community surveillance of coronary heart disease in the Atherosclerosis Risk in Communities (ARIC) Study: methods and initial two years’ experience.  J Clin Epidemiol. 1996;49(2):223-233.PubMedGoogle ScholarCrossref
    26.
    Heckbert  SR, Kooperberg  C, Safford  MM,  et al.  Comparison of self-report, hospital discharge codes, and adjudication of cardiovascular events in the Women’s Health Initiative.  Am J Epidemiol. 2004;160(12):1152-1158.PubMedGoogle ScholarCrossref
    27.
    Yasaitis  LC, Berkman  LF, Chandra  A.  Comparison of self-reported and Medicare claims-identified acute myocardial infarction.  Circulation. 2015;131(17):1477-1485.PubMedGoogle ScholarCrossref
    28.
    Ness  RB; Joint Policy Committee, Societies of Epidemiology.  Influence of the HIPAA Privacy Rule on health research.  JAMA. 2007;298(18):2164-2170.PubMedGoogle ScholarCrossref
    29.
    Houser  SH, Howard  VJ, Hovater  MK, Safford  MM.  Obtaining medical records from healthcare facilities under the HIPAA Privacy Rule: the experience of a national longitudinal cohort study.  Neuroepidemiology. 2007;28(3):162-168.PubMedGoogle ScholarCrossref
    30.
    Colantonio  LD, Richman  JS, Carson  AP,  et al.  Performance of the atherosclerotic cardiovascular disease Pooled Cohort risk equations by social deprivation status.  J Am Heart Assoc. 2017;6(3):e005676. doi:10.1161/JAHA.117.005676PubMedGoogle ScholarCrossref
    31.
    Yadlowsky  S, Hayward  RA, Sussman  JB,  et al.  Clinical implications of revised pooled cohort equations for estimating atherosclerotic cardiovascular disease risk [published online June 5, 2018].  Ann Intern Med. doi:10.7326/M17-3011.Google Scholar
    ×