Further Insight Into the Cardiovascular Risk Calculator: The Roles of Statins, Revascularizations, and Underascertainment in the Women’s Health Study

Importance  While the pooled cohort equations from the recent American College of Cardiology/American Heart Association (ACC/AHA) Guideline on the Assessment of Cardiovascular Risk have overestimated cardiovascular risk in multiple external cohorts, the reasons for the discrepancy are unclear.

Objective  To determine whether increased use of statins over time, incident coronary revascularization procedures, or underascertainment of vascular events explain overestimation of risk in a more contemporary population.

Design, Setting, and Participants  The Women’s Health Study (WHS) is a nationwide cohort of US women free of cardiovascular disease, cancer, or other major illness at baseline from 1992 to 1995. A total of 27 542 women ages 45 to 79 years with complete ascertainment of plasma lipids and other risk factors were followed for a median of 10 years.

Main Outcome and Measure  Atherosclerotic cardiovascular disease (ASCVD), defined as any myocardial infarction, any stroke, or death due to cardiovascular cause.

Results  A total of 632 women experienced an ASCVD event over the course of the follow-up. The average predicted risk from the pooled cohort equations was 3.6% over 10 years, compared with an actual observed risk of 2.2%. Ratios of predicted to observed rates were 1.90 or higher in the groups with 0 to less than 5.0% and 5.0% to less than 7.5% risk and were over 1.40 in the groups with 7.5% to less than 10.0% and 10.0% or higher risk. Rates of statin use and revascularizations increased over follow-up time and by risk group, and in sensitivity analyses, we estimated the hypothetical rates if no women were prescribed statins or underwent revascularization procedures. After adjustment for intervention effects of statins and revascularization as well as hypothetical confounding by indication, ratios of predicted to observed rates remained 1.80 or higher in the lower 2 risk groups and over 1.30 higher in the upper 2 risk groups. Underascertainment is unlikely since follow-up rates in the WHS were 97.2%, and overall we would need approximately 60% more events to match the numbers predicted using the pooled cohort equations.

Conclusions and Relevance  Statin use, revascularization procedures, and underascertainment of events do not explain the discrepancy between observed rates of ASCVD in the WHS and those predicted by the ACC/AHA pooled cohort equations. Other explanations include changing patterns of risk within more contemporary populations.

Quiz Ref IDThe American College of Cardiology/American Heart Association (ACC/AHA) cholesterol guidelines,¹ released in the fall of 2013, provide new recommendations for statin therapy. The pooled cohort equations on which these are based now include stroke in the combined end point of atherosclerotic cardiovascular disease (ASCVD) rather than coronary heart disease only. This is particularly important for women, in whom rates of stroke can be as high as those for myocardial infarction (MI).² The risk equations were developed in a pooled sample of data from 5 cohorts of individuals followed over at least a decade and tested in 3 external validation cohorts.

Concerns have been raised, however, including concern about how well the new prediction model works in data sets that are more contemporary than those used to derive the equations.³ In all 3 validation cohorts used by guideline developers, the models overestimated risk¹; discrimination (the ability to separate cases from noncases) was lower, and calibration (the agreement between predicted risk and actual observed risk) was poor. In both the Reasons for Geographic and Racial Differences in Stroke (REGARDS) study and in the Multi-Ethnic Study of Atherosclerosis (MESA), external cohorts that were not used in model development, the estimated risk of ASCVD using the new pooled cohort equations was too high. The same overestimation occurred using more contemporary data from the derivation cohorts, including the Framingham and Atherosclerosis Risk in Communities (ARIC) studies, and was consistent in men and women and among blacks and whites. Calibration was also poor in 4 additional external cohorts.^3,4 Further analyses in REGARDS⁵ improved calibration, although risk remained overestimated overall, with better fit in a subset with additional case finding through Medicare records. Because new guidelines for statin therapy are based on these equations, overestimation of risk will have strong clinical implications and could lead to overprescription of medication as well as inaccurate data on risks and benefits.

Quiz Ref IDPotential explanations for this discrepancy between observed rates and those predicted from the pooled cohort equations include an increase in statin use over time, an increase in revascularization procedures, and failure to fully ascertain clinical events during follow-up.^6-8 Herein, we explore these possibilities in the Women’s Health Study (WHS), a cohort of initially healthy American women. While this analysis directly relates to the cardiovascular disease (CVD) risk equations, similar concerns regarding changing incidence rates over time, including increasing use of medications, may apply to other risk prediction tools.

The WHS comprises a nationwide cohort of 39 876 US women 45 years or older and free of cardiovascular disease, cancer, or other major illness at study entry (from September 1992 through May 1995).⁹ The WHS was a randomized factorial trial of aspirin and vitamin E, and final results have been published.^9,10 Neither intervention had a significant impact on major cardiovascular events, including MI, stroke, or death due to cardiovascular cause. Self-reported exposure data were collected on age, race/ethnicity, diabetes mellitus (DM), blood pressure, blood pressure treatment, smoking status, and cholesterol-lowering medication. All study participants provided written informed consent. The study protocol was approved by the institutional review board of Brigham and Women’s Hospital (Boston, Massachusetts).

Women eligible for the current analysis were 45 to 79 years old and had complete ascertainment of plasma lipids and information on other risk factors (N = 27 542). Plasma samples were measured for total, low-density (LDL-C) and high-density (HDL-C) lipoprotein cholesterol in a core laboratory certified by the National Heart, Lung, and Blood Institute–Centers for Disease Control and Prevention Lipid Standardization Program.

Women were followed with annual questionnaires through the end of the trial in March 2004, a median of 10.2 years (25th and 75th percentiles = 9.7 and 10.6 years, respectively), for incident MI, stroke, and cardiovascular deaths, as well as coronary revascularization procedures and other study end points. Follow-ups with respect to morbidity and mortality were 97.2% complete and 99.4% complete, respectively.⁹

Reported events were adjudicated by an end-points committee of physicians after medical record review. Myocardial infarction was confirmed if symptoms met World Health Organization criteria and if the event was associated with abnormal levels of cardiac enzymes or diagnostic electrocardiograms. A confirmed stroke was defined as a new neurologic deficit of sudden onset that persisted for at least 24 hours. Death was confirmed to be from cardiovascular causes on the basis of an examination of autopsy reports, death certificates, medical records, and information obtained from the next of kin or other family members. The use of coronary revascularization (coronary artery bypass surgery [CABG] or percutaneous coronary angioplasty [PTCA]) was confirmed by a review of the medical records. For this analysis, the end point was ASCVD, defined as any MI, stroke, or death due to cardiovascular cause. Owing to the difficulty in distinguishing type of cardiovascular death,¹¹ all such deaths were included rather than fatal MI and fatal stroke as used in the derivation cohorts for the pooled risk equations.

Baseline characteristics are reported as percentages or means (SDs) both overall and within strata of predicted risk. Predicted values from the ACC/AHA pooled cohort equations were obtained using published equations.¹ Women were classified into 4 clinical risk groups based on predicted risk with cut points of 5.0%, 7.5%, and 10.0%, as previously described,¹ and average predicted risk was calculated.

For comparison, observed rates of ASCVD were obtained from Kaplan-Meier survival curves. Because all women were followed for a minimum of 8 years, the cumulative incidence of ASCVD was estimated as of 8 years and then extrapolated to 10 years as I10 = 1 − (1 − I8)^1.25, where I10 is the cumulative incidence as of 10 years and I8 is the cumulative incidence as of 8 years. Women were censored at death due to a noncardiovascular cause or at administrative censoring on March 31, 2004.

Calibration was assessed by plotting the observed and predicted average risk within the predefined clinical risk strata defined herein as well as in deciles. Expected-to-observed ratios were calculated.

We examined the potential effects of statin therapy and revascularization procedures on the observed risks of ASCVD in sensitivity analyses. First, we estimated the proportion of women without prior ASCVD taking statins by follow-up time, both overall and within risk strata. Statins were not generally in use at baseline in 1992 to 1995, but questions on use of statins were included on questionnaires at 3, 4, 9, and 10 years of follow-up. Linear interpolation was used to estimate proportions between these times. We estimated the proportions over time of women undergoing coronary revascularization procedures (CABG or PTCA), obtained from annual questionnaires, among those with no reported prior ASCVD, both overall and within risk strata. We performed 5 sensitivity analyses to adjust the observed risk estimates for the use of statins and/or revascularization procedures. Further details of the mathematical adjustments are in the eAppendix in the Supplement.

We increased the rate among those using statins inversely proportional to the estimated effect of statins on ASCVD. In a meta-analysis¹² of randomized clinical trials, among those without prior vascular disease, statins reduced the risk of major vascular events by an aggregate 25.0% per millimole-per-liter reduction in LDL-C, similar to the effect of a typical dose. We increased the rate of ASCVD among those using statins proportionately. The rate was increased for each year of follow-up separately, then aggregated in a life table analysis.

Because of potential confounding by indication, statin users may be at higher risk of ASCVD than those at the same estimated baseline level of risk by conventional risk factors. For example, those who were prescribed statins may have a family history of ASCVD, high CRP, or their lipid levels may have increased over time. We therefore increased the rate further in this group by doubling it, in addition to accounting for the statin effect as in analysis 1.

We adjusted for the potential effects of revascularization therapy using similar methods. First, we conservatively estimated that such procedures would reduce rates of ASCVD by 25.0%. This estimate is optimistic because randomized clinical trials comparing these procedures to optimal medical therapy suggest little difference in the occurrence of ASCVD end points.¹³ Second, we allowed for confounding by indication similar to that for statins by doubling the observed risk.

In a separate sensitivity analysis we censored women at the time of initiation of any cholesterol-lowering medications.

We performed an additional sensitivity analysis censoring women when they began using cholesterol-lowering medications or underwent revascularization, by definition prior to ASCVD.

In our initial analyses, we adjusted for use of statins only because those agents have been found to reduce cardiovascular risk. In alternate analyses, we substituted use of any cholesterol-lowering agent for statins. In addition, since the risk equations are meant to be used clinically for those who without DM, not prescribed cholesterol-lowering medications at baseline, and with LDL-C level in the range of 70 to 189 mg/dL, we also analyzed this subset separately. (To convert LDL-C to millimoles per liter, multiply by 0.0259.)

Finally, we estimated what size of effects would be necessary to reconcile any differences in event rates. In particular, we estimated the numbers of events that would have to be missed through inadequate ascertainment in order to reconcile remaining differences between observed and predicted rates.

Women were of average age of 54.2 years and primarily white (Table 1). Average cholesterol levels were 211.8 mg/dL for total cholesterol, 124.1 mg/dL for LDL-C, and 53.8 mg/dL for HDL-C. (To convert total cholesterol and HDL-C to millimoles per liter, multiply by 0.0259.) Mean untreated and treated blood pressures were 121.4 and 138.6 mm Hg, respectively, and 13.5% of women were taking antihypertensive medications. Twelve percent were current smokers, and 2.5% had DM at baseline. The average predicted 10-year risk using the ACC/AHA ASCVD risk equations was 3.6%. Most women (78.6%) fell into the less than 5.0% 10-year risk category (Table 1).

Throughout follow-up, 632 women experienced an ASCVD event, 250 of which were MIs; 302 experienced a stroke (3 of these women also experienced an MI), with 83 additional deaths due to cardiovascular cause. The observed 10-year ASCVD event rate based on the Kaplan-Meier curve was 2.2%, compared with the 3.6% average rate predicted by the ACC/AHA pooled cohort equations. Quiz Ref IDWhen observed and predicted rates were compared within risk categories, ratios of predicted to the actual observed rates were 1.90 or higher than the actual observed rates among those with risk of less than 7.5%, and over 1.40 among those with risk of 7.5% or higher (Table 2 and Figure 1A). Overestimation was also apparent when considering deciles of predicted risk (Figure 2).

The proportions of women using statins increased over time, reaching 21.8% at 10 years among those without a previous ASCVD event. Use was higher among women at higher risk, reaching 37.5% in the highest-risk women by 10 years (Figure 3A).

To account for use of statins, we adjusted for a 25.0% decrease in CVD risk with statin use, increasing the observed rates slightly, from 10.8% to 11.2% in the highest risk group, vs 15.6% predicted by the pooled cohort equations (Table 2 and Figure 1B). Additional adjustment for indication bias, including a doubling of risk among those taking statins, had little impact on the discrepancy between observed and predicted event rates (Table 2 and Figure 1C).

Revascularization procedures occurred in 523 women, of whom 343 had no prior ASCVD event. Cumulative incidence over 10 years was 1.4% overall but reached 5.2% among those with at least 10.0% predicted risk (Figure 3B). Similar adjustments for a sizeable 25.0% reduction with intervention as well as confounding by indication for both statins and revascularization procedures increased adjusted rates slightly (Table 2 and Figure 1D). A larger 5-fold increase in risk of ASCVD among those with a revascularization procedure increased the rate in those in the highest risk group to 11.7% vs the 15.6% predicted.

Use of all cholesterol-lowering medications, including nonstatins, was 3.2% at baseline and increased to 22.0% overall at 10 years (Figure 3C), when they likely comprised mostly statins. Similar adjustments had little impact (Table 2). Evaluating a subset of 24 084 women without DM, not using medications at baseline, with LDL-C levels in the range of 70 to 189 mg/dL, led to similar results (Table 2).

Additional analyses censoring at initiation of cholesterol-lowering medication use led to little change in the adjusted risk (Table 2). The same was true when also censoring at revascularization. In these analyses, the difference between observed and predicted rates in the lower risk groups worsened.

To understand the extent of bias necessary to reconcile observed and predicted rates, we used the equations provided in the eAppendix in the Supplement. If the average annual proportion using statins was 20.0%, and the effect of statins was a 25.0% reduction in risk, then using equation (1) the rate in the absence of statin use would be only 5.3% higher than that observed and would not reach the rates predicted by the pooled cohort equations. If, in addition, the risk in those taking statins was 2 times higher owing to indication, then the rate in absence of statin use would increase by 9.1% from that observed using equation (2). Considering the treatment effect only and using equation (1), with a 50.0% reduction with statin treatment, the average annual proportion using statins would need to be 57.0% to equalize observed and predicted rates. In addition, allowing for a doubling of risk by confounding by indication, the average annual proportion using statins would need to be 40.0% to explain the 40%.

The relatively low rates of revascularization have little additional impact. Assuming an average proportion of patients who underwent a procedure (and were prescribed statins) of 5.0%, with an additional 15.0% prescribed statins, and the same treatment and indication effects, the rate in the absence of statins or revascularization using equation (3) would be 11.1% higher than observed. Only if all those undergoing a revascularization procedure would otherwise have an ASCVD event would the numbers be high enough to match those predicted.

We can assess how large any potential ascertainment bias would need to be to account for these differences in observed and predicted rates. After adjustment for intervention effects of statins and revascularization as well as hypothetical confounding by indication, ratios of predicted to the observed rates remained 1.80 or higher among those in the lower 2 risk groups and over 1.30 in the upper 2 risk groups. Overall, we would need approximately 60% more events to match the numbers predicted by the pooled cohort equations.

Since the initial publication of the new ACC/AHA guidelines,¹ there has been concern about the new risk equations. First, in the 3 external validation cohorts examined by the committee, the model overpredicted risk. When examined in 3 additional cohorts,³ as well as in a study in Rotterdam, the Netherlands,⁴ the overestimation of risk was repeated. Several possible reasons for the discrepancies have been suggested.^6,7 One is that both the WHS and the Physicians’ Health Study were trials of health professionals; 75% of WHS participants were registered nurses, and another 15% were licensed practical nurses.¹⁴ These 2 cohorts thus may be at lower risk than other more contemporary cohorts. However, much of the lower risk should be reflected in their lower risk factors, which are included in the risk equations. In addition, MESA, REGARDS, the Women’s Health Initiative Observational Study, and the Rotterdam Study were observational cohorts. Overestimation was seen in these, as well as in the “contemporary cohorts” of the original Framingham, Framingham Offspring, and ARIC studies used in guideline validation.¹

The WHS has been criticized because of self-reports.⁶ While women self-reported their ASCVD end points, these were adjudicated using medical records. Lipid levels were directly measured. Blood pressure was self-reported, which has been shown to correlate well with measured blood pressure in a similar cohort of female health professionals.¹⁵ Diabetes mellitus was self-reported, but in a validation of incident reports in the WHS, 91% of cases were confirmed.¹⁶ Underreports of either hypertension or DM, however, should lead to underestimation rather than overestimation of risk by the pooled cohort equations.

Concerns have been raised about follow-up rates in some cohorts.^5,7 Updated results from REGARDS using Medicare claims suggest that incomplete ascertainment could explain the previous lack of fit.⁵ Follow-up in the WHS, however, was excellent, reaching 97.2% for morbidity and 99.4% for mortality.⁹ Because the WHS was a trial, follow-up efforts were extensive, and participants may have had a greater commitment to the study. We would need an additional 60.0% of observed events to reach the number predicted by the pooled cohort equations. It is unlikely that the WHS follow-up missed that many events.

Increasing use of statins is a plausible explanation for discrepancies in more contemporary populations. Use of statins has greatly expanded over the past 2 decades, although these were recommended only for women at high risk or with high LDL-C levels.² Adjustment using strong assumptions, however, could not explain the discrepancy between observed and predicted rates. For example, assuming a 25.0% intervention effect and a doubling of risk owing to confounding by indication, ratios of predicted to observed rates remained 1.80 or higher among the 2 lower risk groups and over 1.3 in the 2 higher risk groups.

Revascularization procedures are another possible explanation for lower event rates in more contemporary cohorts. Rates were too low in the WHS to have much impact, however. Furthermore, more recent randomized clinical trials of revascularization vs optimal medical therapy have suggested little difference in events, despite relief of angina.¹³ Because revascularizations are part of the end point used in most statin trials, incorporation of these into prediction models as in the Reynolds risk scores^17,18 might provide patients with a better overall perspective on vascular risk.

Censoring at the time of use of cholesterol-lowering medications or revascularization actually increased the discrepancy between observed and predicted rates. Such censoring is nonignorable and biased, meaning that those who are prescribed statins are likely at higher risk owing to factors not included in the risk equations. Such factors could include a family history of ASCVD, elevated levels of C-reactive protein, or an increase in lipid levels over time. While censoring can adjust for the intervention effect, it does not take into account such confounding by indication.

Other explanations should be considered. The baseline measurements in the various cohorts took place over different time periods, with some of the earliest cohorts in the development set collecting data as far back as 1968 (Figure 4). All of the validation cohorts, including the WHS, started much later, beginning baseline data collection in the late 1980s to early 2000s. Whether equations developed using earlier data apply to more contemporary cohorts is a substantive issue because cardiovascular event rates have declined in a consistent manner over the past 40 years, particularly among patients with DM.^19,20 Other changes over time may involve additional medical therapies, such as improved treatment of DM or hypertension, or even aspirin use, particularly in men in whom the greatest reductions have been seen.²¹ Changes in lifestyle, such as number of cigarettes smoked or passive smoking, may not be completely controlled. Changes over time also undoubtedly took place in the development cohorts used for the risk equations. If these equations are to be used to recommend statin therapy, the appropriate background risk needs to be clarified. A simple increase in the risk cutoff for statin use from 5.0% to 7.5%,²¹ instead of model recalibration, is not optimal because the risk-benefit discussion remains muddied.

Limitations to the current analyses should be noted. Quiz Ref IDFirst, we did not assess statin use at every follow-up visit, making finer adjustment for individual statin use difficult. Instead, we interpolated rates over time and used life table methods for adjustment. Second, while risk factors were updated on annual questionnaires, we did not collect repeated blood samples, so we could not examine the impact of changing cholesterol levels on statin use. Third, estimates of intervention effects are available from meta-analyses of randomized clinical trials, but our estimates of confounding by indication are hypothetical. Fourth, the WHS excluded those with a history of angina at baseline, which eliminated a small group of women who were included in the pooled cohorts. These would be at high risk but would already have been diagnosed as having clinical ASCVD and should be recommended for statin therapy.²² Fifth, because of difficulty assigning accurate cause of death, the outcome described herein included all cardiovascular deaths. Restriction to deaths owing to MI or stroke would only increase the observed discrepancies. Finally, there have been differences in the diagnosis of MI over time, particularly with the increasing use of troponin levels.²³ These, however, would make the observed rates higher in more contemporary cohorts and thus not explain any overestimation.

The pooled cohort equations have now been found to overpredict the rate of CVD in at least 7 external validation cohorts. Quiz Ref IDWhile alternative explanations may exist, we found that the use of statins, revascularizations, or underascertainment could not explain the extent of overestimation in the WHS. Recalibration of the pooled cohort equations using available contemporary data sets might provide a solution to this problem.

Accepted for Publication: June 27, 2014.

Corresponding Author: Nancy R. Cook, ScD, Division of Preventive Medicine, Brigham and Women’s Hospital, Harvard Medical School, 900 Commonwealth Ave E, Boston, MA 02215-1204 (ncook@rics.bwh.harvard.edu).

Published Online: October 6, 2014. doi:10.1001/jamainternmed.2014.5336.

Author Contributions: Dr Cook had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Both authors.

Acquisition, analysis, or interpretation of data: Both authors.

Drafting of the manuscript: Cook.

Critical revision of the manuscript for important intellectual content: Ridker.

Statistical analysis: Cook.

Administrative, technical, or material support: Ridker.

Conflict of Interest Disclosures: Dr Ridker is listed as a coinventor on patents held by the Brigham and Women’s Hospital that relate to the use of inflammatory biomarkers in cardiovascular disease and DM that have been licensed to AstraZeneca and Seimens, and has received research grant support from AstraZeneca and Pfizer, manufacturers of statin therapy.

Funding/Support: The WHS was supported by grants from the National Heart, Lung, and Blood Institute (HL043851) and the National Cancer Institute (CA047988), Bethesda, Maryland.

Role of the Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.