Cardiovascular Risk Prediction Models for People With Severe Mental Illness: Results From the Prediction and Management of Cardiovascular Risk in People With Severe Mental Illnesses (PRIMROSE) Research Program

Importance  People with severe mental illness (SMI), including schizophrenia and bipolar disorder, have excess rates of cardiovascular disease (CVD). Risk prediction models validated for the general population may not accurately estimate cardiovascular risk in this group.

Objective  To develop and validate a risk model exclusive to predicting CVD events in people with SMI incorporating established cardiovascular risk factors and additional variables.

Design, Setting, and Participants  We used anonymous/deidentified data collected between January 1, 1995, and December 31, 2010, from the Health Improvement Network (THIN) to conduct a primary care, prospective cohort and risk score development study in the United Kingdom. Participants included 38 824 people with a diagnosis of SMI (schizophrenia, bipolar disorder, or other nonorganic psychosis) aged 30 to 90 years. During a median follow-up of 5.6 years, 2324 CVD events (6.0%) occurred.

Main Outcomes and Measures  Ten-year risk of the first cardiovascular event (myocardial infarction, angina pectoris, cerebrovascular accidents, or major coronary surgery). Predictors included age, sex, height, weight, systolic blood pressure, diabetes mellitus, smoking, body mass index (BMI), lipid profile, social deprivation, SMI diagnosis, prescriptions for antidepressants and antipsychotics, and reports of heavy alcohol use.

Results  We developed 2 CVD risk prediction models for people with SMI: the PRIMROSE BMI model and the PRIMROSE lipid model. These models mutually excluded lipids and BMI. In terms of discrimination, from cross-validations for men, the PRIMROSE lipid model D statistic was 1.92 (95% CI, 1.80-2.03) and C statistic was 0.80 (95% CI, 0.76-0.83) compared with 1.74 (95% CI, 1.63-1.86) and 0.78 (95% CI, 0.75-0.82) for published Cox Framingham risk scores. The corresponding results in women were 1.87 (95% CI, 1.76-1.98) and 0.79 (95% CI, 0.76-0.82) for the PRIMROSE lipid model and 1.58 (95% CI, 1.48-1.68) and 0.77 (95% CI, 0.73-0.81) for the Cox Framingham model. Discrimination statistics for the PRIMROSE BMI model were comparable to those for the PRIMROSE lipid model. Calibration plots suggested that both PRIMROSE models were superior to the Cox Framingham models.

Conclusions and Relevance  The PRIMROSE BMI and lipid CVD risk prediction models performed better in SMI compared with models that include only established CVD risk factors. Further work on the clinical effectiveness and cost-effectiveness of the PRIMROSE models is needed to ascertain the best thresholds for offering CVD interventions.

It is well established that people with severe mental illnesses (SMIs), such as schizophrenia and bipolar disorder, have excess rates of cardiovascular disease (CVD), including myocardial infarctions and strokes.¹ The risk of dying from CVD is 3-fold higher in people with SMI younger than 50 years and 2-fold in those aged 50 to 75 years.² There has been an increase in clinical and research efforts addressing this problem, but we lack knowledge regarding the most effective ways to predict and manage cardiovascular risk in people with SMI. We know that the conventional cardiovascular risk factors, including smoking, dyslipidemia (with higher levels of total cholesterol and triglycerides and lower levels of high-density lipoprotein cholesterol [HDL-C]), diabetes mellitus, and obesity and possibly a higher prevalence of hypertension, are more common in people with SMI,^3,4 especially those with well-established mental disorders.⁵ There may even be a shared genetic predisposition to comorbidities in patients with SMI. People with SMI are less likely to exercise, have unhealthy diets, and may receive inferior physical health care.⁶ Antipsychotic medications may contribute to cardiovascular risk through weight gain and its effect on glucose and lipid metabolism, although mortality studies^2,7,8 that explore the role of antipsychotic medication in premature deaths show inconsistent findings. For example, an 11-year follow up study of people with schizophrenia⁷ reported reduced cardiovascular mortality in people treated with olanzapine and clozapine, but methodological issues in the study, particularly unmeasured confounding, have been critiqued in detail.⁹

For the general population, cardiovascular risk is managed by using CVD risk scores to determine the absolute risk for an individual patient and therefore the likely benefit of prescribing lipid-lowering medications and/or other interventions. The most established scores are the Framingham Heart Study risk scores,¹⁰ currently available as both a body mass index (BMI) model (including BMI but not laboratory results for blood lipid levels) and a lipid model (including total cholesterol and HDL-C but not BMI).

Clinical guidelines, such as the UK National Institute for Health and Clinical Excellence guidelines on schizophrenia, recommend more intensive screening for cardiovascular risk in people with SMI.^1,11 However, we do not know how well the modifiable cardiovascular risk factors, included in models such as Cox Framingham, predict CVD risk in people with SMI. There is reason to believe that existing models may not accurately determine the high level of risk conferred by having a long-term SMI. The established scores, such as the Cox Framingham, were developed by excluding people with SMI and since then have not been tested in this population; in addition, the established scores do not consider SMI-specific exposures, such as antipsychotic medication. To our knowledge, no previous studies have assessed the performance of CVD risk prediction models in people with SMI.

Using data from a large UK primary care database, The Health Improvement Network (THIN),¹² we aimed to develop and validate cardiovascular risk prediction models specific to people with SMI: Prediction and Management of Cardiovascular Risk in People With Severe Mental Illnesses (PRIMROSE) models. These new models developed from the PRIMROSE study included traditional cardiovascular risk factors and additional SMI-specific variables. We compared the performance of these new risk prediction models against existing published Cox Framingham scores in people with SMI since the Cox Framingham scores are widely used internationally and the coefficients of the scores are readily available to allow comparison. This work formed part of a program of research, PRIMROSE, which is funded by the UK National Institute for Health Research (http://www.ucl.ac.uk/primrose).

A prospective study of anonymous data collected between January 1, 1995, and December 31, 2010, was conducted to develop and validate a 10-year risk score for predicting newly recorded cardiovascular events in people with SMI. The scheme for THIN to obtain and provide anonymous patient data to researchers was approved by the National Health Service South-East Multicentre Research Ethics Committee in 2002, and scientific approval for this study was obtained from CMD Medical Research’s Scientific Review Committee in March 2012.

We used THIN,¹² a UK primary care database that includes information obtained from routine clinical practice. Primary care physicians and staff use a hierarchical system of Read codes to enter information in THIN, such as symptoms and diagnoses, during clinical appointments and administration¹³ that creates a longitudinal record for each patient. At the time we developed the PRIMROSE risk models, THIN included almost 10 million patients with geographic coverage broadly representative of the UK population.¹⁴ Approximately 98% of the population is registered with a general practitioner in the United Kingdom.¹⁵ THIN data are subject to a range of quality assurance procedures¹⁶ and have been successfully used in wide-ranging studies on CVDs,^17,18 including cardiovascular risk score validation work in the general population.¹⁹ Primary care data are a particularly suitable source for assessing cardiovascular risk in people with SMI in the United Kingdom since most people with SMI are registered with a general practitioner whom they see frequently and most of the required data for risk scores (eg, laboratory and blood pressure measurements) are available owing to policy initiatives that provide incentives for annual cardiovascular screening.²⁰ The SMI diagnoses have been validated in UK general practice.²¹

We included individuals aged 30 to 90 years with a diagnostic entry in their primary care electronic health record for an SMI at any time during their follow-up period. We defined SMI as (1) schizophrenia and schizoaffective disorder, (2) bipolar affective disorder, and (3) other nonorganic psychoses. We created lists of the diagnostic codes used by general practitioners or administrators. The codes are usually based on assessments by mental health specialists. We extracted anonymous/deidentified data between January 1, 1995, and December 31, 2010.

Newly recorded fatal and nonfatal cardiovascular events were defined as a diagnostic record for myocardial infarction, angina pectoris, coronary heart disease, major coronary surgery and revascularization, cerebrovascular accident, and transient ischemic attack.

We developed 2 PRIMROSE risk models: the BMI model and lipid model. Detailed descriptions of the follow-up period, the variables considered in our analysis, the development of the PRIMROSE risk models and their 10-fold internal cross-validation, as well as the imputation of missing data and our sample size calculation, are provided in the eAppendix in the Supplement.

In summary, we performed Cox proportional hazards regression with backward elimination to derive the PRIMROSE models. The variables considered in the models are listed in Tables 1, 2, and 3. We compared the performance of different models by calculating the C index²² and D statistic²³ for discrimination, by constructing calibration plots, and by assessing the numbers of people classified as having high risk of CVD over the course of 10 years (score >20%) who went on to have a CVD event.

After comparing the PRIMROSE risk models against published Cox Framingham models, we performed 2 supplementary comparisons. First, we wanted to ascertain whether our results could be explained by differences between North American and UK source populations. Thus, we reestimated the Cox Framingham model to the UK general population and compared the PRIMROSE risk models against this reestimated model. Second, we wanted to examine whether there are any benefits with use of the PRIMROSE models beyond a simple reestimate of the coefficients for the Cox Framingham model in the SMI population. For this examination, we compared the PRIMROSE model with a model that included only the variables in the Cox Framingham risk score but with their coefficients reestimated within our SMI cohort.

We identified 38 824 individuals who met eligibility criteria (eFigure 1 in the Supplement) in 430 general practices. There was a median follow-up period of 5.6 years (interquartile range, 2.5-9.2 years), which compares favorably with other European risk score studies in primary care databases.¹⁹ A total of 8020 people (20.7%) had 10 years or more of follow-up. There were 18 417 men (47.4%) in the cohort, the mean age was 49.5 years, and more patients lived in areas with greater social deprivation (Table 1). Approximately one-third of the patients had a diagnosis of schizophrenia, with marginally fewer having diagnoses of bipolar disorder and psychosis not otherwise specified. In terms of data completeness, during the period in which the longitudinal imputation model was applied (eAppendix in the Supplement provides details on imputation), 95.7% of the people with SMI had a record for smoking status, 88.9% for systolic blood pressure, 78.6% for weight, 65.5% for height, 52.3% for total cholesterol, and 43.2% for HDL-C. The remaining variables were complete. The imputed data are presented in eTable 1 in the Supplement.

There were 2324 newly recorded CVD events during the follow-up period, corresponding to a crude incidence rate of 9.72 (95% CI, 9.33-10.1) per 1000 person-years. The incidence of CVD increased with age and was higher in men within all age categories except for the very oldest category (eTable 2 in the Supplement). The most common events were ischemic or unspecified stroke (778 [33.5%] of the total events), myocardial infarction (414 [17.8%]), transient ischemic attack (349 [15.0%]), angina (325 [14.0%]), coronary heart disease unspecified (304 [13.1%]), unstable angina (65 [2.8%]), and hemorrhagic stroke (46 [2.0%]).

The associations between age, sex, deprivation, and established cardiovascular risk factors were all in the expected direction for known CVD risk factors (Table 2). After adjusting for age and sex, CVD was positively associated with total cholesterol level, weight, deprivation, blood pressure, age, smoking, and male sex and negatively associated with HDL-C level (Table 2). The SMI diagnosis, antidepressant use, and a history of heavy alcohol use were also predictive of CVD. There was no strong evidence of nonlinear associations between the continuous variables and CVD hazard with the exception of age in years, for which a log transformation improved linearity.

For the PRIMROSE BMI model, all variables listed in Table 2 were retained in the model after backward elimination with the exception of prescription of lithium at baseline (Table 3). For the PRIMROSE lipid model, receipt of both lithium and first-generation antipsychotics were eliminated from the model; however, baseline receipt of second-generation antipsychotics and antidepressants were retained as predictors of 10-year CVD risk (Table 3). The complete formulas for the new PRIMROSE models for application in practice are available in the eAppendix in the Supplement.

Both PRIMROSE risk models (BMI and lipid) performed well compared with the published Cox Framingham BMI and lipid models¹⁰ in the SMI cohort (Table 4, Figure, and eTables 3 and 4 in the Supplement). In terms of their discrimination statistics, the D statistics were better for the PRIMROSE models in both men and women (higher D scores indicate better discrimination, and an increase of 0.1 point has been defined as important²³). C statistics were broadly similar (Table 4 and eTables 3 and 4 in the Supplement). The PRIMROSE lipid model discrimination was not substantially better than the PRIMROSE BMI model discrimination. The calibration plots suggested that the PRIMROSE lipid and BMI models predicted CVD risk more accurately than the published Cox Framingham models, with greater agreement between the predicted and observed risks for CVD (Figure). The published Cox Framingham models tended to overpredict CVD risk in this population, particularly among men.

When high risk for CVD was defined using the conventional threshold of 20% risk over the course of 10 years, the PRIMROSE models were better at predicting the percentage of people with SMI who would go on to experience a CVD event (Table 4). For men categorized as being at high risk (>20% risk score), the proportions developing CVD within 10 years were 19.2% for the PRIMROSE lipid model and 18.6% for the PRIMROSE BMI model, with both performing better than the corresponding published Framingham CVD risk models⁸ (13.3% for the lipid model and 12.3% for the BMI model). For men categorized as being at low risk (<20%), slightly more experienced a CVD event when the PRIMROSE model was applied compared with the published Cox Framingham model (3.3% with the PRIMROSE lipid model, 2.0% with the published Cox Framingham lipid model) (Table 4). For women, the numbers correctly classified at the 20% threshold were similar for the PRIMROSE and published Cox Framingham lipid models.

The calibration of the Cox Framingham model reestimated to the UK general population in the SMI cohort was good for men but somewhat poorer for women, with a degree of underprediction (eFigure 2 in the Supplement). The discrimination of this reestimated model was better than the published Framingham models; however, the PRIMROSE model still appeared somewhat superior to both (eTables 3-5 in the Supplement). Therefore, the superiority of the PRIMROSE model was not simply explained by the differences between US and UK populations.

The Cox Framingham model reestimated to the SMI cohort had good calibration (eFigure 2 in the Supplement) and discrimination (eTables 3-5 in the Supplement) for people with SMI. However, the PRIMROSE lipid model (containing the additional SMI variables) still showed better discrimination. This result suggests that the PRIMROSE model performs better for people with SMI compared with models that contain only the standard variables usually found in risk tools such as the Cox Framingham, including systolic blood pressure, smoking, BMI, and diabetes, even when these models are reestimated in people with SMI.

To our knowledge, this is the first study to develop and assess the performance of CVD risk prediction tools in people with SMI. We derived 2 new CVD risk prediction models specific to people with SMI: the PRIMROSE lipid model and the PRIMROSE BMI model. These new PRIMROSE models include additional variables for psychiatric diagnosis, psychotropic medication at baseline, harmful use of alcohol, use of antidepressants at baseline, and a social deprivation score. Both models performed better than the available published Cox Framingham CVD risk models in predicting newly recorded CVD events in people with SMI, with better discrimination and calibration. The published Cox Framingham models appeared to overpredict CVD risk in this population, especially among men.

Overprediction is likely to reflect differences between historical North American men and contemporary UK men. This phenomenon has been observed when cardiovascular scores have been validated in non-SMI general practice populations, in which the Cox Framingham model overpredicts CVD events by 32% in UK men.¹⁹ This overprediction has been seen at all levels of CVD risk, including among men with an elevated risk equivalent to that seen in SMI.¹⁹

The PRIMROSE BMI model performed almost as well as the PRIMROSE lipid model, supporting use of the BMI model as an alternative to the lipid model in situations in which blood test results are not available. This alternative might be important for people with SMI who are reluctant to provide blood samples or when blood test results are not readily available. Although data regarding alcohol use might be inaccurately reported to general practitioners, we included only recognized severe alcohol problems in the model. The other variables in the model are available to general practitioners, including the medications they prescribe and the Townsend index for the individual’s postcode.

Reestimating the parameters of the published Cox Framingham models to the UK general population and within the SMI cohort improved the discrimination and calibration of the Cox Framingham models in men. However, the PRIMROSE models remained superior. This superiority indicates that the additional variables in PRIMROSE, such as diagnoses and medication at baseline, are important to include in the prediction of CVD risk in patients with SMI. This offering improved prediction beyond accounting for differences between the US and UK populations and between the general UK population and the population with SMI (assessed by evaluating the Framingham model reestimated to the SMI cohort). Among women, the Cox Framingham model reestimated within the SMI cohort also improved discrimination and calibration. However, the Cox Framingham model reestimated to the UK general population appeared to have poorer calibration, with a degree of underprediction. This finding may be explained by women with SMI having a particularly high CVD risk relative to women in the UK general population (greater than the differences among men with and without SMI); as such, application of the general population model for women to the SMI women relies on a degree of extrapolation. Overall, the new PRIMROSE models offer the most consistent performance for people with SMI.

The PRIMROSE models advance our understanding of the best ways to assess and manage the increased cardiovascular risk in people with SMI. Previously, we have not known whether cardiovascular events are predicted by the same risk factors in people with SMI, including smoking, diabetes, hypertension, BMI, and dyslipidemia. Our results suggest that these traditional risk factors do mediate the association between SMI and high rates of CVD events. However, a model including specific SMI diagnosis, baseline prescription of antipsychotics and antidepressants, as well as harmful use of alcohol offers improved prediction, suggesting that these variables are associated with the increased risk in CVD independent of traditional CVD risk factors. Caution is required because the observed associations of the SMI-specific variables and CVD do not necessarily imply a causal relationship between these factors and future CVD events. Rather, they are markers at baseline, and the reason for their inclusion may be related to characteristics of individuals prescribed antipsychotics and antidepressants in the UK, including issues that might directly influence the decision to prescribe these drugs. It would be incorrect to interpret our study results in the same way as epidemiological studies. Our study design was risk score development, for which we developed and validated the best risk score model for people with SMI. Variables may be retained in the final model despite the fact that their coefficients have 95% CIs that cross unity, with corresponding tests of significance outside the conventional 5% level. Furthermore, risk score development studies do not aim to adjust for important confounders; rather, they seek to derive the best prediction model.

Our study benefited from large numbers of people with SMI who are representative of the UK SMI population²⁴ with longitudinal data regarding a range of CVD risk factors and a follow-up period long enough to provide sufficient data to develop and validate a CVD risk prediction model that included additional variables not traditionally included in such models. Finally, we carefully assessed whether the superior performance of the PRIMROSE models could be attributed to the fact that existing CVD models have been developed in different populations. However, our supplementary analyses supported the conclusion that the PRIMROSE models were better than the published Cox Framingham models even when the parameters were reestimated for the general UK population and people with SMI.

Limitations of the study include the fact that routine clinical data may be less complete in terms of predictor variables than data obtained from cohorts designed for research. For instance, levels of detected diabetes and hypertension may have been underestimated if people had not received screening. However, it is reassuring that the proportion of individuals with diabetes in the PRIMROSE sample is intermediate between published meta-analyses of diabetes risk in people with treated and untreated psychoses.⁵ Although UK primary care databases have been used for CVD risk score development studies, we acknowledge that the outcome of CVD is based on general practitioner diagnostic codes rather than review of the whole medical record to establish CVD diagnoses. However, general practitioner records of coronary heart disease have been validated by medical record review in THIN.²⁵

One advantage of routine clinical data is that they are reflective of the data available in primary care and the setting in which the risk scores will be predominantly used clinically. Furthermore, we used multiple imputation techniques to impute the missing data that use the entire patient record, taking into account the temporal patterns of the records rather than relying solely on baseline measurement. We accept that imputation does not guarantee that problems with incomplete data are eliminated. As with many risk models, we only accounted for baseline variables. For many time-varying factors, exposure status may change during the follow-up period. This possibility for change is particularly true for variables such as antipsychotic exposure (first and second generation) and BMI. However, using baseline variables reflects the real-life clinical information available to a physician and a participant when they need to make decisions on the likely risk of a CVD event for an individual during the next 10 years.

In terms of generalizability, the PRIMROSE population contained a diverse range of people of varying ages with SMI that included bipolar disorder, schizophrenia, and other nonorganic psychoses in primary care. Therefore, baseline exposure to medication varied, and a considerable number of patients did not receive 2 consecutive antipsychotic prescriptions during their baseline 6-month period. However, in line with other SMI studies,² half of the sample were smokers and almost 1 in 10 had a record of severe problems with alcohol. Although most people with SMI in the United Kingdom are registered with a general practitioner, including those living in community-supported accommodations, our study might have missed the very small proportion of individuals who are long-stay hospital inpatients. We did not compare our results directly with UK QRISK score¹⁹ since we did not have access to the parameters, but we did find that the PRIMROSE models were better than the more simple Cox Framingham models reestimated to the general UK population. This finding suggests that the PRIMROSE models perform better than the Q-Risk score in people with SMI, but this is an area worth further exploration.

Variables such as atrial fibrillation or renal disease are included in other risk scores.¹⁹ However, the number of individuals in the present cohort with these conditions was too small to provide reliable predictions. Sufficient data were not available to examine individual antipsychotic drugs and specific interactions for subgroups of the SMI population. Furthermore, we were unable to include data regarding ethnicity since historically this variable has not been recorded systematically in primary care. However, because the present models perform well, adding more variables might make them unnecessarily complicated without improving their performance.

The results of the present study suggest that the newly developed SMI-specific PRIMROSE CVD risk prediction models offer improved prediction of CVD in people with SMI compared with other CVD risk scores. The new models could therefore be a valuable tool in the prevention and management of CVD in people with SMI. We recommend analysis of the cost-effectiveness of the new models when used for making treatment decisions in routine clinical practice, as well as analyses to identify the optimum threshold for risk modification. Traditionally, the optimal threshold for risk modification in the general population has been set to 20%, and this is the risk threshold we assessed in this study. However, we do not know whether this is the optimal risk threshold for risk modification in people with SMI. Furthermore, since we conducted this study, there have been changes to international CVD risk management recommendations, precipitating wide-ranging debate. The UK National Institute for Health and Care Excellence suggested initiating therapy with statins at a 10% threshold,²⁶ and the American College of Cardiology/American Heart Association recommended intervening at a 7.5% CVD risk.²⁷ These recommendations reflect a general focus on using CVD risk scores to determine the threshold for statin initiation and moving away from “treating to target” with lipid levels. Our study did not aim to determine the best threshold for initiating statin therapy, but we need to perform this work in populations with SMI to assess the effectiveness of various interventions in reducing CVD risk. Furthermore, the use of single thresholds to make treatment decisions has been criticized as simplistic and neglects to include patients’ preference for interventions as well as the true risk vs benefit for an individual when data are imperfect and derived at the population level. The PRIMROSE models, like other UK-derived risk models, would lead to fewer people being treated with statins in the United Kingdom even if a conventional 20% threshold were used because US published CVD risk models, such as the Cox Framingham model, tend to overpredict CVD in the UK population.

The PRIMROSE models outperformed the Cox Framingham models in the SMI population; however, the discrimination statistics for the published Cox Framingham models were still good. Therefore, clinicians should currently use existing CVD risk prediction tools in people with SMI, but they should preferably use tools that are calibrated or reestimated to their local general population rather than the published Cox Framingham equations from North American populations that may overpredict CVD risk in other settings regardless of SMI status.

Submitted for Publication: April 7, 2014; final revision received July 15, 2014; accepted July 29, 2014.

Corresponding Author: David P. J. Osborn, PhD, Division of Psychiatry, University College London, Charles Bell House, Riding House Street, London W1W 7EJ, United Kingdom (d.osborn@ucl.ac.uk).

Published Online: December 23, 2014. doi:10.1001/jamapsychiatry.2014.2133.

Author Contributions: Drs Osborn and Hardoon contributed equally to the study. Drs Osborn and Hardoon had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Osborn, Omar, Holt, King, Morris, Nazareth, Walters, Petersen.

Acquisition, analysis, or interpretation of data: Osborn, Hardoon, Omar, King, Larsen, Marston, Morris, Nazareth, Walters, Petersen.

Drafting of the manuscript: Osborn, Omar, Petersen.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Hardoon, Omar, Marston.

Obtained funding: Osborn, Omar, Holt, King, Marston, Nazareth, Walters, Petersen.

Administrative, technical, or material support: Osborn, Nazareth.

Study supervision: Osborn, Omar, Morris, Nazareth, Walters, Petersen.

Conflict of Interest Disclosures: None reported.

Funding/Support: This article summarizes independent research funded by the National Institute for Health Research (NIHR) under its National Institute for Health Research’s Programme Grants for Applied Research Programme (grant reference number RP-PG-0609-10156).

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The views expressed are those of the authors and not necessarily those of the National Health Service, the NIHR, or the Department of Health.