[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Download PDF
Figure.
Calibration Plots: Prediction and Management of Cardiovascular Risk in People With Severe Mental Illnesses (PRIMROSE) Models and Published Cox Framingham Models
Calibration Plots: Prediction and Management of Cardiovascular Risk in People With Severe Mental Illnesses (PRIMROSE) Models and Published Cox Framingham Models

Bold line indicates perfect fit between predicted risk and observed events. In a situation of perfect calibration, the plotted points would lie on this line. Points above the line indicate underprediction; points below the line indicate overprediction. BMI indicates body mass index; CVD, cardiovascular disease.

Table 1.  
Characteristics of Eligible Patients With SMI Data Before Imputation
Characteristics of Eligible Patients With SMI Data Before Imputation
Table 2.  
Associations of Predictors With New-Onset CVD After Imputation
Associations of Predictors With New-Onset CVD After Imputation
Table 3.  
Coefficients of Final PRIMROSE Models After Backward Eliminations
Coefficients of Final PRIMROSE Models After Backward Eliminations
Table 4.  
Performance of PRIMROSE and Published Cox Framingham Models in Predicting CVD Events in the SMI Cohort
Performance of PRIMROSE and Published Cox Framingham Models in Predicting CVD Events in the SMI Cohort
1.
De Hert  M, Dekker  JM, Wood  D, Kahl  KG, Holt  RI, Möller  HJ.  Cardiovascular disease and diabetes in people with severe mental illness: position statement from the European Psychiatric Association (EPA), supported by the European Association for the Study of Diabetes (EASD) and the European Society of Cardiology (ESC). Eur Psychiatry. 2009;24(6):412-424.
PubMedArticle
2.
Osborn  DPJ, Levy  G, Nazareth  I, Petersen  I, Islam  A, King  MB.  Relative risk of cardiovascular and cancer mortality in people with severe mental illness from the United Kingdom’s General Practice Research Database. Arch Gen Psychiatry. 2007;64(2):242-249.
PubMedArticle
3.
Osborn  DPJ, Wright  CA, Levy  G, King  MB, Deo  R, Nazareth  I.  Relative risk of diabetes, dyslipidaemia, hypertension and the metabolic syndrome in people with severe mental illnesses: systematic review and metaanalysis. BMC Psychiatry. 2008;8:84.
PubMedArticle
4.
Lieberman  JA, Stroup  TS, McEvoy  JP,  et al; Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) Investigators.  Effectiveness of antipsychotic drugs in patients with chronic schizophrenia. N Engl J Med. 2005;353(12):1209-1223.
PubMedArticle
5.
Mitchell  AJ, Vancampfort  D, De Herdt  A, Yu  W, De Hert  M.  Is the prevalence of metabolic syndrome and metabolic abnormalities increased in early schizophrenia? a comparative meta-analysis of first episode, untreated and treated patients. Schizophr Bull. 2013;39(2):295-305.
PubMedArticle
6.
De Hert  M, Correll  CU, Bobes  J,  et al.  Physical illness in patients with severe mental disorders, I: prevalence, impact of medications and disparities in health care. World Psychiatry. 2011;10(1):52-77.
PubMed
7.
Tiihonen  J, Lönnqvist  J, Wahlbeck  K,  et al.  11-Year follow-up of mortality in patients with schizophrenia: a population-based cohort study (FIN11 study). Lancet. 2009;374(9690):620-627.
PubMedArticle
8.
Crump  C, Winkleby  MA, Sundquist  K, Sundquist  J.  Comorbidities and mortality in persons with schizophrenia: a Swedish national cohort study. Am J Psychiatry. 2013;170(3):324-333.
PubMedArticle
9.
De Hert  M, Correll  CU, Cohen  D.  Do antipsychotic medications reduce or increase mortality in schizophrenia? a critical appraisal of the FIN-11 study. Schizophr Res. 2010;117(1):68-74.
PubMedArticle
10.
D’Agostino  RB  Sr, Vasan  RS, Pencina  MJ,  et al.  General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008;117(6):743-753.
PubMedArticle
11.
National Institute for Health and Clinical Excellence. Schizophrenia: core interventions in the treatment and management of schizophrenia in adults in primary and secondary care: NICE clinical guideline 82. London, United Kingdom: National Institute for Health and Clinical Excellence; 2009. http://www.nice.org.uk/CG082. Accessed November 17, 2014.
12.
The Health Improvement Network home page.http://www.THIN-UK.com. Accessed January 18, 2014.
13.
Davé  S, Petersen  I.  Creating medical and drug code lists to identify cases in primary care databases. Pharmacoepidemiol Drug Saf. 2009;18(8):704-707.
PubMedArticle
14.
Blak  BT, Thompson  M, Dattani  H, Bourke  A.  Generalisability of The Health Improvement Network (THIN) database: demographics, chronic disease prevalence and mortality rates. Inform Prim Care. 2011;19(4):251-255.
PubMed
15.
Lis  Y, Mann  RD.  The VAMP Research multi-purpose database in the U.K. J Clin Epidemiol. 1995;48(3):431-443.
PubMedArticle
16.
Horsfall  L, Walters  K, Petersen  I.  Identifying periods of acceptable computer usage in primary care research databases. Pharmacoepidemiol Drug Saf. 2013;22(1):64-69.
PubMedArticle
17.
Fardet  L, Petersen  I, Nazareth  I.  Risk of cardiovascular events in people prescribed glucocorticoids with iatrogenic Cushing’s syndrome: cohort study. BMJ. 2012;345:e4928. doi:10.1136/bmj.e4928.
PubMedArticle
18.
Horsfall  LJ, Nazareth  I, Petersen  I.  Cardiovascular events as a function of serum bilirubin levels in a large, statin-treated cohort. Circulation. 2012;126(22):2556-2564.
PubMedArticle
19.
Collins  GS, Altman  DG.  An independent external validation and evaluation of QRISK cardiovascular risk prediction: a prospective open cohort study. BMJ. 2009;339:b2584.
PubMedArticle
20.
Quality and outcomes framework guidance for GMS contract 2011/12.http://www.nhsemployers.org/PayAndContracts/GeneralMedicalServicesContract/QOF/Pages/ChangestoQOF2011-12.aspx. Accessed November 17, 2014.
21.
Nazareth  I, King  M, Haines  A, Rangel  L, Myers  S.  Accuracy of diagnosis of psychosis on general practice computer system. BMJ. 1993;307(6895):32-34.
PubMedArticle
22.
Uno  H, Cai  T, Pencina  MJ, D’Agostino  RB, Wei  LJ.  On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Stat Med. 2011;30(10):1105-1117.
PubMed
23.
Royston  P, Sauerbrei  W.  A new measure of prognostic separation in survival data. Stat Med. 2004;23(5):723-748.
PubMedArticle
24.
Hardoon  S, Hayes  JF, Blackburn  R,  et al.  Recording of severe mental illness in United Kingdom primary care, 2000-2010. PLoS One. 2013;8(12):e82365. doi:10.1371/journal.pone.0082365.
PubMedArticle
25.
García Rodríguez  LA, Tacconelli  S, Patrignani  P.  Role of dose potency in the prediction of risk of myocardial infarction associated with nonsteroidal anti-inflammatory drugs in the general population. J Am Coll Cardiol. 2008;52(20):1628-1636.
PubMedArticle
26.
Majeed  A.  Statins for primary prevention of cardiovascular disease. BMJ. 2014;348:g3491.
PubMedArticle
27.
Hayward  RA.  Moneyball, gambling, and the new cholesterol guidelines. Circ Cardiovasc Qual Outcomes. 2014;7(2):311-314.
PubMedArticle
Original Investigation
February 2015

Cardiovascular Risk Prediction Models for People With Severe Mental IllnessResults From the Prediction and Management of Cardiovascular Risk in People With Severe Mental Illnesses (PRIMROSE) Research Program

Author Affiliations
  • 1Division of Psychiatry, UCL (University College London), London, United Kingdom
  • 2Camden and Islington National Health Service Foundation Trust, London, United Kingdom
  • 3Research Department of Primary Care and Population Health, UCL, London, United Kingdom
  • 4Department of Statistical Science, UCL, London, United Kingdom
  • 5Human Development and Health Academic Unit, Faculty of Medicine, University of Southampton, Southampton, United Kingdom
  • 6Rethink Mental Illness, London, United Kingdom
JAMA Psychiatry. 2015;72(2):143-151. doi:10.1001/jamapsychiatry.2014.2133
Abstract

Importance  People with severe mental illness (SMI), including schizophrenia and bipolar disorder, have excess rates of cardiovascular disease (CVD). Risk prediction models validated for the general population may not accurately estimate cardiovascular risk in this group.

Objective  To develop and validate a risk model exclusive to predicting CVD events in people with SMI incorporating established cardiovascular risk factors and additional variables.

Design, Setting, and Participants  We used anonymous/deidentified data collected between January 1, 1995, and December 31, 2010, from the Health Improvement Network (THIN) to conduct a primary care, prospective cohort and risk score development study in the United Kingdom. Participants included 38 824 people with a diagnosis of SMI (schizophrenia, bipolar disorder, or other nonorganic psychosis) aged 30 to 90 years. During a median follow-up of 5.6 years, 2324 CVD events (6.0%) occurred.

Main Outcomes and Measures  Ten-year risk of the first cardiovascular event (myocardial infarction, angina pectoris, cerebrovascular accidents, or major coronary surgery). Predictors included age, sex, height, weight, systolic blood pressure, diabetes mellitus, smoking, body mass index (BMI), lipid profile, social deprivation, SMI diagnosis, prescriptions for antidepressants and antipsychotics, and reports of heavy alcohol use.

Results  We developed 2 CVD risk prediction models for people with SMI: the PRIMROSE BMI model and the PRIMROSE lipid model. These models mutually excluded lipids and BMI. In terms of discrimination, from cross-validations for men, the PRIMROSE lipid model D statistic was 1.92 (95% CI, 1.80-2.03) and C statistic was 0.80 (95% CI, 0.76-0.83) compared with 1.74 (95% CI, 1.63-1.86) and 0.78 (95% CI, 0.75-0.82) for published Cox Framingham risk scores. The corresponding results in women were 1.87 (95% CI, 1.76-1.98) and 0.79 (95% CI, 0.76-0.82) for the PRIMROSE lipid model and 1.58 (95% CI, 1.48-1.68) and 0.77 (95% CI, 0.73-0.81) for the Cox Framingham model. Discrimination statistics for the PRIMROSE BMI model were comparable to those for the PRIMROSE lipid model. Calibration plots suggested that both PRIMROSE models were superior to the Cox Framingham models.

Conclusions and Relevance  The PRIMROSE BMI and lipid CVD risk prediction models performed better in SMI compared with models that include only established CVD risk factors. Further work on the clinical effectiveness and cost-effectiveness of the PRIMROSE models is needed to ascertain the best thresholds for offering CVD interventions.

Introduction

It is well established that people with severe mental illnesses (SMIs), such as schizophrenia and bipolar disorder, have excess rates of cardiovascular disease (CVD), including myocardial infarctions and strokes.1 The risk of dying from CVD is 3-fold higher in people with SMI younger than 50 years and 2-fold in those aged 50 to 75 years.2 There has been an increase in clinical and research efforts addressing this problem, but we lack knowledge regarding the most effective ways to predict and manage cardiovascular risk in people with SMI. We know that the conventional cardiovascular risk factors, including smoking, dyslipidemia (with higher levels of total cholesterol and triglycerides and lower levels of high-density lipoprotein cholesterol [HDL-C]), diabetes mellitus, and obesity and possibly a higher prevalence of hypertension, are more common in people with SMI,3,4 especially those with well-established mental disorders.5 There may even be a shared genetic predisposition to comorbidities in patients with SMI. People with SMI are less likely to exercise, have unhealthy diets, and may receive inferior physical health care.6 Antipsychotic medications may contribute to cardiovascular risk through weight gain and its effect on glucose and lipid metabolism, although mortality studies2,7,8 that explore the role of antipsychotic medication in premature deaths show inconsistent findings. For example, an 11-year follow up study of people with schizophrenia7 reported reduced cardiovascular mortality in people treated with olanzapine and clozapine, but methodological issues in the study, particularly unmeasured confounding, have been critiqued in detail.9

For the general population, cardiovascular risk is managed by using CVD risk scores to determine the absolute risk for an individual patient and therefore the likely benefit of prescribing lipid-lowering medications and/or other interventions. The most established scores are the Framingham Heart Study risk scores,10 currently available as both a body mass index (BMI) model (including BMI but not laboratory results for blood lipid levels) and a lipid model (including total cholesterol and HDL-C but not BMI).

Clinical guidelines, such as the UK National Institute for Health and Clinical Excellence guidelines on schizophrenia, recommend more intensive screening for cardiovascular risk in people with SMI.1,11 However, we do not know how well the modifiable cardiovascular risk factors, included in models such as Cox Framingham, predict CVD risk in people with SMI. There is reason to believe that existing models may not accurately determine the high level of risk conferred by having a long-term SMI. The established scores, such as the Cox Framingham, were developed by excluding people with SMI and since then have not been tested in this population; in addition, the established scores do not consider SMI-specific exposures, such as antipsychotic medication. To our knowledge, no previous studies have assessed the performance of CVD risk prediction models in people with SMI.

Using data from a large UK primary care database, The Health Improvement Network (THIN),12 we aimed to develop and validate cardiovascular risk prediction models specific to people with SMI: Prediction and Management of Cardiovascular Risk in People With Severe Mental Illnesses (PRIMROSE) models. These new models developed from the PRIMROSE study included traditional cardiovascular risk factors and additional SMI-specific variables. We compared the performance of these new risk prediction models against existing published Cox Framingham scores in people with SMI since the Cox Framingham scores are widely used internationally and the coefficients of the scores are readily available to allow comparison. This work formed part of a program of research, PRIMROSE, which is funded by the UK National Institute for Health Research (http://www.ucl.ac.uk/primrose).

Methods
Study Design

A prospective study of anonymous data collected between January 1, 1995, and December 31, 2010, was conducted to develop and validate a 10-year risk score for predicting newly recorded cardiovascular events in people with SMI. The scheme for THIN to obtain and provide anonymous patient data to researchers was approved by the National Health Service South-East Multicentre Research Ethics Committee in 2002, and scientific approval for this study was obtained from CMD Medical Research’s Scientific Review Committee in March 2012.

Setting

We used THIN,12 a UK primary care database that includes information obtained from routine clinical practice. Primary care physicians and staff use a hierarchical system of Read codes to enter information in THIN, such as symptoms and diagnoses, during clinical appointments and administration13 that creates a longitudinal record for each patient. At the time we developed the PRIMROSE risk models, THIN included almost 10 million patients with geographic coverage broadly representative of the UK population.14 Approximately 98% of the population is registered with a general practitioner in the United Kingdom.15 THIN data are subject to a range of quality assurance procedures16 and have been successfully used in wide-ranging studies on CVDs,17,18 including cardiovascular risk score validation work in the general population.19 Primary care data are a particularly suitable source for assessing cardiovascular risk in people with SMI in the United Kingdom since most people with SMI are registered with a general practitioner whom they see frequently and most of the required data for risk scores (eg, laboratory and blood pressure measurements) are available owing to policy initiatives that provide incentives for annual cardiovascular screening.20 The SMI diagnoses have been validated in UK general practice.21

Participants

We included individuals aged 30 to 90 years with a diagnostic entry in their primary care electronic health record for an SMI at any time during their follow-up period. We defined SMI as (1) schizophrenia and schizoaffective disorder, (2) bipolar affective disorder, and (3) other nonorganic psychoses. We created lists of the diagnostic codes used by general practitioners or administrators. The codes are usually based on assessments by mental health specialists. We extracted anonymous/deidentified data between January 1, 1995, and December 31, 2010.

Main Outcome

Newly recorded fatal and nonfatal cardiovascular events were defined as a diagnostic record for myocardial infarction, angina pectoris, coronary heart disease, major coronary surgery and revascularization, cerebrovascular accident, and transient ischemic attack.

Statistical Analysis

We developed 2 PRIMROSE risk models: the BMI model and lipid model. Detailed descriptions of the follow-up period, the variables considered in our analysis, the development of the PRIMROSE risk models and their 10-fold internal cross-validation, as well as the imputation of missing data and our sample size calculation, are provided in the eAppendix in the Supplement.

In summary, we performed Cox proportional hazards regression with backward elimination to derive the PRIMROSE models. The variables considered in the models are listed in Tables 1, 2, and 3. We compared the performance of different models by calculating the C index22 and D statistic23 for discrimination, by constructing calibration plots, and by assessing the numbers of people classified as having high risk of CVD over the course of 10 years (score >20%) who went on to have a CVD event.

After comparing the PRIMROSE risk models against published Cox Framingham models, we performed 2 supplementary comparisons. First, we wanted to ascertain whether our results could be explained by differences between North American and UK source populations. Thus, we reestimated the Cox Framingham model to the UK general population and compared the PRIMROSE risk models against this reestimated model. Second, we wanted to examine whether there are any benefits with use of the PRIMROSE models beyond a simple reestimate of the coefficients for the Cox Framingham model in the SMI population. For this examination, we compared the PRIMROSE model with a model that included only the variables in the Cox Framingham risk score but with their coefficients reestimated within our SMI cohort.

Results

We identified 38 824 individuals who met eligibility criteria (eFigure 1 in the Supplement) in 430 general practices. There was a median follow-up period of 5.6 years (interquartile range, 2.5-9.2 years), which compares favorably with other European risk score studies in primary care databases.19 A total of 8020 people (20.7%) had 10 years or more of follow-up. There were 18 417 men (47.4%) in the cohort, the mean age was 49.5 years, and more patients lived in areas with greater social deprivation (Table 1). Approximately one-third of the patients had a diagnosis of schizophrenia, with marginally fewer having diagnoses of bipolar disorder and psychosis not otherwise specified. In terms of data completeness, during the period in which the longitudinal imputation model was applied (eAppendix in the Supplement provides details on imputation), 95.7% of the people with SMI had a record for smoking status, 88.9% for systolic blood pressure, 78.6% for weight, 65.5% for height, 52.3% for total cholesterol, and 43.2% for HDL-C. The remaining variables were complete. The imputed data are presented in eTable 1 in the Supplement.

There were 2324 newly recorded CVD events during the follow-up period, corresponding to a crude incidence rate of 9.72 (95% CI, 9.33-10.1) per 1000 person-years. The incidence of CVD increased with age and was higher in men within all age categories except for the very oldest category (eTable 2 in the Supplement). The most common events were ischemic or unspecified stroke (778 [33.5%] of the total events), myocardial infarction (414 [17.8%]), transient ischemic attack (349 [15.0%]), angina (325 [14.0%]), coronary heart disease unspecified (304 [13.1%]), unstable angina (65 [2.8%]), and hemorrhagic stroke (46 [2.0%]).

The associations between age, sex, deprivation, and established cardiovascular risk factors were all in the expected direction for known CVD risk factors (Table 2). After adjusting for age and sex, CVD was positively associated with total cholesterol level, weight, deprivation, blood pressure, age, smoking, and male sex and negatively associated with HDL-C level (Table 2). The SMI diagnosis, antidepressant use, and a history of heavy alcohol use were also predictive of CVD. There was no strong evidence of nonlinear associations between the continuous variables and CVD hazard with the exception of age in years, for which a log transformation improved linearity.

Development of PRIMROSE Models

For the PRIMROSE BMI model, all variables listed in Table 2 were retained in the model after backward elimination with the exception of prescription of lithium at baseline (Table 3). For the PRIMROSE lipid model, receipt of both lithium and first-generation antipsychotics were eliminated from the model; however, baseline receipt of second-generation antipsychotics and antidepressants were retained as predictors of 10-year CVD risk (Table 3). The complete formulas for the new PRIMROSE models for application in practice are available in the eAppendix in the Supplement.

Discrimination and Calibration

Both PRIMROSE risk models (BMI and lipid) performed well compared with the published Cox Framingham BMI and lipid models10 in the SMI cohort (Table 4, Figure, and eTables 3 and 4 in the Supplement). In terms of their discrimination statistics, the D statistics were better for the PRIMROSE models in both men and women (higher D scores indicate better discrimination, and an increase of 0.1 point has been defined as important23). C statistics were broadly similar (Table 4 and eTables 3 and 4 in the Supplement). The PRIMROSE lipid model discrimination was not substantially better than the PRIMROSE BMI model discrimination. The calibration plots suggested that the PRIMROSE lipid and BMI models predicted CVD risk more accurately than the published Cox Framingham models, with greater agreement between the predicted and observed risks for CVD (Figure). The published Cox Framingham models tended to overpredict CVD risk in this population, particularly among men.

Risk Classification

When high risk for CVD was defined using the conventional threshold of 20% risk over the course of 10 years, the PRIMROSE models were better at predicting the percentage of people with SMI who would go on to experience a CVD event (Table 4). For men categorized as being at high risk (>20% risk score), the proportions developing CVD within 10 years were 19.2% for the PRIMROSE lipid model and 18.6% for the PRIMROSE BMI model, with both performing better than the corresponding published Framingham CVD risk models8 (13.3% for the lipid model and 12.3% for the BMI model). For men categorized as being at low risk (<20%), slightly more experienced a CVD event when the PRIMROSE model was applied compared with the published Cox Framingham model (3.3% with the PRIMROSE lipid model, 2.0% with the published Cox Framingham lipid model) (Table 4). For women, the numbers correctly classified at the 20% threshold were similar for the PRIMROSE and published Cox Framingham lipid models.

Supplementary and Confirmatory Analyses

The calibration of the Cox Framingham model reestimated to the UK general population in the SMI cohort was good for men but somewhat poorer for women, with a degree of underprediction (eFigure 2 in the Supplement). The discrimination of this reestimated model was better than the published Framingham models; however, the PRIMROSE model still appeared somewhat superior to both (eTables 3-5 in the Supplement). Therefore, the superiority of the PRIMROSE model was not simply explained by the differences between US and UK populations.

The Cox Framingham model reestimated to the SMI cohort had good calibration (eFigure 2 in the Supplement) and discrimination (eTables 3-5 in the Supplement) for people with SMI. However, the PRIMROSE lipid model (containing the additional SMI variables) still showed better discrimination. This result suggests that the PRIMROSE model performs better for people with SMI compared with models that contain only the standard variables usually found in risk tools such as the Cox Framingham, including systolic blood pressure, smoking, BMI, and diabetes, even when these models are reestimated in people with SMI.

Discussion

To our knowledge, this is the first study to develop and assess the performance of CVD risk prediction tools in people with SMI. We derived 2 new CVD risk prediction models specific to people with SMI: the PRIMROSE lipid model and the PRIMROSE BMI model. These new PRIMROSE models include additional variables for psychiatric diagnosis, psychotropic medication at baseline, harmful use of alcohol, use of antidepressants at baseline, and a social deprivation score. Both models performed better than the available published Cox Framingham CVD risk models in predicting newly recorded CVD events in people with SMI, with better discrimination and calibration. The published Cox Framingham models appeared to overpredict CVD risk in this population, especially among men.

Overprediction is likely to reflect differences between historical North American men and contemporary UK men. This phenomenon has been observed when cardiovascular scores have been validated in non-SMI general practice populations, in which the Cox Framingham model overpredicts CVD events by 32% in UK men.19 This overprediction has been seen at all levels of CVD risk, including among men with an elevated risk equivalent to that seen in SMI.19

The PRIMROSE BMI model performed almost as well as the PRIMROSE lipid model, supporting use of the BMI model as an alternative to the lipid model in situations in which blood test results are not available. This alternative might be important for people with SMI who are reluctant to provide blood samples or when blood test results are not readily available. Although data regarding alcohol use might be inaccurately reported to general practitioners, we included only recognized severe alcohol problems in the model. The other variables in the model are available to general practitioners, including the medications they prescribe and the Townsend index for the individual’s postcode.

Reestimating the parameters of the published Cox Framingham models to the UK general population and within the SMI cohort improved the discrimination and calibration of the Cox Framingham models in men. However, the PRIMROSE models remained superior. This superiority indicates that the additional variables in PRIMROSE, such as diagnoses and medication at baseline, are important to include in the prediction of CVD risk in patients with SMI. This offering improved prediction beyond accounting for differences between the US and UK populations and between the general UK population and the population with SMI (assessed by evaluating the Framingham model reestimated to the SMI cohort). Among women, the Cox Framingham model reestimated within the SMI cohort also improved discrimination and calibration. However, the Cox Framingham model reestimated to the UK general population appeared to have poorer calibration, with a degree of underprediction. This finding may be explained by women with SMI having a particularly high CVD risk relative to women in the UK general population (greater than the differences among men with and without SMI); as such, application of the general population model for women to the SMI women relies on a degree of extrapolation. Overall, the new PRIMROSE models offer the most consistent performance for people with SMI.

The PRIMROSE models advance our understanding of the best ways to assess and manage the increased cardiovascular risk in people with SMI. Previously, we have not known whether cardiovascular events are predicted by the same risk factors in people with SMI, including smoking, diabetes, hypertension, BMI, and dyslipidemia. Our results suggest that these traditional risk factors do mediate the association between SMI and high rates of CVD events. However, a model including specific SMI diagnosis, baseline prescription of antipsychotics and antidepressants, as well as harmful use of alcohol offers improved prediction, suggesting that these variables are associated with the increased risk in CVD independent of traditional CVD risk factors. Caution is required because the observed associations of the SMI-specific variables and CVD do not necessarily imply a causal relationship between these factors and future CVD events. Rather, they are markers at baseline, and the reason for their inclusion may be related to characteristics of individuals prescribed antipsychotics and antidepressants in the UK, including issues that might directly influence the decision to prescribe these drugs. It would be incorrect to interpret our study results in the same way as epidemiological studies. Our study design was risk score development, for which we developed and validated the best risk score model for people with SMI. Variables may be retained in the final model despite the fact that their coefficients have 95% CIs that cross unity, with corresponding tests of significance outside the conventional 5% level. Furthermore, risk score development studies do not aim to adjust for important confounders; rather, they seek to derive the best prediction model.

Our study benefited from large numbers of people with SMI who are representative of the UK SMI population24 with longitudinal data regarding a range of CVD risk factors and a follow-up period long enough to provide sufficient data to develop and validate a CVD risk prediction model that included additional variables not traditionally included in such models. Finally, we carefully assessed whether the superior performance of the PRIMROSE models could be attributed to the fact that existing CVD models have been developed in different populations. However, our supplementary analyses supported the conclusion that the PRIMROSE models were better than the published Cox Framingham models even when the parameters were reestimated for the general UK population and people with SMI.

Limitations of the study include the fact that routine clinical data may be less complete in terms of predictor variables than data obtained from cohorts designed for research. For instance, levels of detected diabetes and hypertension may have been underestimated if people had not received screening. However, it is reassuring that the proportion of individuals with diabetes in the PRIMROSE sample is intermediate between published meta-analyses of diabetes risk in people with treated and untreated psychoses.5 Although UK primary care databases have been used for CVD risk score development studies, we acknowledge that the outcome of CVD is based on general practitioner diagnostic codes rather than review of the whole medical record to establish CVD diagnoses. However, general practitioner records of coronary heart disease have been validated by medical record review in THIN.25

One advantage of routine clinical data is that they are reflective of the data available in primary care and the setting in which the risk scores will be predominantly used clinically. Furthermore, we used multiple imputation techniques to impute the missing data that use the entire patient record, taking into account the temporal patterns of the records rather than relying solely on baseline measurement. We accept that imputation does not guarantee that problems with incomplete data are eliminated. As with many risk models, we only accounted for baseline variables. For many time-varying factors, exposure status may change during the follow-up period. This possibility for change is particularly true for variables such as antipsychotic exposure (first and second generation) and BMI. However, using baseline variables reflects the real-life clinical information available to a physician and a participant when they need to make decisions on the likely risk of a CVD event for an individual during the next 10 years.

In terms of generalizability, the PRIMROSE population contained a diverse range of people of varying ages with SMI that included bipolar disorder, schizophrenia, and other nonorganic psychoses in primary care. Therefore, baseline exposure to medication varied, and a considerable number of patients did not receive 2 consecutive antipsychotic prescriptions during their baseline 6-month period. However, in line with other SMI studies,2 half of the sample were smokers and almost 1 in 10 had a record of severe problems with alcohol. Although most people with SMI in the United Kingdom are registered with a general practitioner, including those living in community-supported accommodations, our study might have missed the very small proportion of individuals who are long-stay hospital inpatients. We did not compare our results directly with UK QRISK score19 since we did not have access to the parameters, but we did find that the PRIMROSE models were better than the more simple Cox Framingham models reestimated to the general UK population. This finding suggests that the PRIMROSE models perform better than the Q-Risk score in people with SMI, but this is an area worth further exploration.

Variables such as atrial fibrillation or renal disease are included in other risk scores.19 However, the number of individuals in the present cohort with these conditions was too small to provide reliable predictions. Sufficient data were not available to examine individual antipsychotic drugs and specific interactions for subgroups of the SMI population. Furthermore, we were unable to include data regarding ethnicity since historically this variable has not been recorded systematically in primary care. However, because the present models perform well, adding more variables might make them unnecessarily complicated without improving their performance.

Conclusions

The results of the present study suggest that the newly developed SMI-specific PRIMROSE CVD risk prediction models offer improved prediction of CVD in people with SMI compared with other CVD risk scores. The new models could therefore be a valuable tool in the prevention and management of CVD in people with SMI. We recommend analysis of the cost-effectiveness of the new models when used for making treatment decisions in routine clinical practice, as well as analyses to identify the optimum threshold for risk modification. Traditionally, the optimal threshold for risk modification in the general population has been set to 20%, and this is the risk threshold we assessed in this study. However, we do not know whether this is the optimal risk threshold for risk modification in people with SMI. Furthermore, since we conducted this study, there have been changes to international CVD risk management recommendations, precipitating wide-ranging debate. The UK National Institute for Health and Care Excellence suggested initiating therapy with statins at a 10% threshold,26 and the American College of Cardiology/American Heart Association recommended intervening at a 7.5% CVD risk.27 These recommendations reflect a general focus on using CVD risk scores to determine the threshold for statin initiation and moving away from “treating to target” with lipid levels. Our study did not aim to determine the best threshold for initiating statin therapy, but we need to perform this work in populations with SMI to assess the effectiveness of various interventions in reducing CVD risk. Furthermore, the use of single thresholds to make treatment decisions has been criticized as simplistic and neglects to include patients’ preference for interventions as well as the true risk vs benefit for an individual when data are imperfect and derived at the population level. The PRIMROSE models, like other UK-derived risk models, would lead to fewer people being treated with statins in the United Kingdom even if a conventional 20% threshold were used because US published CVD risk models, such as the Cox Framingham model, tend to overpredict CVD in the UK population.

The PRIMROSE models outperformed the Cox Framingham models in the SMI population; however, the discrimination statistics for the published Cox Framingham models were still good. Therefore, clinicians should currently use existing CVD risk prediction tools in people with SMI, but they should preferably use tools that are calibrated or reestimated to their local general population rather than the published Cox Framingham equations from North American populations that may overpredict CVD risk in other settings regardless of SMI status.

Back to top
Article Information

Submitted for Publication: April 7, 2014; final revision received July 15, 2014; accepted July 29, 2014.

Corresponding Author: David P. J. Osborn, PhD, Division of Psychiatry, University College London, Charles Bell House, Riding House Street, London W1W 7EJ, United Kingdom (d.osborn@ucl.ac.uk).

Published Online: December 23, 2014. doi:10.1001/jamapsychiatry.2014.2133.

Author Contributions: Drs Osborn and Hardoon contributed equally to the study. Drs Osborn and Hardoon had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Osborn, Omar, Holt, King, Morris, Nazareth, Walters, Petersen.

Acquisition, analysis, or interpretation of data: Osborn, Hardoon, Omar, King, Larsen, Marston, Morris, Nazareth, Walters, Petersen.

Drafting of the manuscript: Osborn, Omar, Petersen.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Hardoon, Omar, Marston.

Obtained funding: Osborn, Omar, Holt, King, Marston, Nazareth, Walters, Petersen.

Administrative, technical, or material support: Osborn, Nazareth.

Study supervision: Osborn, Omar, Morris, Nazareth, Walters, Petersen.

Conflict of Interest Disclosures: None reported.

Funding/Support: This article summarizes independent research funded by the National Institute for Health Research (NIHR) under its National Institute for Health Research’s Programme Grants for Applied Research Programme (grant reference number RP-PG-0609-10156).

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The views expressed are those of the authors and not necessarily those of the National Health Service, the NIHR, or the Department of Health.

References
1.
De Hert  M, Dekker  JM, Wood  D, Kahl  KG, Holt  RI, Möller  HJ.  Cardiovascular disease and diabetes in people with severe mental illness: position statement from the European Psychiatric Association (EPA), supported by the European Association for the Study of Diabetes (EASD) and the European Society of Cardiology (ESC). Eur Psychiatry. 2009;24(6):412-424.
PubMedArticle
2.
Osborn  DPJ, Levy  G, Nazareth  I, Petersen  I, Islam  A, King  MB.  Relative risk of cardiovascular and cancer mortality in people with severe mental illness from the United Kingdom’s General Practice Research Database. Arch Gen Psychiatry. 2007;64(2):242-249.
PubMedArticle
3.
Osborn  DPJ, Wright  CA, Levy  G, King  MB, Deo  R, Nazareth  I.  Relative risk of diabetes, dyslipidaemia, hypertension and the metabolic syndrome in people with severe mental illnesses: systematic review and metaanalysis. BMC Psychiatry. 2008;8:84.
PubMedArticle
4.
Lieberman  JA, Stroup  TS, McEvoy  JP,  et al; Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) Investigators.  Effectiveness of antipsychotic drugs in patients with chronic schizophrenia. N Engl J Med. 2005;353(12):1209-1223.
PubMedArticle
5.
Mitchell  AJ, Vancampfort  D, De Herdt  A, Yu  W, De Hert  M.  Is the prevalence of metabolic syndrome and metabolic abnormalities increased in early schizophrenia? a comparative meta-analysis of first episode, untreated and treated patients. Schizophr Bull. 2013;39(2):295-305.
PubMedArticle
6.
De Hert  M, Correll  CU, Bobes  J,  et al.  Physical illness in patients with severe mental disorders, I: prevalence, impact of medications and disparities in health care. World Psychiatry. 2011;10(1):52-77.
PubMed
7.
Tiihonen  J, Lönnqvist  J, Wahlbeck  K,  et al.  11-Year follow-up of mortality in patients with schizophrenia: a population-based cohort study (FIN11 study). Lancet. 2009;374(9690):620-627.
PubMedArticle
8.
Crump  C, Winkleby  MA, Sundquist  K, Sundquist  J.  Comorbidities and mortality in persons with schizophrenia: a Swedish national cohort study. Am J Psychiatry. 2013;170(3):324-333.
PubMedArticle
9.
De Hert  M, Correll  CU, Cohen  D.  Do antipsychotic medications reduce or increase mortality in schizophrenia? a critical appraisal of the FIN-11 study. Schizophr Res. 2010;117(1):68-74.
PubMedArticle
10.
D’Agostino  RB  Sr, Vasan  RS, Pencina  MJ,  et al.  General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008;117(6):743-753.
PubMedArticle
11.
National Institute for Health and Clinical Excellence. Schizophrenia: core interventions in the treatment and management of schizophrenia in adults in primary and secondary care: NICE clinical guideline 82. London, United Kingdom: National Institute for Health and Clinical Excellence; 2009. http://www.nice.org.uk/CG082. Accessed November 17, 2014.
12.
The Health Improvement Network home page.http://www.THIN-UK.com. Accessed January 18, 2014.
13.
Davé  S, Petersen  I.  Creating medical and drug code lists to identify cases in primary care databases. Pharmacoepidemiol Drug Saf. 2009;18(8):704-707.
PubMedArticle
14.
Blak  BT, Thompson  M, Dattani  H, Bourke  A.  Generalisability of The Health Improvement Network (THIN) database: demographics, chronic disease prevalence and mortality rates. Inform Prim Care. 2011;19(4):251-255.
PubMed
15.
Lis  Y, Mann  RD.  The VAMP Research multi-purpose database in the U.K. J Clin Epidemiol. 1995;48(3):431-443.
PubMedArticle
16.
Horsfall  L, Walters  K, Petersen  I.  Identifying periods of acceptable computer usage in primary care research databases. Pharmacoepidemiol Drug Saf. 2013;22(1):64-69.
PubMedArticle
17.
Fardet  L, Petersen  I, Nazareth  I.  Risk of cardiovascular events in people prescribed glucocorticoids with iatrogenic Cushing’s syndrome: cohort study. BMJ. 2012;345:e4928. doi:10.1136/bmj.e4928.
PubMedArticle
18.
Horsfall  LJ, Nazareth  I, Petersen  I.  Cardiovascular events as a function of serum bilirubin levels in a large, statin-treated cohort. Circulation. 2012;126(22):2556-2564.
PubMedArticle
19.
Collins  GS, Altman  DG.  An independent external validation and evaluation of QRISK cardiovascular risk prediction: a prospective open cohort study. BMJ. 2009;339:b2584.
PubMedArticle
20.
Quality and outcomes framework guidance for GMS contract 2011/12.http://www.nhsemployers.org/PayAndContracts/GeneralMedicalServicesContract/QOF/Pages/ChangestoQOF2011-12.aspx. Accessed November 17, 2014.
21.
Nazareth  I, King  M, Haines  A, Rangel  L, Myers  S.  Accuracy of diagnosis of psychosis on general practice computer system. BMJ. 1993;307(6895):32-34.
PubMedArticle
22.
Uno  H, Cai  T, Pencina  MJ, D’Agostino  RB, Wei  LJ.  On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Stat Med. 2011;30(10):1105-1117.
PubMed
23.
Royston  P, Sauerbrei  W.  A new measure of prognostic separation in survival data. Stat Med. 2004;23(5):723-748.
PubMedArticle
24.
Hardoon  S, Hayes  JF, Blackburn  R,  et al.  Recording of severe mental illness in United Kingdom primary care, 2000-2010. PLoS One. 2013;8(12):e82365. doi:10.1371/journal.pone.0082365.
PubMedArticle
25.
García Rodríguez  LA, Tacconelli  S, Patrignani  P.  Role of dose potency in the prediction of risk of myocardial infarction associated with nonsteroidal anti-inflammatory drugs in the general population. J Am Coll Cardiol. 2008;52(20):1628-1636.
PubMedArticle
26.
Majeed  A.  Statins for primary prevention of cardiovascular disease. BMJ. 2014;348:g3491.
PubMedArticle
27.
Hayward  RA.  Moneyball, gambling, and the new cholesterol guidelines. Circ Cardiovasc Qual Outcomes. 2014;7(2):311-314.
PubMedArticle
×