Deconstructing Pretest Risk Enrichment to Optimize Prediction of Psychosis in Individuals at Clinical High Risk

Key Points

Question  What are the factors modulating pretest risk of psychosis onset in help-seeking individuals referred for clinical assessment on suspicion of psychosis risk?

Findings  This cohort study of 710 individuals assessed for suspected psychosis risk indicated substantial 6-year pretest risk enrichment (15%) and provided a stratification model that is based on race/ethnicity and source of referral.

Meaning  Stratification of pretest risk enrichment in individuals undergoing assessment for suspected psychosis risk may inform outreach campaigns and subsequent testing and eventually optimize psychosis prediction.

Importance  Pretest risk estimation is routinely used in clinical medicine to inform further diagnostic testing in individuals with suspected diseases. To our knowledge, the overall characteristics and specific determinants of pretest risk of psychosis onset in individuals undergoing clinical high risk (CHR) assessment are unknown.

Objectives  To investigate the characteristics and determinants of pretest risk of psychosis onset in individuals undergoing CHR assessment and to develop and externally validate a pretest risk stratification model.

Design, Setting, and Participants  Clinical register-based cohort study. Individuals were drawn from electronic, real-world, real-time clinical records relating to routine mental health care of CHR services in South London and the Maudsley National Health Service Trust in London, United Kingdom. The study included nonpsychotic individuals referred on suspicion of psychosis risk and assessed by the Outreach and Support in South London CHR service from 2002 to 2015. Model development and validation was performed with machine-learning methods based on Least Absolute Shrinkage and Selection Operator for Cox proportional hazards model.

Main Outcomes and Measures  Pretest risk of psychosis onset in individuals undergoing CHR assessment. Predictors included age, sex, age × sex interaction, race/ethnicity, socioeconomic status, marital status, referral source, and referral year.

Results  A total of 710 nonpsychotic individuals undergoing CHR assessment were included. The mean age was 23 years. Three hundred ninety-nine individuals were men (56%), their race/ethnicity was heterogenous, and they were referred from a variety of sources. The cumulative 6-year pretest risk of psychosis was 14.55% (95% CI, 11.71% to 17.99%), confirming substantial pretest risk enrichment during the recruitment of individuals undergoing CHR assessment. Race/ethnicity and source of referral were associated with pretest risk enrichment. The predictive model based on these factors was externally validated, showing moderately good discrimination and sufficient calibration. It was used to stratify individuals undergoing CHR assessment into 4 classes of pretest risk (6-year): low, 3.39% (95% CI, 0.96% to 11.56%); moderately low, 11.58% (95% CI, 8.10% to 16.40%); moderately high, 23.69% (95% CI, 16.58% to 33.20%); and high, 53.65% (95% CI, 36.78% to 72.46%).

Conclusions and Relevance  Significant risk enrichment occurs before individuals are assessed for a suspected CHR state. Race/ethnicity and source of referral are associated with pretest risk enrichment in individuals undergoing CHR assessment. A stratification model can identify individuals at differential pretest risk of psychosis. Identification of these subgroups may inform outreach campaigns and subsequent testing and eventually optimize psychosis prediction.

The detection of individuals at clinical high risk (CHR) of developing psychosis¹ is increasingly recognized as an important component of clinical services for early psychosis intervention² (eg, UK National Institute for Health Care and Excellence guidelines,³ UK National Health Service England Access and Waiting Time standard,² and the DSM-5⁴). Relying solely on the CHR signs and symptoms leads to a correct 2-year disease prediction in approximately one-third of cases.⁵ To improve psychosis prediction, several prognostic models have been applied to stratify the risk levels of individuals who have been assessed on suspicion of psychosis risk and test positive for the CHR criteria.^6,7

However, evidence suggests that risk enrichment occurs even before individuals undergo a CHR assessment (pretest risk) and are assigned to a test outcome (posttest risk).⁸ Therefore, the degree of risk associated with meeting CHR criteria depends on the variance of pretest risk enrichment in the individuals being assessed (eg, lower pretest risk dilutes the posttest risk).^9,10 The meta-analytical pretest risk (within 38 months) of psychosis across 11 independent studies conducted worldwide (Europe, North America, Australia, and Asia; n = 2519) was 15%, with high heterogeneity (95% CI, 9% to 24%)¹¹ across sites. The meta-analytical clinical gain of testing positive for CHR was relatively modest (as indexed by a small positive likelihood ratio⁸ of 1.82⁹) and associated with a 26% risk of developing psychosis (within 38 months),⁹ in line with previous estimates.¹² Thus, pretest risk enrichment in these samples is substantial and heterogeneous,¹¹ accounting for most of their actual risk (15 of 26 [58%]).⁹ Despite this, to our knowledge, predictors of pretest risk enrichment in individuals with suspected CHR are unknown. Meta-analytical evidence suggests that the type of recruitment strategies and outreach campaigns may affect pretest risk enrichment in individuals undergoing CHR assessment.¹¹ On the basis of existing knowledge, risk factors associated with psychosis, age,^13,14 sex,¹³ interaction of age × sex,¹³ race/ethnicity,¹³ socioeconomic status,¹⁵ marital status,¹⁶ and referral year¹² may additionally modulate pretest risk enrichment in individuals undergoing CHR assessment. Characterizing and understanding pretest risk enrichment is necessary to optimize psychosis prediction¹⁰ and ultimately improve the clinical practice.

We present here what is, to our knowledge, the first original study exploring the characteristics of pretest risk of psychosis onset in a large sample of individuals undergoing CHR assessment during a long-term follow-up period. We additionally investigated potential predictors of pretest risk in individuals with a suspected CHR state and suggested a clinical pretest risk stratification model.

We included all nonpsychotic individuals assessed on suspicion of psychosis risk by the Outreach and Support in South London (OASIS) CHR service.¹⁷ Approval for the study was granted by the Oxfordshire Research Ethics Committee C. All individuals referred to the OASIS in the period between January 1, 2002, and December 31, 2015, were initially considered eligible. Procedures for anonymized data linkage under Section 251 of the UK NHS Act 2006 are fully detailed elsewhere. Because the data set was made up of deidentified data, informed consent was not required. ¹⁸ Then, we discharged those who were referred but never assessed by the team and those who were already psychotic at baseline. The remaining sample was therefore composed of all nonpsychotic individuals undergoing a Comprehensive Assessment of At Risk Mental State–based CHR assessment¹⁹ at the OASIS. Details of the clinical care received at the OASIS team have been described elsewhere.²⁰

The primary outcome of interest for the study was the cumulative pretest risk of developing psychosis in nonpsychotic individuals undergoing a CHR assessment. Psychosis onset was defined by the presence of International Statistical Classification of Diseases and Related Health Problems, Tenth Revision²¹ diagnoses of psychotic disorders in the electronic clinical records. Time to diagnosis of a psychotic disorder was measured from the date of first referral to OASIS, censored at February 1, 2016.

The predictors of the pretest risk in individuals undergoing a CHR assessment were age,¹³ sex,¹³ race/ethnicity¹³ (black, white, Asian, Caribbean, mixed, and other), marital status¹⁶ (married, divorced or separated, single, and in a relationship), referral year¹² (2002-2005, 2006-2010, and 2011-2015), referral source¹¹ (self, caregivers or relatives, schools and colleges, social services or supported accommodations, general medical practitioners, community mental health services, inpatient mental health services, child and adolescent mental health services, early intervention for psychosis services, accident and emergency departments, police and criminal justice system, and physical health services), socioeconomic status¹⁵ (index of multiple deprivation²²) and the interaction of age × sex.¹³ All predictor data obtained were those closest to the time of first referral to OASIS. Antipsychotic exposure during follow-up was additionally extracted as a confounding factor.

This was a clinical register-based cohort study. Outcome and predictor measures were automatically extracted with the use of the Clinical Record Interactive Search tool¹⁸ (see eMethods in the Supplement).

Sociodemographic characteristics of the sample were described with mean and standard deviation for continuous variables and absolute and relative frequencies for categorical variables. To investigate the characteristics of pretest risk of psychosis onset in individuals undergoing the CHR assessment, we reported its cumulative incidence, estimated by the Kaplan-Meier failure function (1-survival)²³ and Greenwood 95% CIs.²⁴ We further used smoothed curves to describe the baseline hazard function,²⁵ computed with kernel density estimation.

Model development and validation followed the guidelines of Royston et al²⁶ and was performed with a machine-learning method that is recommended in the TRIPOD checklist,²⁷ the Least Absolute Shrinkage and Selection Operator (LASSO) for Cox proportional hazards model. Least Absolute Shrinkage and Selection Operator is a penalized regression analysis method that performs both variable selection and regularization (shrinking the sum of the absolute values of the regression coefficients²⁸) to enhance the prediction accuracy and interpretability of the statistical model it produces. The LASSO Cox regression analysis is particularly indicated to control overfitting problems when the number of events is small.²⁹ We used k-fold cross-validation (repeated 100 times) to find the optimal value of shrinkage parameter λ that gives the minimum mean cross-validated error. Because of significant sociodemographic differences between the borough of Lambeth and the other boroughs (see Table 1 and Figure 2 and 3 from Perera et al³⁰), we used nonrandom split-sample development and external validation,²⁷ with the Lambeth cases in the derivation sample and all the other cases in the validation sample. First, the model with all the candidate predictors (categorical predictors were split into dummy variables) was fitted to the derivation data to estimate the optimal regression coefficients. Applying the model selected by LASSO to each case of the derivation data set, we then generated individual prognostic scores, allowing a 4-level (defined with the 25th, 75th, and 95th percentiles) prognostic index (PI) for pretest risk to be developed in the derivation data set.³¹ The regression coefficients estimated in the derivation data sets were then applied to each case in the validation data set to generate prognostic scores and the PI. Model performance was assessed with the calibration slope (discrimination, model fit),²⁶ calibration intercept (calibration),³² Kaplan-Meier curves for risk groups (discrimination, calibration),²⁶ Harrel c-index (discrimination),²⁶ and hazard ratios (HRs) across risk groups (discrimination).²⁶ In a further step, we updated the model, refitting it to the whole sample rerunning the LASSO, and we tested the confounding effect of antipsychotic exposure. As a supplementary analysis, we additionally reported the risk of psychosis in individuals referred for but not undergoing a CHR assessment. All analyses were conducted in Stata 13 (StataCorp) and R 3.3.0 (package glmnet version 2.0-5; R Programming).

From 2002 to 2015, a total of 1115 individuals were referred to the OASIS clinic for CHR assessment. Among them, 125 individuals did not undergo the CHR assessment and had no contacts with the OASIS service. An additional 280 individuals were already experiencing psychosis at baseline (the clinical outcome of these individuals is described elsewhere³³). A final sample of 710 nonpsychotic individuals who underwent the CHR assessment was used in the study (Table 1).

The mean follow-up was 1472 days (median, 1181; range, 8-5015). The mean age was 23 years, and 399 were men (56%). Half of the sample was of white race/ethnicity. Most were single. Approximately one-third of referrals (34%) came from general practitioners. The index of multiple deprivation score was 32% (eResults 1 in the Supplement). Characteristics of the derivation and validation data sets are appended in eTable 1 in the Supplement.

The cumulative incidence (Kaplan-Meier failure function) of pretest risk of psychosis in the 710 individuals undergoing the CHR assessment is depicted in Figure 1. There were 570 individuals at risk at 1 year, 445 at 2 years, 370 at 3 years, 308 at 4 years, and 260 at 5 years. The cumulative pretest risk of psychosis was 14.55% at 6 years (95% CI, 11.71% to 17.99%). The Kaplan-Meier survival function and the smoothed hazard function are reported in eFigure 1 and eFigure 2 in the Supplement, respectively. There were 81 events, with the last transition observed at day 2192 (ie, at 6.01 years) (see eResults 2 in the Supplement for the specific diagnoses), when 193 individuals were still at risk. The mean (SE) time to event was 4376 (67) days (ie, 11.98 years).

The LASSO Cox regression analysis in the derivation data set selected race/ethnicity and source of referral as predictors of pretest risk of psychosis onset. The PI showed moderately good discrimination in stratifying 4 groups at differential pretest risk of psychosis. Kaplan-Meier curves and discrimination indexes in the derivation datasets are detailed in eFigure 3 and eTable 2 in the Supplement.

The PI estimated in the validation data set retained moderately good discrimination and sufficient calibration (eFigure 4 and eTable 2 in the Supplement). The calibration slope was 0.759 and not different from 1 (95% CI, 0.346-1.173), and the calibration intercept was −2.405.

The model selected in the development phase, based on race/ethnicity and source of referral, was then updated in the entire sample. The final coefficients and the equation to estimate the PI are detailed in eTable 3 in the Supplement. The PI defined 4 classes of risk that were associated with distinctive pretest risk of psychosis (Figure 2 and Table 2). Harrel c-index in the updated model was 0.70. For descriptive purposes, we also reported the point estimates of pretest risk of psychosis for different source of referral at the time of the last failure, 2192 days, in eFigure 5 in the Supplement. Race/ethnicity and source of referrals survived as predictors of pretest risk when antipsychotic exposure was entered in the model (172 individuals [26%] received antipsychotic treatment during follow-up). Supplementary analyses are reported in eFigure 6 in the Supplement.

To our knowledge, this is the first original study to describe the characteristics of pretest risk of psychosis in a large sample of individuals undergoing CHR assessment and followed up over the long term. The cumulative 6-year pretest risk of psychosis was 15%, confirming risk enrichment during the recruitment of individuals undergoing CHR assessment. Race/ethnicity and source of referral were significantly associated with pretest risk enrichment. A predictive model was externally validated and used to stratify individuals undergoing CHR assessment into 4 classes of pretest risk of psychosis.

The first aim of this study was to address characteristics of the pretest risk of psychosis in individuals referred to high-risk services and undergoing CHR assessment. In what is, to our knowledge, the largest “real-world” sample of individuals undergoing CHR assessment and with the longest follow-up published, we confirmed a 15% pretest risk of developing psychosis at the 6-year follow-up (Figure 1). This risk is well in line with previous meta-analytical estimates (which did not include these data),⁹ and it is 35 times higher than the 6-year 0.43% risk of psychosis in the local age-matched general population (Figure 3). Thus, we confirm the substantial risk enrichment occurring before the CHR assessment. We also report the pretest baseline hazard function over time (eFigure 2 in the Supplement), which indicates a higher risk in the first years following referrals and the last transition observed at 6 years. Such a rapid peak in risk of psychosis parallels the actual risk of transition to psychosis reported in CHR samples.⁵ Of relevance, in the supplementary analysis, we additionally found that individuals referred to the CHR service but not assessed had a comparable high risk of psychosis (12%) and that this level of risk was still higher than in the local general population. This may support the case for a more assertive approach to the assessment of individuals referred to CHR services but not wishing to undergo clinical testing.³⁴

The second aim was to investigate potential predictors of pretest risk of psychosis onset in individuals undergoing CHR assessment. We found that pretest risk of psychosis was modulated by race/ethnicity, with reduced risk in white individuals (HR, 0.53; eTable 3 in the Supplement) or mixed-ethnicity individuals (HR, 0.64; eTable 3 in the Supplement) and increased risk in Asian individuals (HR, 1.23; eTable 3 in the Supplement) or Caribbean individuals (HR, 1.23; eTable 3 in the Supplement). The effect of race/ethnicity on psychosis incidence has been confirmed by meta-analytical studies.¹³ There is specific evidence for Asian or Caribbean race/ethnicity to be associated with a higher risk of developing psychosis than white race/ethnicity,¹³ even after controlling for socioeconomic status³⁵ (we found no effect for the index of multiple deprivation score). We also found that source of referrals modulates pretest risk, with reduced risk in self (HR, 0.25; eTable 3 in the Supplement), caregivers or relatives (HR, 0.27; eTable 3 in the Supplement), school or colleges (HR, 0.63; eTable 3 in the Supplement), social services and supported accommodation (HR, 0.27; eTable 3 in the Supplement), child and adolescent mental health services (HR, 0.62; eTable 3 in the Supplement), police and criminal justice system referrals (HR = 0.64; eTable 3 in the Supplement) and increased risk from inpatient mental health units (HR, 7.02; eTable 3 in the Supplement), early intervention for psychosis services (HR, 2.43; eTable 3 in the Supplement), community mental health services (HR, 1.36; eTable 3 in the Supplement), accident and emergency departments (HR, 1.42; eTable 3 in the Supplement), and physical health services referrals (HR, 1.16; eTable 3 in the Supplement). These findings confirm that risk enrichment in individuals undergoing CHR assessment is dependent on the adopted recruitment strategies and therefore on the referral source.³⁶ Individuals who had passed through several adult mental health service filters, such as early intervention for psychosis services or inpatient units, show the highest risk enrichment (referrals from child and adolescent mental health services show a reduced pretest risk, in line with studies showing low transition risk in these samples³⁷). In contrast, referrals outside adult mental health (ie, self, caregiver or relatives, schools or colleges, police and criminal justice system, or social services) diluted risk enrichment.¹¹ It is possible to speculate that referrals filtered by adult first-episode psychosis or inpatient mental health services may have accumulated risk factors for psychosis (eg, comorbidities³⁸), with more prominent and functionally impairing symptoms so that transition becomes more likely (for a meta-analysis on functional impairments in CHR individuals, see Fusar-Poli et al³⁹). Variability in referral sources may also explain the high heterogeneity of pretest risk that has been observed across CHR services worldwide.^40,41 Overall, our findings inform outreach campaigns, confirming that CHR assessment should be primarily offered to selected samples of individuals “already distressed by mental problems and seeking help for them” (European Psychiatric Association recommendation n.4¹⁴) and referred from mental health services (in line with the psychometric properties of the CHR instruments⁹), in particular, from early intervention for psychosis services. This brings the CHR paradigm back to its origin. In the first months of operation (1996), the original CHR clinic (the Personal Assessment and Crisis Evaluation) received most referrals from the local early intervention for psychosis clinic (the Early Psychosis Prevention and Intervention Centre). As noted by the authors, the early intervention service “was an important factor” in the recruitment process.⁴² Our results also provide scientific support for the new Access and Waiting Time standards in the UK² that require CHR assessment to be offered to all individuals accessing early intervention for psychosis services.

Age had no effect on pretest risk. Meta-analytical evidence indicates that the incidence of psychosis increases from childhood to the age of 20 to 24 years, then decreases over time (with an age × sex interaction¹³). It is possible that we did not have enough power to detect significant age effects in the younger subgroup (there were only 30 individuals younger than 16 years in our sample). We also found no effect of referral year on pretest risk for psychosis, suggesting no changes in the patterns of risk enrichment over time. Meta-analyses confirmed no change in the incidence of psychotic disorders during the past decades.¹³

In the third aim of this study, we developed and externally validated a prognostic model to stratify pretest risk of psychosis onset in individuals undergoing a CHR assessment. The final model was based on simple variables that are easily collected in clinical practice and defined 4 distinct classes of pretest risk: low, moderately low, moderately high, and high. The high-risk class was also distinct with respect to shorter time to transition. Because the discrimination power of our model was only moderately good⁴³ (different C statistics can yield values that are as far as 0.10 apart⁴⁴), sequential testing after initial pretest risk stratification is required. This may involve an initial CHR assessment to rule out psychosis (on the basis of the large negative likelihood ratio of 0.09 yielded by CHR instruments⁹) and potential additional testing based on more sophisticated neurobiological models. The theoretical potentials of sequential testing in individuals undergoing CHR assessment have been recently illustrated by our group.⁴⁵ Pretest risk estimates and sequential testing have been used since the 1980s⁴⁶ in cardiovascular medicine and are still part of the clinical routine to guide and inform further testing and tailored treatments,⁴⁷ eg, in individuals with suspected coronary artery disease⁴⁸ (eg, those with chest pain). These individuals should receive a thorough history and physical examination to assess the risk of ischemic heart disease “before additional testing.”⁴⁹ Similarly, stratification of low, intermediate, and high pretest risk of recent-onset chest pain is currently being recommended by the UK National Institute for Health Care and Excellence clinical guidelines (CG95 1.3.3, reproduced in eTable 4 in the Supplement). As in this study, the most widely used parameters are based on simple clinical variables such as the patient’s history,⁵⁰ the description of chest pain, sex, and age⁴⁶ (eTable 4 in the Supplement). For the low-risk population, exercise treadmill testing alone is frequently sufficient; however, in individuals with a moderate to high risk for coronary artery disease, additional specific testing is usually required.^49,51

Because services and referral patterns are heterogeneous¹¹ and likely to be influenced by national and local factors, clinical validity of our model should be confirmed and refined by external replication studies conducted in other clinical scenarios. To facilitate this, we provided the required statistical information (eTable 3 and eFigure 2 in the Supplement) as recommended by international guidelines.²⁶ Other relevant limitations of this study are related to the use of the clinical case register and are fully discussed in the supplementary material (eLimitations in the Supplement).

There is substantial psychosis risk enrichment during the recruitment of individuals undergoing CHR assessment. Race/ethnicity and source of referral are associated with pretest risk enrichment. A pretest risk stratification model has been developed and externally validated, which may inform outreach campaigns and help to optimize subsequent testing and the prediction of psychosis.

Corresponding Author: Paolo Fusar-Poli, MD, PhD, RCPsych, Department of Psychosis Studies, Institute of Psychiatry, Psychology, and Neuroscience PO63, De Crespigny Park, London SE5 8AF, United Kingdom (paolo.fusar-poli@kcl.ac.uk).

Correction: In the Original Investigation titled “Deconstructing Pretest Risk Enrichment to Optimize Prediction of Psychosis in Individuals at Clinical High Risk,” published online October 26, 2016, there was an error in the x-axis label of Figure 2. The x-axis label of the graph should read as follows: “Time Since Initial Referral to OASIS, y.” This article was corrected online on December 7, 2016.

Accepted for Publication: September 2, 2016.

Published Online: October 26, 2016. doi:10.1001/jamapsychiatry.2016.2707

Author Contributions: Drs Fusar-Poli and Rutigliano had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Fusar-Poli, Rutigliano, McGuire.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Fusar-Poli, Schmidt.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Fusar-Poli, Stahl, Schmidt.

Administrative, technical, or material support: Fusar-Poli, Rutigliano, Schmidt, McGuire.

Study supervision: Fusar-Poli.

No additional contributions: Ramella-Cravaro.

Conflict of Interest Disclosures: None reported.

Funding/Support: This work was supported by the National Institute for Health Research Biomedical Research Centre at South London, Maudsley National Health Service Foundation Trust, King’s College London, and by a 2014 NARSAD Young Investigator Award to Dr Fusar-Poli. Dr Schmidt was supported by the Swiss National Science Foundation grant P2ZHP3_155184.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: We thank the Outreach and Support in South London individuals.

Disclaimer: The views expressed are those of the authors and not necessarily those of the National Health Services, the National Institute for Health Research, or the Department of Health.