[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.
Flowchart of Study Population
Flowchart of Study Population

ARMS indicates At Risk Mental State; SLaM, South London and the Maudsley.

Figure 2.
Decision Curve Analysis
Decision Curve Analysis

Decision curve analysis estimated in the validation data set, showing the potential clinical usefulness of the risk calculator at different threshold probabilities (risk of developing psychosis by 5 years) for focused interventions to prevent psychosis (assessment and care), compared with treating all patients or to treating no patients at all.

Table 1.  
Sociodemographic Characteristics of Study Population Including the Derivation and Validation Data Set
Sociodemographic Characteristics of Study Population Including the Derivation and Validation Data Set
Table 2.  
Statistics for Individual Predictor Variables in the Multivariable Cox Proportional Hazards Regression Analysis of Risk for Psychosis in the Derivation Data Set
Statistics for Individual Predictor Variables in the Multivariable Cox Proportional Hazards Regression Analysis of Risk for Psychosis in the Derivation Data Set
Table 3.  
Performance of the Risk Calculator for Transdiagnostic Prediction of Psychosis in Secondary Mental Health Care and Clinical Usefulness
Performance of the Risk Calculator for Transdiagnostic Prediction of Psychosis in Secondary Mental Health Care and Clinical Usefulness
1.
Millan  MJ, Andrieux  A, Bartzokis  G,  et al.  Altering the course of schizophrenia: progress and perspectives.  Nat Rev Drug Discov. 2016;15(7):485-515.PubMedGoogle ScholarCrossref
2.
Jääskeläinen  E, Juola  P, Hirvonen  N,  et al.  A systematic review and meta-analysis of recovery in schizophrenia.  Schizophr Bull. 2013;39(6):1296-1306.PubMedGoogle ScholarCrossref
3.
Fusar Poli  P, McGorry  P, Kane  J.  Improving outcomes of first episode psychosis.  World Psychiatry. in press.Google Scholar
4.
Yung  AR, McGorry  PD, McFarlane  CA, Jackson  HJ, Patton  GC, Rakkar  A.  Monitoring and care of young people at incipient risk of psychosis.  Schizophr Bull. 1996;22(2):283-303.PubMedGoogle ScholarCrossref
5.
Fusar-Poli  P, Cappucciati  M, Rutigliano  G,  et al.  At risk or not at risk? a meta-analysis of the prognostic accuracy of psychometric interviews for psychosis prediction.  World Psychiatry. 2015;14(3):322-332.PubMedGoogle ScholarCrossref
6.
Braham  A, Bannour  AS, Ben Romdhane  A,  et al.  Validation of the Arabic version of the Comprehensive Assessment of At Risk Mental States (CAARMS) in Tunisian adolescents and young adults.  Early Interv Psychiatry. 2014;8(2):147-154.PubMedGoogle ScholarCrossref
7.
Krebs  MO, Magaud  E, Willard  D,  et al.  Évaluation des états mentaux à risque de transition psychotique: validation de la version française de la CAARMS [Assessment of mental states at risk of psychotic transition: validation of the French version of the CAARMS].  Encephale. 2014;40(6):447-456.PubMedGoogle ScholarCrossref
8.
Fusar-Poli  P, Schultze-Lutter  F.  Predicting the onset of psychosis in patients at clinical high risk: practical guide to probabilistic prognostic reasoning.  Evid Based Ment Health. 2016;19(1):10-15.PubMedGoogle ScholarCrossref
9.
Fusar-Poli  P, Bechdolf  A, Taylor  MJ,  et al.  At risk for schizophrenic or affective psychoses? a meta-analysis of DSM/ICD diagnostic outcomes in individuals at high clinical risk.  Schizophr Bull. 2013;39(4):923-932.PubMedGoogle ScholarCrossref
10.
Webb  JR, Addington  J, Perkins  DO,  et al.  Specificity of incident diagnostic outcomes in patients at clinical high risk for psychosis.  Schizophr Bull. 2015;41(5):1066-1075.PubMedGoogle ScholarCrossref
11.
Fusar-Poli  P, Rutigliano  G, Stahl  D,  et al.  Long-term validity of the At Risk Mental State (ARMS) for predicting psychotic and non-psychotic mental disorders.  Eur Psychiatry. 2016;42:49-54.PubMedGoogle ScholarCrossref
12.
Kempton  MJ, Bonoldi  I, Valmaggia  L, McGuire  P, Fusar-Poli  P.  Speed of psychosis progression in people at ultra-high clinical risk: a complementary meta-analysis.  JAMA Psychiatry. 2015;72(6):622-623.PubMedGoogle ScholarCrossref
13.
Stafford  MR, Jackson  H, Mayo-Wilson  E, Morrison  AP, Kendall  T.  Early interventions to prevent psychosis: systematic review and meta-analysis.  BMJ. 2013;346:f185.PubMedGoogle ScholarCrossref
14.
Deas  G, Kelly  C, Hadjinicolaou  AV, Holt  C, Agius  M, Zaman  R.  An update on: meta-analysis of medical and non-medical treatments of the prodromal phase of psychotic illness in at risk mental states.  Psychiatr Danub. 2016;28(suppl 1):31-38.PubMedGoogle Scholar
15.
NICE. Psychosis and schizophrenia in children and young people: recognition and management. http://www.nice.org.uk/Guidance/CG155. Published 2013. Accessed January 17, 2016.
16.
Fusar-Poli  P, Nelson  B, Valmaggia  L, Yung  AR, McGuire  PK.  Comorbid depressive and anxiety disorders in 509 individuals with an at-risk mental state: impact on psychopathology and transition to psychosis.  Schizophr Bull. 2014;40(1):120-131.PubMedGoogle ScholarCrossref
17.
Conrad  AM, Lewin  TJ, Sly  KA,  et al.  Utility of risk-status for predicting psychosis and related outcomes: evaluation of a 10-year cohort of presenters to a specialised early psychosis community mental health service.  Psychiatry Res. 2017;247:336-344.PubMedGoogle ScholarCrossref
18.
van Os  J, Murray  RM.  Can we identify and treat “schizophrenia light” to prevent true psychotic illness?  BMJ. 2013;346:f304.PubMedGoogle ScholarCrossref
19.
Stewart  R, Soremekun  M, Perera  G,  et al.  The South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLAM BRC) case register: development and descriptive data.  BMC Psychiatry. 2009;9:51.PubMedGoogle ScholarCrossref
20.
Studerus  E, Ramyead  A, Riecher-Rössler  A.  Prediction of transition to psychosis in patients with a clinical high risk for psychosis: a systematic review of methodology and reporting.  Psychol Med. 2017;1-16.PubMedGoogle Scholar
21.
Benchimol  EI, Smeeth  L, Guttmann  A,  et al; RECORD Working Committee.  The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement.  PLoS Med. 2015;12(10):e1001885.PubMedGoogle ScholarCrossref
22.
Kaplan  E, Meier  P.  Nonparametric estimation from incomplete observations.  J Am Stat Assoc. 1958;53(282):457-481.Google ScholarCrossref
23.
Greenwood  M.  The Natural Duration of Cancer. London, England: Her Majesty's Stationery Office; 1926.
24.
Kirkbride  JB, Jackson  D, Perez  J,  et al.  A population-level prediction tool for the incidence of first-episode psychosis: translational epidemiology based on cross-sectional data.  BMJ Open. 2013;3(2):e001998.PubMedGoogle ScholarCrossref
25.
Royston  P, Altman  DG.  External validation of a Cox prognostic model: principles and methods.  BMC Med Res Methodol. 2013;13:33.PubMedGoogle ScholarCrossref
26.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138.PubMedGoogle ScholarCrossref
27.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement.  Ann Intern Med. 2015;162(1):55-63.PubMedGoogle ScholarCrossref
28.
Grambsch  P, Therneau  T.  Proportional hazards tests and diagnostics based on weighted residuals.  Biometrika. 1994;81:(3)515-526. doi:10.1093/biomet/81.3.515Google ScholarCrossref
29.
Perera  G, Broadbent  M, Callard  F,  et al.  Cohort profile of the South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLaM BRC) Case Register: current status and recent enhancement of an Electronic Mental Health Record-derived data resource.  BMJ Open. 2016;6(3):e008721.PubMedGoogle ScholarCrossref
30.
Machin  D, Cheung  Y, Parmar  M.  Survival Analysis: A Practical Approach. 2nd ed. Chichester, England: John Wiley and Sons Ltd; 2006.Crossref
31.
Royston  P.  Explained variation for survival models.  Stata J. 2006;6(1):83-96.Google Scholar
32.
Crowson  CS, Atkinson  EJ, Therneau  TM.  Assessing calibration of prognostic risk scores.  Stat Methods Med Res. 2016;25(4):1692-1706.PubMedGoogle ScholarCrossref
33.
Harrell  F. Regression modeling strategies. http://biostat.mc.vanderbilt.edu/rms. Published 2017. Accessed January 13, 2017.
34.
Collins  GS, Ogundimu  EO, Altman  DG.  Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.  Stat Med. 2016;35(2):214-226.PubMedGoogle ScholarCrossref
35.
Vickers  AJ, Van Calster  B, Steyerberg  EW.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.  BMJ. 2016;352:i6.PubMedGoogle ScholarCrossref
36.
Vickers  AJ, Cronin  AM, Elkin  EB, Gonen  M.  Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers.  BMC Med Inform Decis Mak. 2008;8:53.PubMedGoogle ScholarCrossref
37.
Hardoon  S, Hayes  JF, Blackburn  R,  et al.  Recording of severe mental illness in United Kingdom primary care, 2000-2010.  PLoS One. 2013;8(12):e82365.PubMedGoogle ScholarCrossref
38.
Francesconi  M, Minichino  A, Carrión  RE,  et al.  Psychosis prediction in secondary mental health services: a broad, comprehensive approach to the “at risk mental state” syndrome.  Eur Psychiatry. 2016;40:96-104.PubMedGoogle ScholarCrossref
39.
Fusar-Poli  P, Cappucciati  M, De Micheli  A,  et al.  Diagnostic and Prognostic Significance of Brief Limited Intermittent Psychotic Symptoms (BLIPS) in individuals at ultra high risk.  Schizophr Bull. 2017;43(1):48-56.PubMedGoogle ScholarCrossref
40.
Fusar-Poli  P, Cappucciati  M, Bonoldi  I,  et al.  Prognosis of brief psychotic episodes: a meta-analysis.  JAMA Psychiatry. 2016;73(3):211-220.PubMedGoogle ScholarCrossref
41.
Fusar-Poli  P, Cappucciati  M, Borgwardt  S,  et al.  Heterogeneity of psychosis risk within individuals at clinical high risk: a meta-analytical stratification.  JAMA Psychiatry. 2016;73(2):113-120.PubMedGoogle ScholarCrossref
42.
Fusar-Poli  P, Byrne  M, Badger  S, Valmaggia  LR, McGuire  PK.  Outreach and support in south London (OASIS), 2001-2011: ten years of early diagnosis and treatment for young individuals at high clinical risk for psychosis.  Eur Psychiatry. 2013;28(5):315-326.PubMedGoogle ScholarCrossref
43.
Schultze-Lutter  F, Michel  C, Schmidt  SJ,  et al.  EPA guidance on the early detection of clinical high risk states of psychoses.  Eur Psychiatry. 2015;30(3):405-416.PubMedGoogle ScholarCrossref
44.
Fusar-Poli  P, Rutigliano  G, Stahl  D,  et al.  Deconstructing pretest risk enrichment to optimize prediction of psychosis in individuals at clinical high risk.  JAMA Psychiatry. 2016;73(12):1260-1267.PubMedGoogle ScholarCrossref
45.
Valmaggia  LR, Byrne  M, Day  F,  et al.  Duration of untreated psychosis and need for admission in patients who engage with mental health services in the prodromal phase.  Br J Psychiatry. 2015;207(2):130-134.PubMedGoogle ScholarCrossref
46.
Woodberry  KA, Seidman  LJ, Bryant  C,  et al.  Treatment precedes positive symptoms in north american adolescent and young adult clinical high risk cohort.  J Clin Child Adolesc Psychol. 2016;1-10.PubMedGoogle Scholar
47.
Carrión  RE, Correll  CU, Auther  AM, Cornblatt  BA.  A severity-based clinical staging model for the psychosis prodrome: longitudinal findings from the new york recognition and prevention program.  Schizophr Bull. 2017;43(1):64-74.PubMedGoogle ScholarCrossref
48.
Addington  J, Liu  L, Perkins  DO, Carrion  RE, Keefe  RS, Woods  SW.  The role of cognition and social functioning as predictors in the transition to psychosis for youth with attenuated psychotic symptoms.  Schizophr Bull. 2017;43(1):57-63.PubMedGoogle ScholarCrossref
49.
Schmidt  A, Cappucciati  M, Radua  J,  et al.  Improving prognostic accuracy in subjects at clinical high risk for psychosis: systematic review of predictive models and meta-analytical sequential testing simulation.  Schizophr Bull. 2016;sbw098.PubMedGoogle Scholar
50.
Clark  SR, Baune  BT, Schubert  KO,  et al.  Prediction of transition from ultra-high risk to first-episode psychosis using a probabilistic model combining history, clinical assessment and fatty-acid biomarkers.  Transl Psychiatry. 2016;6(9):e897.PubMedGoogle ScholarCrossref
51.
Scott  J, Marwaha  S, Ratheesh  A,  et al.  Bipolar at-risk criteria: an examination of which clinical features have optimal utility for identifying youth at risk of early transition from depression to bipolar disorders.  Schizophr Bull. 2016;sbw154.PubMedGoogle Scholar
Original Investigation
May 2017

Development and Validation of a Clinically Based Risk Calculator for the Transdiagnostic Prediction of Psychosis

Author Affiliations
  • 1Early Psychosis: Interventions and Clinical Detection Lab, Department of Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, England
  • 2Outreach and Support in South London Service, South London and the Maudsley National Health Service Foundation Trust, London, England
  • 3National Institute for Health Research Biomedical Research Centre for Mental Health, IoPPN, King’s College London, London, England
  • 4Department of Clinical and Experimental Medicine, University of Pisa, Pisa, Italy
  • 5Department of Biostatistics, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, England
  • 6Department of Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, England
JAMA Psychiatry. 2017;74(5):493-500. doi:10.1001/jamapsychiatry.2017.0284
Key Points

Question  Can we improve the detection of individuals at risk of developing psychosis among patients accessing secondary mental health care?

Finding  This clinical register-based cohort study of 91 199 patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder developed and externally validated an online individualized risk calculator tool, based on simple predictor variables that are easily and routinely collected in clinical settings, for the transdiagnostic prediction of psychosis in secondary mental health care.

Meaning  This individualized risk calculator tool can be used by clinicians and researchers, facilitating the prediction of psychosis and the subsequent implementation of preventive focused interventions.

Abstract

Importance  The overall effect of At Risk Mental State (ARMS) services for the detection of individuals who will develop psychosis in secondary mental health care is undetermined.

Objective  To measure the proportion of individuals with a first episode of psychosis detected by ARMS services in secondary mental health services, and to develop and externally validate a practical web-based individualized risk calculator tool for the transdiagnostic prediction of psychosis in secondary mental health care.

Design, Setting, and Participants  Clinical register-based cohort study. Patients were drawn from electronic, real-world, real-time clinical records relating to 2008 to 2015 routine secondary mental health care in the South London and the Maudsley National Health Service Foundation Trust. The study included all patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within the South London and the Maudsley National Health Service Foundation Trust in the period between January 1, 2008, and December 31, 2015. Data analysis began on September 1, 2016.

Main Outcomes and Measures  Risk of development of nonorganic International Statistical Classification of Diseases and Related Health Problems, Tenth Revision psychotic disorders.

Results  A total of 91 199 patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within South London and the Maudsley National Health Service Foundation Trust were included in the derivation (n = 33 820) or external validation (n = 54 716) data sets. The mean age was 32.97 years, 50.88% were men, and 61.05% were white race/ethnicity. The mean follow-up was 1588 days. The overall 6-year risk of psychosis in secondary mental health care was 3.02 (95% CI, 2.88-3.15), which is higher than the 6-year risk in the local general population (0.62). Compared with the ARMS designation, all of the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnoses showed a lower risk of psychosis, with the exception of bipolar mood disorders (similar risk) and brief psychotic episodes (higher risk). The ARMS designation accounted only for a small proportion of transitions to psychosis (n = 52 of 1001; 5.19% in the derivation data set), indicating the need for transdiagnostic prediction of psychosis in secondary mental health care. A prognostic risk stratification model based on preselected variables, including index diagnosis, age, sex, age by sex, and race/ethnicity, was developed and externally validated, showing good performance and potential clinical usefulness.

Conclusions and Relevance  This online individualized risk calculator can be of clinical usefulness for the transdiagnostic prediction of psychosis in secondary mental health care. The risk calculator can help to identify those patients at risk of developing psychosis who require an ARMS assessment and specialized care. The use of this calculator may eventually facilitate the implementation of an individualized provision of preventive focused interventions and improve outcomes of first episode psychosis.

Introduction

Existing treatments for psychotic disorders have little effect on the course of illness within standard care.1,2 Prevention and early intervention may be the only available clinical possibility to alter the course of psychosis.3 Prevention of psychosis has been feasible since the introduction of the At Risk Mental State (ARMS) construct 2 decades ago.4 The ARMS construct has been validated internationally,5-7 and it can reliably identify young individuals at specific enhanced risk for the development of psychotic disorders,8 mostly schizophrenia spectrum disorders,9 but not of nonpsychotic disorders,10,11 during the 2 to 3 years after initial assessment.12 Randomized clinical trials have shown that focused interventions, if offered to ARMS individuals, can effectively reduce the risk of future illness.13,14 Owing to these unprecedented potentials, the ARMS construct has gained traction to the point that specialized assessment and treatment is recognized as a key component of secondary mental health services by National Institute for Health and Care Excellence guidelines.15

However, the overall clinical impact of the ARMS on psychosis prevention in secondary mental health care and the value of using the ARMS designation compared with standard mental diagnoses (eg, those defined by the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision [ICD-10] ) is not completely clear. For example, most individuals meeting ARMS criteria would also meet criteria for a secondary diagnosis of comorbid mental disorder, mostly depression or anxiety.16 As a result, some authors have claimed that the ARMS construct is not strictly necessary17 and that psychosis could be predicted (and prevented) within the existing diagnostic categories of common mental disorders.18 Whether we can pragmatically predict psychosis outside the ARMS designation or not remains unclear because, to our knowledge, no studies have ever addressed this issue. Such a gap of knowledge may have clinical implications for the provision of preventive intervention services and policy makers. In fact, some authors suggest that it would be better to detect and treat psychosis as it emerges from common mental disorders rather than promoting new ARMS services.18 To further compound the issue, the overall burden of psychosis risk in secondary mental health care is mostly undetermined, and it is not clear whether the ARMS designation is sufficient to pragmatically detect all individuals who will later develop a first episode of psychosis. At Risk Mental State services usually receive referrals on suspicion of psychosis risk, and their referral depends on the subjective judgement of clinicians. Consequently, it is possible that not all individuals in secondary mental health care who will later develop a first episode of psychosis would eventually be detected by ARMS services.

This study investigates for the first time, to our knowledge, the proportion of first-episode individuals detected by ARMS services in secondary mental health services as well as the transdiagnostic risk of developing psychotic disorders across any ICD-10–defined mental disorder. The primary aim was to develop and validate a clinically based, practical, individualized risk calculator tool to facilitate the transdiagnostic prediction of psychosis in secondary mental health care and increase the proportion of individuals at risk for psychosis detected by ARMS services to improve outcomes of first-episode psychosis.

Methods
Data Source

A clinical register-based cohort was selected through a Clinical Record Interactive Search tool19 (eMethods 1 in the Supplement). The study was registered through http://www.researchregistry.com, study number 1487.

Study Population

All individuals accessing South London and the Maudsley (SLaM) National Health Service Foundation Trust services in the period between January 1, 2008, and December 31, 2015, and who received a first index primary diagnosis of any nonorganic and nonpsychotic mental disorder were initially considered eligible. We then excluded those who developed psychosis in the 3 months immediately following the first index diagnosis. Approval for the study was granted by the Oxfordshire Research Ethics Committee C. Because the data set comprised deidentified data, informed consent was not required.19

Study Measures

The outcome (risk of developing any psychotic disorder), predictors (index diagnosis, age, sex, race/ethnicity, and age by sex interaction), and time to event were automatically extracted using Clinical Record Interactive Search.19 Predictors were preselected on the basis of previous meta-analytical clinical knowledge, as recommended20 (eMethods 2 and eTable 1 in the Supplement for full details).

Statistical Analysis

This clinical register-based cohort study was conducted according to the Reporting of Studies Conducted Using Observational Routinely-Collected Health Data Statement21 (eTable 2 Supplement).

Baseline clinical and sociodemographic characteristics of the sample (including missing data) were described by means and standard deviations for continuous variables, and absolute and relative frequencies for categorical variables. The overall cumulative risk of psychosis onset in SLaM patients was described with the Kaplan-Meier failure function (1-survival)22 and Greenwood 95% confidence intervals23 and was qualitatively compared with the risk of psychosis in the local general population (mean predicted cases across SLaM boroughs, estimated with PsyMaptic [http://www.psymaptic.org/]).24

Two-sided P values less than .05 were considered significant. Model development and validation followed the guidelines of Royston and Altman,25 Steyerberg et al,26 and the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis.27

Model Development

We used Cox proportional hazards multivariable complete-case analyses to evaluate the effects of the preselected predictors (index diagnosis, age, sex, race/ethnicity, and age by sex interaction) on the development of nonorganic ICD-10 psychotic disorders and time to development of psychosis, after checking the proportional hazards assumption.28 Model development was not based on stepwise methods, which are not recommended,25 but on a priori selection of predictors based on previous knowledge, as detailed in the eMethods 2 in the Supplement. Continuous variables were not dichotomized.25 Because of significant sociodemographic differences between the SLaM boroughs,29 we used nonrandom split-sample development and external validation,27 with the Lambeth and Southwark cases in the derivation sample and all other cases in the validation sample. The model with all preselected predictors was first fitted to the derivation data to estimate the optimal regression coefficients. Performance diagnostics of individual predictor variables in the derivation data set were explored with Harrell C index,25 which is similar to the area under the receiver operating characteristic curve. We then generated individual prognostic scores, allowing a prognostic index for risk of psychosis onset to be developed in the derivation data set.30 As a supplementary analysis, we fitted the model after excluding the acute and transient psychotic disorder cases.

External Model Validation

The regression coefficients as estimated in the derivation data set were then applied to each case in the external validation data set to generate the prognostic index in the validation data set. Overall model performance (the distance between the predicted outcome and actual outcome26) was assessed with the Brier score (the average mean squared difference between predicted probabilities and actual outcomes, which also captures calibration and discrimination aspects).26 A lower score indicates higher precision and less bias, but interpretation depends on the incidence of the outcome.26 Overall performance was further investigated with the Royston modification of the Nagelkerke R2 (indexing the proportion of variation explained by the model).31 Calibration (the agreement between observed outcomes and predictions26) was assessed with the regression slope of prognostic index26 (which also captures discrimination and model fit),25 with the regression intercept (calibration in the large,26 estimated as previously detailed32) and with the calibration plot (resampling model calibration with hare function).33 Discrimination (accurate predictions discriminate between those with and those without the outcome26) was addressed with Harrell C index25 and with the discrimination slope (difference in mean of predictions between outcomes).26 Studies indicate that unbiased and precise estimation of performance measures can be achieved with a minimum of 100 events in the external validation data set.34

Potential Clinical Usefulness for Psychosis Prevention

We additionally explored the potential clinical usefulness of the risk calculator as recommended by Vickers et al.35 Performance measures do not tell us whether the risk calculator would do more good than harm if used in clinical practice.35 Net benefit analyses35,36 tackle such limitations by including an exchange rate, a clinical judgment of the relative value of benefits (such as preventing psychosis in secondary mental health care) and harms (such as unnecessary treatment) associated with the predictive model (details on the exchange rate in eMethods 3 in the Supplement). Because definition of the exchange rates is subjective, we additionally plotted net benefit for a range of reasonable exchange rates in a decision curve analysis, as recommended.35 All analyses were conducted in STATA, version 13 (StataCorp) and R, version 3.3.0 (R Programming).

Results
Sociodemographic and Clinical Characteristics of the Sample

Of 92 227 patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within SLaM in the period between 2008 and 2015, 91 199 fulfilled the study inclusion criteria and were included in the derivation or validation data sets, as indicated in Figure 1.

Quiz Ref IDTable 1 shows the baseline characteristics of the study population, as well as the derivation and validation data sets. As expected, there were significant sociodemographic differences across the derivation and validation data sets (Table 1), particularly with respect to race/ethnicity and index diagnosis. The mean follow-up was 1588 days (95% CI, 1582-1595), with no differences between the derivation and validation data sets (derivation: mean, 1589; 95% CI, 1579-1599; validation: mean, 1588; 95% CI, 1580-1596). The overall 6-year risk of developing a psychotic disorder was 3.02 (95% CI, 2.88-3.15) and higher than the 6-year risk of psychosis of 0.62 in the local general population (eResults 1 and eFigure 1 in the Supplement). The baseline hazard function is reported in eFigure 2 in the Supplement.

Model Development

Quiz Ref IDIn the derivation data set, there were 1001 transitions to psychosis (52 of which were observed in the ARMS construct [5.19%], eTable 3 in the Supplement). The multivariable model significantly predicted psychosis onset (likelihood ratio χ2 test, 1767.59; P < .001).Quiz Ref ID Age and male sex were significantly associated with an increased risk of psychosis (Table 2). Across men, risk of psychosis decreased with increasing age (Table 2). Relative to white race/ethnicity, black, Asian, mixed, and other races/ethnicities were associated with an increased risk of developing psychosis (Table 2). Compared with the reference ARMS designation, all of the other ICD-10 mental disorders were associated with a lower risk of developing psychosis, with 2 exceptions (Table 2). Bipolar mood disorders and acute and transient psychotic disorders showed a comparable and higher risk of psychosis than the ARMS construct, respectively (Table 2). Post hoc analyses showed that relative to the ARMS construct, bipolar mood disorders and acute and transient psychotic disorders had a higher risk of developing affective spectrum psychoses (HR, 4.63; 95% CI, 1.66-12.94) and schizophrenia spectrum psychoses (HR, 5.46; 95% CI, 2.74-10.89), respectively. Supplementary analyses using the attenuated psychotic symptoms subgroup of the ARMS as a reference group confirmed the model, showing that the brief limited intermittent psychotic symptoms subgroup was at higher risk of developing psychosis than the attenuated psychotic symptoms subgroup. Model diagnostics using the C index are detailed in Table 2. The model showed very good overall apparent performance (good discrimination, C index, 0.80) and explained approximately 75% of the observed variation (Table 3). The model remained significant after removing the acute and transient psychotic disorder cases (eTable 4 in the Supplement).

Model Validation

Quiz Ref IDIn the external validation data set there were 1010 transitions to psychosis (12 of which were observed in the ARMS [1.19%]; eTable 3 in the Supplement), a value that greatly exceeds the minimum of 100 events required for robust external validation.34 The model retained an overall good performance and was able to explain around 72% of the observed heterogeneity (Table 3). Model discrimination was fair to good, with a C index of 0.79 (Table 3). The mean risk of psychosis in the validation data set was lower than in the derivation data set, but there were no major miscalibration issues (Table 3; eFigure 3 in the Supplement).

Potential Clinical Usefulness of the Risk Calculator

At the reference threshold for recommending focused interventions to prevent psychosis, the use of the model was associated with significant net benefits in both the derivation and validation data sets (Table 3). The decision curve estimated in the validation data set (Figure 2) shows that compared with conducting no tests, testing on the basis of the risk calculator is associated with net benefits for a 1% to 50% range of threshold probability (risk of developing psychosis by 5 years; eResults 2 in the Supplement).

An online version of the risk calculator was built to facilitate numeric calculation of the predicted probability of conversion to psychosis in secondary mental health care (http://www.psychosis-risk.net).

Discussion

A total of 91 199 patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within SLaM were included in the study, either in the derivation (33 820) or validation (54 716) data sets, with a mean follow-up of 1588 days. The overall 6-year risk of psychosis in secondary mental health care was 3.02 (95% CI, 2.88-3.15) and was higher than the 6-year risk of psychosis in the local general population (0.62). Compared with the ARMS designation, all of the other ICD-10 diagnoses were associated with a lower risk of psychosis, with 2 exceptions. Bipolar mood disorders and acute and transient psychotic disorders showed a similar and higher risk of psychosis than the ARMS, respectively. The ARMS designation accounted only for a small proportion of transitions to psychosis (n = 52 of 1001; 5.19%), indicating the need for transdiagnostic prediction of psychosis in secondary mental health care. The prognostic risk stratification model based on preselected clinically based variables (index diagnosis, age, sex, age by sex, and race/ethnicity) showed good prognostic accuracy in the derivation data set. The risk calculator was externally validated, confirming good performance and potential clinical usefulness for the transdiagnostic prediction of psychosis in secondary mental health care.

This study has 3 significant clinical implications. First, we confirmed substantial psychosis risk enrichment in individuals accessing secondary mental health care. The 6-year risk of psychosis was 5-fold higher than in the local general population (3.02/0.62 = 4.87) and in individuals accessing primary care,37 highlighting a clear window of opportunity for the transdiagnostic prevention of psychosis within secondary mental health care.38

Second, we have shown that the ARMS designation, in particular its attenuated psychotic symptoms subgroup (footnotes to Table 2), is necessary to predict psychosis in individuals who have never experienced psychotic (eg, brief limited intermittent psychotic symptoms39) symptoms. The ICD-10 categories of substance use disorders, nonbipolar mood disorders, anxiety disorders, personality disorders, developmental disorders, childhood/adolescence onset disorders, physiological syndromes, and mental retardation showed a lower level of psychosis risk. Accordingly, the use of ICD-10 categories of comorbid mental disorders, such as anxiety or depression, is unlikely to be of any clinical usefulness to predict psychosis. The ICD-10 acute and transient psychotic disorders and the brief limited intermittent psychotic symptoms subgroup of the ARMS were both associated with a very high risk of developing psychosis, in particular schizophrenia spectrum psychoses, but only in individuals with remitting symptoms at the time of the index diagnosis.40,41 Similarly, bipolar mood disorders specifically predicted the onset of affective spectrum psychoses.

Third, we have demonstrated that the ARMS designation, although necessary, is not sufficient to intercept the overall burden of psychosis risk in secondary mental health care (eDiscussion 1 in the Supplement). In fact, although Outreach and Support in South London was established in Lambeth and Southwark several years before the start of the current cohort,42 only 314 of 33 820 individuals (0.93%) were receiving Outreach and Support in South London care, accounting for only 5.19% of the total cases of emerging psychosis across the 2 boroughs. More importantly, none of the patients outside of Outreach and Support in South London care had ever been assessed for an ARMS. This seems like a missed clinical opportunity because screening for an ARMS is specifically indicated for individuals “already distressed by mental problems”43 and accessing secondary mental health care,44 to prevent psychosis with indicated interventions,13 and improve outcomes in those who go on to develop psychosis (by reducing the duration of untreated psychosis, admission to hospital, and compulsory treatments45 or unnecessary treatment).46

Building on the aforementioned points, our findings highlight a significant unmet need for transdiagnostic prevention of psychosis in secondary mental health care, which is not addressed by existing ICD-10 categories (that are not specific enough for psychosis prediction) or the ARMS designation (which does not include most individuals at risk for psychosis). To overcome these limitations, this study developed a practical, individualized risk calculator tool for the transdiagnostic prediction of psychosis in secondary mental health care. A well-performing risk calculator was developed from easily collectable clinical and demographic predictor variables (age, sex, age by sex, race/ethnicity, and index ICD-10/ARMS diagnosis). The overall externally validated model was robust and achieved good performance, which is in the range of values for established calculators in use for cancer and cardiovascular, neurological, and endocrine diseases (Table 3 in Fusar-Poli et al5). The risk calculator was implemented online and designed to generate a number representing the probability of transition to psychosis, given a particular profile of input variables. A key advantage of the risk calculator is that it inherently accommodates heterogeneity in profiles of risk factors among high-risk individuals.47,48 At the same time, the risk calculator assumes that individuals have accessed secondary mental health care and that the predictor variables are coded as indicated in our methods (eg, ICD-10 categories for the index diagnosis). Therefore, the risk prediction tool would not be usable in primary care or the general population, or if other diagnostic criteria have been used (eg, DSM).

Quiz Ref IDThis tool is therefore most useful to clinicians using the calculator for patients who have accessed secondary mental health services. The online calculator could also be easily integrated into electronic case registers, such as the Clinical Record Interactive Search, to facilitate the automatic and individualized prediction of psychosis. Critically, risk determinations should be communicated to patients by clinicians who can help patients understand the meaning of the risk estimates and provide commensurate treatment recommendations. The decision curve analysis presented in our study can further help clinicians to tailor individualized focused interventions, such as selecting patients to be referred to ARMS services. Focused interventions may include a detailed clinical assessment for psychosis risk (ie, the ARMS assessment) combined with sequential testing,49,50 close-in clinical monitoring for the emergence of psychosis, and psychological treatments recommended to prevent psychosis.

Future studies are needed to refine the focused interventions targeting the high-risk individuals detected by our risk calculator. It is also possible that not all high-risk individuals, even when properly referred and assessed, would eventually meet ARMS criteria. For example, research in high-risk individuals with an index diagnosis of bipolar disorders may help to refine the ARMS construct and its ability to predict the onset of affective psychoses.51 Similarly, the effectiveness of preventive psychological treatments in individuals deemed at risk by our calculator but not meeting ARMS criteria should be further investigated.

Limitations

Limitations of this study are addressed in eDiscussion 2 of the Supplement and primarily involve the use of a clinical case register database that shows high ecological validity but lack of formal validation with research-based criteria. Furthermore, the transportability of our model to different clinical scenarios should be confirmed and refined by external replication studies.

Conclusions

Individuals accessing secondary mental health services are at enhanced risk of developing psychosis compared with the local general population. The use of this novel individualized risk calculator can be of clinical usefulness to improve the transdiagnostic detection of at-risk individuals and prevention of psychosis in secondary mental health care.

Back to top
Article Information

Corresponding Author: Paolo Fusar-Poli, MD, PhD, Department of Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, 5th Floor, PO63, 16 De Crespigny Park, London SE5 8AF, England (paolo.fusar-poli@kcl.ac.uk).

Accepted for Publication: February 8, 2017.

Published Online: March 29, 2017. doi:10.1001/jamapsychiatry.2017.0284

Correction: This article was corrected on May 9, 2018, to fix errors in Table 2.

Author Contributions: Drs Fusar-Poli and Rutigliano had full access to all of the data in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Fusar-Poli, Rutigliano, McGuire.

Acquisition, analysis, or interpretation of data: Fusar-Poli, Rutigliano, Stahl, Davies, Bonoldi, Reilly.

Drafting of the manuscript: Fusar-Poli.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Fusar-Poli, Stahl, Davies.

Obtained funding: Fusar-Poli, McGuire.

Administrative, technical, or material support: Fusar-Poli, Rutigliano, Davies, Reilly, McGuire.

Supervision: Fusar-Poli, Stahl, Bonoldi, McGuire.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was supported in part by a 2014 NARSAD Young Investigator Award to Dr Fusar-Poli and in part by the National Institute for Health Research Biomedical Research Centre at South London and Maudsley National Health Services Foundation Trust and King’s College London.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The views expressed are those of the authors and not necessarily those of the National Health Service, the National Institute for Health Research, or the Department of Health.

References
1.
Millan  MJ, Andrieux  A, Bartzokis  G,  et al.  Altering the course of schizophrenia: progress and perspectives.  Nat Rev Drug Discov. 2016;15(7):485-515.PubMedGoogle ScholarCrossref
2.
Jääskeläinen  E, Juola  P, Hirvonen  N,  et al.  A systematic review and meta-analysis of recovery in schizophrenia.  Schizophr Bull. 2013;39(6):1296-1306.PubMedGoogle ScholarCrossref
3.
Fusar Poli  P, McGorry  P, Kane  J.  Improving outcomes of first episode psychosis.  World Psychiatry. in press.Google Scholar
4.
Yung  AR, McGorry  PD, McFarlane  CA, Jackson  HJ, Patton  GC, Rakkar  A.  Monitoring and care of young people at incipient risk of psychosis.  Schizophr Bull. 1996;22(2):283-303.PubMedGoogle ScholarCrossref
5.
Fusar-Poli  P, Cappucciati  M, Rutigliano  G,  et al.  At risk or not at risk? a meta-analysis of the prognostic accuracy of psychometric interviews for psychosis prediction.  World Psychiatry. 2015;14(3):322-332.PubMedGoogle ScholarCrossref
6.
Braham  A, Bannour  AS, Ben Romdhane  A,  et al.  Validation of the Arabic version of the Comprehensive Assessment of At Risk Mental States (CAARMS) in Tunisian adolescents and young adults.  Early Interv Psychiatry. 2014;8(2):147-154.PubMedGoogle ScholarCrossref
7.
Krebs  MO, Magaud  E, Willard  D,  et al.  Évaluation des états mentaux à risque de transition psychotique: validation de la version française de la CAARMS [Assessment of mental states at risk of psychotic transition: validation of the French version of the CAARMS].  Encephale. 2014;40(6):447-456.PubMedGoogle ScholarCrossref
8.
Fusar-Poli  P, Schultze-Lutter  F.  Predicting the onset of psychosis in patients at clinical high risk: practical guide to probabilistic prognostic reasoning.  Evid Based Ment Health. 2016;19(1):10-15.PubMedGoogle ScholarCrossref
9.
Fusar-Poli  P, Bechdolf  A, Taylor  MJ,  et al.  At risk for schizophrenic or affective psychoses? a meta-analysis of DSM/ICD diagnostic outcomes in individuals at high clinical risk.  Schizophr Bull. 2013;39(4):923-932.PubMedGoogle ScholarCrossref
10.
Webb  JR, Addington  J, Perkins  DO,  et al.  Specificity of incident diagnostic outcomes in patients at clinical high risk for psychosis.  Schizophr Bull. 2015;41(5):1066-1075.PubMedGoogle ScholarCrossref
11.
Fusar-Poli  P, Rutigliano  G, Stahl  D,  et al.  Long-term validity of the At Risk Mental State (ARMS) for predicting psychotic and non-psychotic mental disorders.  Eur Psychiatry. 2016;42:49-54.PubMedGoogle ScholarCrossref
12.
Kempton  MJ, Bonoldi  I, Valmaggia  L, McGuire  P, Fusar-Poli  P.  Speed of psychosis progression in people at ultra-high clinical risk: a complementary meta-analysis.  JAMA Psychiatry. 2015;72(6):622-623.PubMedGoogle ScholarCrossref
13.
Stafford  MR, Jackson  H, Mayo-Wilson  E, Morrison  AP, Kendall  T.  Early interventions to prevent psychosis: systematic review and meta-analysis.  BMJ. 2013;346:f185.PubMedGoogle ScholarCrossref
14.
Deas  G, Kelly  C, Hadjinicolaou  AV, Holt  C, Agius  M, Zaman  R.  An update on: meta-analysis of medical and non-medical treatments of the prodromal phase of psychotic illness in at risk mental states.  Psychiatr Danub. 2016;28(suppl 1):31-38.PubMedGoogle Scholar
15.
NICE. Psychosis and schizophrenia in children and young people: recognition and management. http://www.nice.org.uk/Guidance/CG155. Published 2013. Accessed January 17, 2016.
16.
Fusar-Poli  P, Nelson  B, Valmaggia  L, Yung  AR, McGuire  PK.  Comorbid depressive and anxiety disorders in 509 individuals with an at-risk mental state: impact on psychopathology and transition to psychosis.  Schizophr Bull. 2014;40(1):120-131.PubMedGoogle ScholarCrossref
17.
Conrad  AM, Lewin  TJ, Sly  KA,  et al.  Utility of risk-status for predicting psychosis and related outcomes: evaluation of a 10-year cohort of presenters to a specialised early psychosis community mental health service.  Psychiatry Res. 2017;247:336-344.PubMedGoogle ScholarCrossref
18.
van Os  J, Murray  RM.  Can we identify and treat “schizophrenia light” to prevent true psychotic illness?  BMJ. 2013;346:f304.PubMedGoogle ScholarCrossref
19.
Stewart  R, Soremekun  M, Perera  G,  et al.  The South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLAM BRC) case register: development and descriptive data.  BMC Psychiatry. 2009;9:51.PubMedGoogle ScholarCrossref
20.
Studerus  E, Ramyead  A, Riecher-Rössler  A.  Prediction of transition to psychosis in patients with a clinical high risk for psychosis: a systematic review of methodology and reporting.  Psychol Med. 2017;1-16.PubMedGoogle Scholar
21.
Benchimol  EI, Smeeth  L, Guttmann  A,  et al; RECORD Working Committee.  The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement.  PLoS Med. 2015;12(10):e1001885.PubMedGoogle ScholarCrossref
22.
Kaplan  E, Meier  P.  Nonparametric estimation from incomplete observations.  J Am Stat Assoc. 1958;53(282):457-481.Google ScholarCrossref
23.
Greenwood  M.  The Natural Duration of Cancer. London, England: Her Majesty's Stationery Office; 1926.
24.
Kirkbride  JB, Jackson  D, Perez  J,  et al.  A population-level prediction tool for the incidence of first-episode psychosis: translational epidemiology based on cross-sectional data.  BMJ Open. 2013;3(2):e001998.PubMedGoogle ScholarCrossref
25.
Royston  P, Altman  DG.  External validation of a Cox prognostic model: principles and methods.  BMC Med Res Methodol. 2013;13:33.PubMedGoogle ScholarCrossref
26.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138.PubMedGoogle ScholarCrossref
27.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement.  Ann Intern Med. 2015;162(1):55-63.PubMedGoogle ScholarCrossref
28.
Grambsch  P, Therneau  T.  Proportional hazards tests and diagnostics based on weighted residuals.  Biometrika. 1994;81:(3)515-526. doi:10.1093/biomet/81.3.515Google ScholarCrossref
29.
Perera  G, Broadbent  M, Callard  F,  et al.  Cohort profile of the South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLaM BRC) Case Register: current status and recent enhancement of an Electronic Mental Health Record-derived data resource.  BMJ Open. 2016;6(3):e008721.PubMedGoogle ScholarCrossref
30.
Machin  D, Cheung  Y, Parmar  M.  Survival Analysis: A Practical Approach. 2nd ed. Chichester, England: John Wiley and Sons Ltd; 2006.Crossref
31.
Royston  P.  Explained variation for survival models.  Stata J. 2006;6(1):83-96.Google Scholar
32.
Crowson  CS, Atkinson  EJ, Therneau  TM.  Assessing calibration of prognostic risk scores.  Stat Methods Med Res. 2016;25(4):1692-1706.PubMedGoogle ScholarCrossref
33.
Harrell  F. Regression modeling strategies. http://biostat.mc.vanderbilt.edu/rms. Published 2017. Accessed January 13, 2017.
34.
Collins  GS, Ogundimu  EO, Altman  DG.  Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.  Stat Med. 2016;35(2):214-226.PubMedGoogle ScholarCrossref
35.
Vickers  AJ, Van Calster  B, Steyerberg  EW.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.  BMJ. 2016;352:i6.PubMedGoogle ScholarCrossref
36.
Vickers  AJ, Cronin  AM, Elkin  EB, Gonen  M.  Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers.  BMC Med Inform Decis Mak. 2008;8:53.PubMedGoogle ScholarCrossref
37.
Hardoon  S, Hayes  JF, Blackburn  R,  et al.  Recording of severe mental illness in United Kingdom primary care, 2000-2010.  PLoS One. 2013;8(12):e82365.PubMedGoogle ScholarCrossref
38.
Francesconi  M, Minichino  A, Carrión  RE,  et al.  Psychosis prediction in secondary mental health services: a broad, comprehensive approach to the “at risk mental state” syndrome.  Eur Psychiatry. 2016;40:96-104.PubMedGoogle ScholarCrossref
39.
Fusar-Poli  P, Cappucciati  M, De Micheli  A,  et al.  Diagnostic and Prognostic Significance of Brief Limited Intermittent Psychotic Symptoms (BLIPS) in individuals at ultra high risk.  Schizophr Bull. 2017;43(1):48-56.PubMedGoogle ScholarCrossref
40.
Fusar-Poli  P, Cappucciati  M, Bonoldi  I,  et al.  Prognosis of brief psychotic episodes: a meta-analysis.  JAMA Psychiatry. 2016;73(3):211-220.PubMedGoogle ScholarCrossref
41.
Fusar-Poli  P, Cappucciati  M, Borgwardt  S,  et al.  Heterogeneity of psychosis risk within individuals at clinical high risk: a meta-analytical stratification.  JAMA Psychiatry. 2016;73(2):113-120.PubMedGoogle ScholarCrossref
42.
Fusar-Poli  P, Byrne  M, Badger  S, Valmaggia  LR, McGuire  PK.  Outreach and support in south London (OASIS), 2001-2011: ten years of early diagnosis and treatment for young individuals at high clinical risk for psychosis.  Eur Psychiatry. 2013;28(5):315-326.PubMedGoogle ScholarCrossref
43.
Schultze-Lutter  F, Michel  C, Schmidt  SJ,  et al.  EPA guidance on the early detection of clinical high risk states of psychoses.  Eur Psychiatry. 2015;30(3):405-416.PubMedGoogle ScholarCrossref
44.
Fusar-Poli  P, Rutigliano  G, Stahl  D,  et al.  Deconstructing pretest risk enrichment to optimize prediction of psychosis in individuals at clinical high risk.  JAMA Psychiatry. 2016;73(12):1260-1267.PubMedGoogle ScholarCrossref
45.
Valmaggia  LR, Byrne  M, Day  F,  et al.  Duration of untreated psychosis and need for admission in patients who engage with mental health services in the prodromal phase.  Br J Psychiatry. 2015;207(2):130-134.PubMedGoogle ScholarCrossref
46.
Woodberry  KA, Seidman  LJ, Bryant  C,  et al.  Treatment precedes positive symptoms in north american adolescent and young adult clinical high risk cohort.  J Clin Child Adolesc Psychol. 2016;1-10.PubMedGoogle Scholar
47.
Carrión  RE, Correll  CU, Auther  AM, Cornblatt  BA.  A severity-based clinical staging model for the psychosis prodrome: longitudinal findings from the new york recognition and prevention program.  Schizophr Bull. 2017;43(1):64-74.PubMedGoogle ScholarCrossref
48.
Addington  J, Liu  L, Perkins  DO, Carrion  RE, Keefe  RS, Woods  SW.  The role of cognition and social functioning as predictors in the transition to psychosis for youth with attenuated psychotic symptoms.  Schizophr Bull. 2017;43(1):57-63.PubMedGoogle ScholarCrossref
49.
Schmidt  A, Cappucciati  M, Radua  J,  et al.  Improving prognostic accuracy in subjects at clinical high risk for psychosis: systematic review of predictive models and meta-analytical sequential testing simulation.  Schizophr Bull. 2016;sbw098.PubMedGoogle Scholar
50.
Clark  SR, Baune  BT, Schubert  KO,  et al.  Prediction of transition from ultra-high risk to first-episode psychosis using a probabilistic model combining history, clinical assessment and fatty-acid biomarkers.  Transl Psychiatry. 2016;6(9):e897.PubMedGoogle ScholarCrossref
51.
Scott  J, Marwaha  S, Ratheesh  A,  et al.  Bipolar at-risk criteria: an examination of which clinical features have optimal utility for identifying youth at risk of early transition from depression to bipolar disorders.  Schizophr Bull. 2016;sbw154.PubMedGoogle Scholar
×