Multimodal Machine Learning Workflows for Prediction of Psychosis in Patients With Clinical High-Risk Syndromes and Recent-Onset Depression | Depressive Disorders | JAMA Psychiatry | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Predictive Signatures Underlying the Clinical-Neurocognitive Models
Predictive Signatures Underlying the Clinical-Neurocognitive Models

The reliability of predictive pattern elements was evaluated using cross-validation ratio mapping (A). In addition, the significance of predictive features used by the clinical-neurocognitive model was assessed by means of sign-based consistency mapping (B). Both visualization methods are detailed in the eMethods in the Supplement. CTQ indicates Childhood Trauma Questionnaire; DANVA, Diagnostic Analysis of Non-Verbal Accuracy; DSST, Digit-Symbol Substitution Test; FDR, false discovery rate; PVF, Phonetic Verbal Fluency; ROCF, Rey-Osterreith Figure; SIPS, Structured Interview for Psychosis-Risk Syndromes; SOPT, Self-Ordered Pointing Task; SVF, Semantic Verbal Fluency; TMT, Trail Making Test; and WAIS, Wechsler Adult Intelligence Scale.

Figure 2.  Predictive Signatures Underlying the Polygenic Risk Score (PRS)–Based and Cybernetic Risk Calculator Models
Predictive Signatures Underlying the Polygenic Risk Score (PRS)–Based and Cybernetic Risk Calculator Models

The reliability of predictive pattern elements was evaluated using cross-validation ratio (CVR) mapping (A). In addition, the significance of predictive features used by the PRS-based model was assessed by means of sign-based consistency mapping (B) (as described in the eMethods in the Supplement). The cybernetic model combines all algorithmic and human components (C). FDR indicates false discovery rate; and sMRI, structural magnetic resonance imaging.

Figure 3.  Statistical Comparison of Prognostic Models
Statistical Comparison of Prognostic Models

Cohorts include patients with follow-up of 18 months or longer (PRONIA plus 18M), the complete PRONIA cohort, and the Zurich Early Recognition Program (ZInEP). Data points indicate median. The Quade test51 was used to compare the models’ median balanced accuracy (BAC) computed across the cross-validation cycle (CV2) test data partitions. The BAC measures obtained for the ZInEP cohort (C) were produced by applying the condensed clinical-neurocognitive (Clin-NC), structural magnetic resonance imaging (sMRI)–based, and respective stacked risk calculators of the complete PRONIA sample (B) to this external sample (eFigure 2 in the Supplement). Post hoc comparisons were performed using the t distribution approximation described by Heckert and Filliben.52 P values were corrected for multiple comparisons using the false discovery rate (FDR). The upper graphs represent the median BAC for each risk calculator in analyses A, B, and C along with the lower and upper quartiles of the BAC distributions (whiskers of the error bars). The lower figures show the logarithmized, FDR-corrected P matrix for the pairwise post hoc classifier comparisons. For an in-depth analysis of the prognostic sequence included in the risk classifier comparison, see eFigures 14 and 15 in the Supplement. The cybernetic risk calculator analyzed the combined predictions of raters, Clin-NC, polygenic risk score (PRS)–based, and sMRI-based risk calculators; the stacked risk calculator, the combined predictions of Clin-NC, PRS-based, and sMRI-based risk calculators.

aIndicates risk calculator encompassing the condensed Clin-NCs and sMRI-based models and specifically trained to externally validate the effect of stacking on prognostic performance in the ZInEP cohort.

Table 1.  Study-Related, Sociodemographic, Physical, Functional, and Clinical Differences in Patients With vs Without Transition to Psychosis During the Follow-up Period
Study-Related, Sociodemographic, Physical, Functional, and Clinical Differences in Patients With vs Without Transition to Psychosis During the Follow-up Period
Table 2.  Prediction Performance of Clinical Raters; Unimodal, Stacked, and Cybernetic Risk Calculators; and Prognostic Workflowsa
Prediction Performance of Clinical Raters; Unimodal, Stacked, and Cybernetic Risk Calculators; and Prognostic Workflowsa
Supplement.

eMethods. Participants and Analysis

eTable 1. Study Inclusion/Exclusion Criteria of the Study

eTable 2. Antipsychotic Medication Thresholds Based on the Previous Version of the S3 Guidelines of the German Association for Psychiatry, Psychotherapy and Psychosomatics

eTable 3. Group-Level Comparison of CHR, ROD and HC Individuals

eTable 4. Diagnostic Breakdown of Psychotic Disorders in PRONIA Cases With a Disease Transition

eTable 5. Sociodemographic, Clinical and Neurocognitive Variables Used in the Clinical Prediction Models

eTable 6. MR Scanner Systems and Structural MRI Sequence Parameters Used at the Respective PRONIA Sites

eTable 7. Effects of Baseline Treatments on the Decision Scores Generated by 5 Different Risk Calculators

eTable 8. Correlations Between Decision Scores of Unimodal and Cybernetic Risk Calculators and Patients’ Maximum Follow-up Intervals and Number of Follow-up Examinations

eTable 9. Analysis of the Site-Related Variation in Follow-up Duration, Time to Transition, Age and Sex Distributions in the PRONIA Cohort

eTable 10. Effects of the Factor ‘Site’ on Predictive Performance of Raters, Unimodal, Stacked, and Cybernetic Risk Calculators

eTable 11. Heterogeneity of Raters’ and Models’ Predictive Performance

eTable 12. Differential Diagnostic Performance of Classification Models Trained to Separate Between CHR and ROD Patients Using Clinical-Neurocognitive, PRS-Based, and sMRI-Based Data Domain

eTable 13. Explained Variances of Pairwise Prognostic, Diagnostic, and Prognostic-Diagnostic Classifier Combinations

eTable 14. ROD Depletion and Substitution Analyses Assessing the Performance Effects Induced by the ROD Group in the Prediction of Psychosis Transitions in the CHR Sample

eTable 15. Discriminative Performance of Raters and Risk Calculators in Distinguishing Between Transition Cases, Cases With Nonremitting/De novo CHR States and Cases Developing Asymptomatic CHR Trajectories

eTable 16. Comparison of Nonpsychotic Diagnostic Outcomes Between Patients With a Predicted Transition vs Predicted Nontransition to Psychosis During the Follow-up Period of the Study

eTable 17. Prognostic Sequences Tested in the Sequence Optimization Algorithm

eTable 18. Study-Related, Sociodemographic, Physical, Functional, and Clinical Differences in the ZInEP Cohort

eTable 19. Study-Related, Sociodemographic, Physical, Functional, and Clinical Differences in the BEARS-Kid Cohort

eTable 20. Performance Gains Produced by the Stacking of the Clinical-Neurocognitive and PRS-Based Models

eFigure 1. CONSORT Chart and Follow-up Protocol of the PRONIA Study for the Clinical Participants

eFigure 2. Schematic Analysis Workflow of the Study

eFigure 3. Experimental Design of the Machine Learning Pipelines Used to Train and Cross-validate the Unimodal and Stacked Risk Calculators

eFigure 4. Schematic Representation of the NeuroMiner Model Optimization Process Used to Train the Structural MRI Predictors

eFigure 5. Predictive Signature Underlying the sMRI-Based Risk Calculator

eFigure 6. Comparison of Predictive Performance of Expert-Based, Unimodal, Stacked, Cybernetic and Sequentially Stacked Risk Calculators Trained and Cross-validated Using the PRONIA-18M and Complete PRONIA Cohorts

eFigure 7. Comparison of Standardized Clinical-Neurocognitive Variables Between Healthy Volunteers and CHR/ROD Patients Who Were Labeled With a Transition or Nontransition to Psychosis by the Clinical-Neurocognitive Risk Calculator

eFigure 8. Comparison of Polygenic Risk Scores (PRS) Between Healthy Volunteers and CHR/ROD Patients Who Were Labeled With a Transition or Nontransition to Psychosis by the PRS-Based Risk Calculator

eFigure 9. Univariate Volumetric Comparisons Between Healthy Volunteers and Patients Labeled With a Transition or Nontransition to Psychosis by the sMRI-Based Risk Calculator

eFigure 10. Image Quality Assessment of T1-Weighted Images Analyzed in the Study Using the Quality Assessment Tools of the CAT12 Toolbox

eFigure 11. Interaction Analysis Assessing the Effects of the Number of Examinations Available and the Longest Interval Duration on the Predictions of Four Different Risk Calculators

eFigure 12. Statistical Analysis of Prognostic Performance Effects Induced by the ROD Depletion and Substitution Strategies in the Three Unimodal Risk Calculators and the Stacked Model

eFigure 13. Prognostic Stratification Effects of Raters’ Predictions, Unimodal and Cybernetic Risk Calculators on the Trajectories of CHR Syndromes and Functioning as Determined by Nonnegative Matrix Factorization and Linear Mixed-Effects Modeling

eFigure 14. Detailed Analysis of the Optimal Sequential Prediction Algorithm

eFigure 15. Regularization of the Prognostic Workflow for DIFFERENT LEVELS of Examination Sparsity and Analysis of Effects of Regularization on Prognostic Performance

eFigure 16. Comparison of Clinical-Neurocognitive Signatures Used by the Prognostic Risk Calculator and the Differential Diagnostic Classifier (CHR vs ROD)

eReferences.

1.
McGorry  PD, Mei  C.  Ultra-high-risk paradigm: lessons learnt and new directions.   Evid Based Ment Health. 2018;21(4):131-133. doi:10.1136/ebmental-2018-300061 PubMedGoogle ScholarCrossref
2.
Fusar-Poli  P, Salazar de Pablo  G, Correll  CU,  et al.  Prevention of psychosis: advances in detection, prognosis, and intervention.   JAMA Psychiatry. 2020;77(7):755-765. doi:10.1001/jamapsychiatry.2019.4779 PubMedGoogle ScholarCrossref
3.
Fusar-Poli  P, Rutigliano  G, Stahl  D,  et al.  Development and validation of a clinically based risk calculator for the transdiagnostic prediction of psychosis.   JAMA Psychiatry. 2017;74(5):493-500. doi:10.1001/jamapsychiatry.2017.0284 PubMedGoogle ScholarCrossref
4.
Schultze-Lutter  F, Michel  C, Schmidt  SJ,  et al. EPA guidance on the early detection of clinical high risk states of psychoses.  Eur Psychiatry. 2015;30(3):405-416. doi:10.1016/j.eurpsy.2015.01.010
5.
Carrión  RE, McLaughlin  D, Goldberg  TE,  et al.  Prediction of functional outcome in individuals at clinical high risk for psychosis.   JAMA Psychiatry. 2013;70(11):1133-1142. doi:10.1001/jamapsychiatry.2013.1909 PubMedGoogle ScholarCrossref
6.
Cannon  TD, Yu  C, Addington  J,  et al.  An individualized risk calculator for research in prodromal psychosis.   Am J Psychiatry. 2016;173(10):980-988. doi:10.1176/appi.ajp.2016.15070890 PubMedGoogle ScholarCrossref
7.
Zhang  T, Xu  L, Tang  Y,  et al; SHARP (ShangHai At Risk for Psychosis) Study Group.  Prediction of psychosis in prodrome: development and validation of a simple, personalized risk calculator.   Psychol Med. 2019;49(12):1990-1998. doi:10.1017/S0033291718002738PubMedGoogle ScholarCrossref
8.
Koutsouleris  N, Meisenzahl  EM, Davatzikos  C,  et al.  Use of neuroanatomical pattern classification to identify subjects in at-risk mental states of psychosis and predict disease transition.   Arch Gen Psychiatry. 2009;66(7):700-712. doi:10.1001/archgenpsychiatry.2009.62 PubMedGoogle ScholarCrossref
9.
Koutsouleris  N, Riecher-Rössler  A, Meisenzahl  EM,  et al.  Detecting the psychosis prodrome across high-risk populations using neuroanatomical biomarkers.   Schizophr Bull. 2015;41(2):471-482. doi:10.1093/schbul/sbu078 PubMedGoogle ScholarCrossref
10.
Das  T, Borgwardt  S, Hauke  DJ,  et al.  Disorganized gyrification network properties during the transition to psychosis.   JAMA Psychiatry. 2018;75(6):613-622. doi:10.1001/jamapsychiatry.2018.0391 PubMedGoogle ScholarCrossref
11.
Mechelli  A, Lin  A, Wood  S,  et al.  Using clinical information to make individualized prognostic predictions in people at ultra high risk for psychosis.   Schizophr Res. 2017;184:32-38. doi:10.1016/j.schres.2016.11.047 PubMedGoogle ScholarCrossref
12.
Clark  SR, Baune  BT, Schubert  KO,  et al.  Prediction of transition from ultra-high risk to first-episode psychosis using a probabilistic model combining history, clinical assessment and fatty-acid biomarkers.   Transl Psychiatry. 2016;6(9):e897. doi:10.1038/tp.2016.170 PubMedGoogle Scholar
13.
Koutsouleris  N, Kambeitz-Ilankovic  L, Ruhrmann  S,  et al; PRONIA Consortium.  Prediction models of functional outcomes for individuals in the clinical high-risk state for psychosis or with recent-onset depression: a multimodal, multisite machine learning analysis.   JAMA Psychiatry. 2018;75(11):1156-1172. doi:10.1001/jamapsychiatry.2018.2165 PubMedGoogle ScholarCrossref
14.
Fusar-Poli  P, Werbeloff  N, Rutigliano  G,  et al.  Transdiagnostic risk calculator for the automatic detection of individuals at risk and the prediction of psychosis: second replication in an independent national health service trust.   Schizophr Bull. 2019;45(3):562-570. doi:10.1093/schbul/sby070PubMedGoogle ScholarCrossref
15.
Carrión  RE, Cornblatt  BA, Burton  CZ,  et al.  Personalized prediction of psychosis: external validation of the NAPLS-2 psychosis risk calculator with the EDIPPP project.   Am J Psychiatry. 2016;173(10):989-996. doi:10.1176/appi.ajp.2016.15121565 PubMedGoogle ScholarCrossref
16.
Perkins  DO, Olde Loohuis  L, Barbee  J,  et al.  Polygenic risk score contribution to psychosis prediction in a target population of persons at clinical high risk.   Am J Psychiatry. 2020;177(2):155-163. doi:10.1176/appi.ajp.2019.18060721 PubMedGoogle ScholarCrossref
17.
Worthington  MA, Cao  H, Cannon  TD.  Discovery and validation of prediction algorithms for psychosis in youth at clinical high risk.   Biol Psychiatry Cogn Neurosci Neuroimaging. 2020;5(8):738-747. doi:10.1016/j.bpsc.2019.10.006PubMedGoogle Scholar
18.
Studerus  E, Ramyead  A, Riecher-Rössler  A.  Prediction of transition to psychosis in patients with a clinical high risk for psychosis: a systematic review of methodology and reporting.   Psychol Med. 2017;47(7):1163-1178. doi:10.1017/S0033291716003494 PubMedGoogle ScholarCrossref
19.
Sanfelici  R, Dwyer  DB, Antonucci  LA, Koutsouleris  N.  Individualized diagnostic and prognostic models for patients with psychosis risk syndromes: a meta-analytic view on the state-of-the-art.   Biol Psychiatry. 2020;88(4):349-360. doi:10.1016/j.biopsych.2020.02.009 PubMedGoogle ScholarCrossref
20.
Yarkoni  T, Westfall  J.  Choosing prediction over explanation in psychology: lessons from machine learning.   Perspect Psychol Sci. 2017;12(6):1100-1122. doi:10.1177/1745691617693393 PubMedGoogle ScholarCrossref
21.
Steyerberg  EW, Harrell  FE  Jr.  Prediction models need appropriate internal, internal-external, and external validation.   J Clin Epidemiol. 2016;69:245-247. doi:10.1016/j.jclinepi.2015.04.005 PubMedGoogle ScholarCrossref
22.
Fusar-Poli  P, Hijazi  Z, Stahl  D, Steyerberg  EW.  The science of prognosis in psychiatry: a review.   JAMA Psychiatry. 2018;75(12):1289-1297. doi:10.1001/jamapsychiatry.2018.2530 PubMedGoogle ScholarCrossref
23.
Lee  TY, Lee  J, Kim  M, Choe  E, Kwon  JS.  Can we predict psychosis outside the clinical high-risk state? a systematic review of non-psychotic risk syndromes for mental disorders.   Schizophr Bull. 2018;44(2):276-285. doi:10.1093/schbul/sbx173 PubMedGoogle ScholarCrossref
24.
McGorry  PD, Hartmann  JA, Spooner  R, Nelson  B.  Beyond the “at risk mental state” concept: transitioning to transdiagnostic psychiatry.   World Psychiatry. 2018;17(2):133-142. doi:10.1002/wps.20514 PubMedGoogle ScholarCrossref
25.
Fusar-Poli  P, Nelson  B, Valmaggia  L, Yung  AR, McGuire  PK.  Comorbid depressive and anxiety disorders in 509 individuals with an at-risk mental state: impact on psychopathology and transition to psychosis.   Schizophr Bull. 2014;40(1):120-131. doi:10.1093/schbul/sbs136 PubMedGoogle ScholarCrossref
26.
Koutsouleris  N, Meisenzahl  EM, Borgwardt  S,  et al.  Individualized differential diagnosis of schizophrenia and mood disorders using neuroanatomical biomarkers.   Brain. 2015;138(pt 7):2059-2073. doi:10.1093/brain/awv111 PubMedGoogle ScholarCrossref
27.
Häfner  H, Maurer  K, Trendler  G, an der Heiden  W, Schmidt  M, Könnecke  R.  Schizophrenia and depression: challenging the paradigm of two separate diseases—a controlled study of schizophrenia, depression and healthy controls.   Schizophr Res. 2005;77(1):11-24. doi:10.1016/j.schres.2005.01.004 PubMedGoogle ScholarCrossref
28.
Wigman  JTW, van Nierop  M, Vollebergh  WA,  et al.  Evidence that psychotic symptoms are prevalent in disorders of anxiety and depression, impacting on illness onset, risk, and severity—implications for diagnosis and ultra-high risk research.   Schizophr Bull. 2012;38(2):247-257. doi:10.1093/schbul/sbr196 PubMedGoogle ScholarCrossref
29.
Siebald  C, Khandaker  GM, Zammit  S, Lewis  G, Jones  PB.  Association between childhood psychiatric disorders and psychotic experiences in adolescence: a population-based longitudinal study.   Compr Psychiatry. 2016;69:45-52. doi:10.1016/j.comppsych.2016.05.004 PubMedGoogle ScholarCrossref
30.
Musliner  KL, Mortensen  PB, McGrath  JJ,  et al; Bipolar Disorder Working Group of the Psychiatric Genomics Consortium.  Association of polygenic liabilities for major depression, bipolar disorder, and schizophrenia with risk for depression in the Danish population.   JAMA Psychiatry. 2019;76(5):516-525. doi:10.1001/jamapsychiatry.2018.4166 PubMedGoogle ScholarCrossref
31.
Schmidt  A, Cappucciati  M, Radua  J,  et al.  Improving prognostic accuracy in subjects at clinical high risk for psychosis: systematic review of predictive models and meta-analytical sequential testing simulation.   Schizophr Bull. 2017;43(2):375-388. doi:10.1093/schbul/sbw098 PubMedGoogle Scholar
32.
Rajkomar  A, Dean  J, Kohane  I.  Machine learning in medicine.   N Engl J Med. 2019;380(14):1347-1358. doi:10.1056/NEJMra1814259 PubMedGoogle ScholarCrossref
33.
Wiener  N.  The Human Use of Human Beings. Houghton Mifflin; 1950.
34.
Clark  SR, Schubert  KO, Baune  BT.  Towards indicated prevention of psychosis: using probabilistic assessments of transition risk in psychosis prodrome.   J Neural Transm (Vienna). 2015;122(1):155-169. doi:10.1007/s00702-014-1325-9 PubMedGoogle ScholarCrossref
35.
Riecher-Rössler  A, Gschwandtner  U, Aston  J,  et al.  The Basel early-detection-of-psychosis (FEPSY)-study—design and preliminary results.   Acta Psychiatr Scand. 2007;115(2):114-125. doi:10.1111/j.1600-0447.2006.00854.x PubMedGoogle ScholarCrossref
36.
Theodoridou  A, Heekeren  K, Dvorsky  D,  et al.  Early recognition of high risk of bipolar disorder and psychosis: an overview of the ZInEP “early recognition” study.   Front Public Health. 2014;2:166. doi:10.3389/fpubh.2014.00166 PubMedGoogle ScholarCrossref
37.
Schimmelmann  BG, Michel  C, Martz-Irngartinger  A, Linder  C, Schultze-Lutter  F.  Age matters in the prevalence and clinical significance of ultra-high-risk for psychosis symptoms and criteria in the general population: findings from the BEAR and BEARS-Kid studies.   World Psychiatry. 2015;14(2):189-197. doi:10.1002/wps.20216 PubMedGoogle ScholarCrossref
38.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement.   Br J Surg. 2015;102(3):148-158. doi:10.1002/bjs.9736 PubMedGoogle ScholarCrossref
39.
McGlashan  T, Walsh  B, Woods  S.  The Psychosis-Risk Syndrome: Handbook for Diagnosis and Follow-up. Oxford University Press; 2010.
40.
Yung  AR, Nelson  B, Thompson  A, Wood  SJ.  The psychosis threshold in ultra high risk (prodromal) research: is it valid?   Schizophr Res. 2010;120(1-3):1-6. doi:10.1016/j.schres.2010.03.014 PubMedGoogle ScholarCrossref
41.
Dwyer  DB, Falkai  P, Koutsouleris  N.  Machine learning approaches for clinical psychology and psychiatry.   Annu Rev Clin Psychol. 2018;14:91-118. doi:10.1146/annurev-clinpsy-032816-045037 PubMedGoogle ScholarCrossref
42.
Schultze-Lutter  F, Addington  J, Ruhrmann  S, Klosterkötter  J.  Schizophrenia Proneness Instrument, Adult Version. Giovanni Fioriti Editore; 2007.
43.
Cornblatt  BA, Auther  AM, Niendam  T,  et al.  Preliminary findings for two new measures of social and role functioning in the prodromal phase of schizophrenia.   Schizophr Bull. 2007;33(3):688-702. doi:10.1093/schbul/sbm029 PubMedGoogle ScholarCrossref
44.
Pedersen  G, Hagtvet  KA, Karterud  S.  Generalizability studies of the Global Assessment of Functioning–Split version.   Compr Psychiatry. 2007;48(1):88-94. doi:10.1016/j.comppsych.2006.03.008 PubMedGoogle ScholarCrossref
45.
Bernstein  D, Fink  L.  CTQ: Childhood Trauma Questionnaire: A Retrospective Self-report. Psychological Corp; 1998.
46.
Redman  SL, Corcoran  CM, Kimhy  D, Malaspina  D.  Effects of early trauma on psychosis development in clinical high-risk individuals and stability of trauma assessment across studies: a review.   Arch Psychol (Chic). 2017;1(3):28.PubMedGoogle Scholar
47.
Koutsouleris  N, Davatzikos  C, Bottlender  R,  et al.  Early recognition and disease prediction in the at-risk mental states for psychosis using neurocognitive pattern classification.   Schizophr Bull. 2012;38(6):1200-1215. doi:10.1093/schbul/sbr037 PubMedGoogle ScholarCrossref
48.
Ripke  S, Neale  BM, Corin  A,  et al; Schizophrenia Working Group of the Psychiatric Genomics Consortium.  Biological insights from 108 schizophrenia-associated genetic loci.   Nature. 2014;511(7510):421-427. doi:10.1038/nature13595 PubMedGoogle ScholarCrossref
49.
Karrer  TM, Bassett  DS, Derntl  B,  et al.  Brain-based ranking of cognitive domains to predict schizophrenia.   Hum Brain Mapp. 2019;40(15):4487-4507. doi:10.1002/hbm.24716 PubMedGoogle ScholarCrossref
50.
Gennatas  ED, Friedman  JH, Ungar  LH,  et al.  Expert-augmented machine learning.   Proc Natl Acad Sci U S A. 2020;117(9):4571-4577. doi:10.1073/pnas.1906831117 PubMedGoogle ScholarCrossref
51.
Quade  D.  Using weighted rankings in the analysis of complete blocks with additive block effects.   J Am Stat Assoc. 1979;74(367):680-683. doi:10.1080/01621459.1979.10481670 Google ScholarCrossref
52.
Heckert  NA, Filliben  JJ. Dataplot Reference Manual, Volume 2: LET Subcommands and Library Functions. Published 2003. Accessed November 2, 2020. https://www.itl.nist.gov/div898/handbook/eda/section3/eda3672.htm
53.
Gómez-Verdejo  V, Parrado-Hernández  E, Tohka  J; Alzheimer’s Disease Neuroimaging Initiative.  Sign-consistency based variable importance for machine learning in brain imaging.   Neuroinformatics. 2019;17(4):593-609. doi:10.1007/s12021-019-9415-3 PubMedGoogle ScholarCrossref
54.
Koutsouleris  N, Borgwardt  S, Meisenzahl  EM, Bottlender  R, Möller  HJ, Riecher-Rössler  A.  Disease prediction in the at-risk mental state for psychosis using neuroanatomical biomarkers: results from the FePsy study.   Schizophr Bull. 2012;38(6):1234-1246. doi:10.1093/schbul/sbr145 PubMedGoogle ScholarCrossref
55.
Madras  D, Pitassi  T, Zemel  R.  Predict responsibly: improving fairness and accuracy by learning to defer.  In: Bengio S, Wallach H, Larochelle H, Grauman K, CesaBianchi N, Garnett R, eds.  Advances in Neural Information Processing Systems. Volume 31. Curran Associates Inc; 2018:6147–6157.Google Scholar
56.
Zavos  HMS, Eley  TC, McGuire  P,  et al.  Shared etiology of psychotic experiences and depressive symptoms in adolescence: a longitudinal twin study.   Schizophr Bull. 2016;42(5):1197-1206. doi:10.1093/schbul/sbw021 PubMedGoogle ScholarCrossref
57.
Callaway  DA, Perkins  DO, Woods  SW, Liu  L, Addington  J.  Movement abnormalities predict transitioning to psychosis in individuals at clinical high risk for psychosis.   Schizophr Res. 2014;159(2-3):263-266. doi:10.1016/j.schres.2014.09.031 PubMedGoogle ScholarCrossref
58.
Green  MF, Horan  WP, Lee  J.  Nonsocial and social cognition in schizophrenia: current evidence and future directions.   World Psychiatry. 2019;18(2):146-161. doi:10.1002/wps.20624 PubMedGoogle ScholarCrossref
59.
Thompson  A, Papas  A, Bartholomeusz  C,  et al.  Social cognition in clinical “at risk” for psychosis and first episode psychosis populations.   Schizophr Res. 2012;141(2-3):204-209. doi:10.1016/j.schres.2012.08.007 PubMedGoogle ScholarCrossref
60.
Barbato  M, Liu  L, Cadenhead  KS,  et al.  Theory of mind, emotion recognition and social perception in individuals at clinical high risk for psychosis: findings from the NAPLS-2 cohort.   Schizophr Res Cogn. 2015;2(3):133-139. doi:10.1016/j.scog.2015.04.004 PubMedGoogle ScholarCrossref
61.
Schiffman  J.  Motor issues in the clinical high risk phase of psychosis.   Schizophr Bull. 2017;43(5):937-938. doi:10.1093/schbul/sbx086 PubMedGoogle ScholarCrossref
62.
Bora  E, Berk  M.  Theory of mind in major depressive disorder: a meta-analysis.   J Affect Disord. 2016;191:49-55. doi:10.1016/j.jad.2015.11.023 PubMedGoogle ScholarCrossref
63.
Lindgren  M, Jonninen  M, Jokela  M, Therman  S.  Adolescent psychosis risk symptoms predicting persistent psychiatric service use: a 7-year follow-up study.   Eur Psychiatry. 2019;55:102-108. doi:10.1016/j.eurpsy.2018.10.004 PubMedGoogle ScholarCrossref
64.
Beck  K, Studerus  E, Andreou  C,  et al.  Clinical and functional ultra-long-term outcome of patients with a clinical high risk (CHR) for psychosis.   Eur Psychiatry. 2019;62:30-37. doi:10.1016/j.eurpsy.2019.08.005 PubMedGoogle ScholarCrossref
65.
Woods  SW, Powers  AR  III, Taylor  JH,  et al.  Lack of diagnostic pluripotentiality in patients at clinical high risk for psychosis: specificity of comorbidity persistence and search for pluripotential subgroups.   Schizophr Bull. 2018;44(2):254-263. doi:10.1093/schbul/sbx138 PubMedGoogle ScholarCrossref
66.
Jonas  KG, Lencz  T, Li  K,  et al.  Schizophrenia polygenic risk score and 20-year course of illness in psychotic disorders.   Transl Psychiatry. 2019;9(1):300. doi:10.1038/s41398-019-0612-5 PubMedGoogle ScholarCrossref
67.
Hess  JL, Tylee  DS, Mattheisen  M,  et al; Schizophrenia Working Group of the Psychiatric Genomics Consortium; Lundbeck Foundation Initiative for Integrative Psychiatric Research (iPSYCH).  A polygenic resilience score moderates the genetic risk for schizophrenia.  Published online September 6, 2019.  Mol Psychiatry. doi:10.1038/s41380-019-0463-8 PubMedGoogle Scholar
68.
Takahashi  T, Suzuki  M.  Brain morphologic changes in early stages of psychosis: implications for clinical application and early intervention.   Psychiatry Clin Neurosci. 2018;72(8):556-571. doi:10.1111/pcn.12670 PubMedGoogle ScholarCrossref
69.
Sasabayashi  D, Takayanagi  Y, Takahashi  T,  et al.  Increased occipital gyrification and development of psychotic disorders in individuals with an at-risk mental state: a multicenter study.   Biol Psychiatry. 2017;82(10):737-745. doi:10.1016/j.biopsych.2017.05.018 PubMedGoogle ScholarCrossref
70.
Okada  N, Yahata  N, Koshiyama  D,  et al.  Abnormal asymmetries in subcortical brain volume in early adolescents with subclinical psychotic experiences.   Transl Psychiatry. 2018;8(1):254. doi:10.1038/s41398-018-0312-6 PubMedGoogle ScholarCrossref
71.
de Wit  S, Wierenga  LM, Oranje  B,  et al.  Brain development in adolescents at ultra-high risk for psychosis: longitudinal changes related to resilience.   Neuroimage Clin. 2016;12:542-549. doi:10.1016/j.nicl.2016.08.013 PubMedGoogle ScholarCrossref
72.
Escudero  J, Ifeachor  E, Zajicek  JP, Green  C, Shearer  J, Pearson  S; Alzheimer’s Disease Neuroimaging Initiative.  Machine learning-based method for personalized and cost-effective detection of Alzheimer’s disease.   IEEE Trans Biomed Eng. 2013;60(1):164-168. doi:10.1109/TBME.2012.2212278 PubMedGoogle ScholarCrossref
73.
Schultze-Lutter  F, Schmidt  SJ, Theodoridou  A.  Psychopathology: a precision tool in need of re-sharpening.   Front Psychiatry. 2018;9:446. doi:10.3389/fpsyt.2018.00446 PubMedGoogle ScholarCrossref
74.
Varma  S, Simon  R.  Bias in error estimation when using cross-validation for model selection.   BMC Bioinformatics. 2006;7:91. doi:10.1186/1471-2105-7-91 PubMedGoogle ScholarCrossref
75.
Brady  LS, Potter  WZ, Gordon  JA.  Redirecting the revolution: new developments in drug development for psychiatry.   Expert Opin Drug Discov. 2019;14(12):1213-1219. doi:10.1080/17460441.2019.1666102 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    December 2, 2020

    Multimodal Machine Learning Workflows for Prediction of Psychosis in Patients With Clinical High-Risk Syndromes and Recent-Onset Depression

    Author Affiliations
    • 1Department of Psychiatry and Psychotherapy, Ludwig-Maximilian-University, Munich, Germany
    • 2Max-Planck Institute of Psychiatry, Munich, Germany
    • 3Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, United Kingdom
    • 4Institute of Human Genetics, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany
    • 5Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital Essen, University of Duisburg-Essen, Essen, Germany
    • 6Institute of Genomic Statistics and Bioinformatics, University of Bonn, Bonn, Germany
    • 7Max-Planck School of Cognition, Leipzig, Germany
    • 8International Max-Planck Research School for Translational Psychiatry, Munich, Germany
    • 9Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, New York
    • 10Department of Psychiatry and Psychotherapy, Faculty of Medicine and University Hospital, University of Cologne, Cologne, Germany
    • 11Department of Education, Psychology, and Communication, University of Bari Aldo Moro, Bari, Italy
    • 12Department of Psychiatry and Psychotherapy, Medical Faculty, Heinrich-Heine University, Düsseldorf, Germany
    • 13Institute for Mental Health, University of Birmingham, Birmingham, United Kingdom
    • 14Department of Psychiatry, Psychiatric University Hospital, University of Basel, Switzerland
    • 15Department of Psychiatry, University of Turku, Turku, Finland
    • 16GE Healthcare GmbH (previously GE Global Research GmbH), Munich, Germany
    • 17Department of Child and Adolescent Psychiatry, University of Münster, Münster, Germany
    • 18Department of Child and Adolescent Psychiatry, Psychotherapy and Psychosomatics, LVR Clinic Düsseldorf, Düsseldorf, Germany
    • 19Department of Child and Adolescent Psychiatry and Psychotherapy, University of Zürich, Zürich, Switzerland
    • 20University Hospital of Child and Adolescent Psychiatry, University Hospital Hamburg-Eppendorf, Hamburg, Germany
    • 21University Hospital of Child and Adolescent Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
    • 22Department of Psychiatry, Psychotherapy and Psychosomatics, University Hospital of Psychiatry Zurich, Zurich, Switzerland
    • 23Department of Psychiatry and Psychotherapy I, LVR Hospital Cologne, Cologne, Germany
    • 24Department of Neuroradiology, University Hospital of Zurich, Zurich, Switzerland
    • 25Melbourne Neuropsychiatry Centre, University of Melbourne and Melbourne Health, Melbourne, Australia
    • 26Department of Psychiatry and Psychotherapy, University of Münster, Münster, Germany
    • 27Department of Psychiatry and Psychotherapy, University of Lübeck, Lübeck, Germany
    • 28Department of Basic Medical Science, Neuroscience and Sense Organs, University of Bari Aldo Moro, Bari, Italy
    • 29Department of Neurosciences and Mental Health, Fondazione Istituto di Ricovero e Cura a Carattere Scientifico Ca’ Granda Ospedale Maggiore Policlinico, Milan, Italy
    • 30Department of Pathophysiology and Transplantation, University of Milan, Milan, Italy
    • 31Centre for Youth Mental Health, University of Melbourne, Melbourne, Australia
    • 32Orygen, the National Centre of Excellence for Youth Mental Health, Melbourne, Australia
    • 33Department of Psychology and Mental Health, Faculty of Psychology, Airlangga University, Surabaya, Indonesia
    JAMA Psychiatry. 2021;78(2):195-209. doi:10.1001/jamapsychiatry.2020.3604
    Key Points

    Question  Can a transition to psychosis be predicted in patients with clinical high-risk states or recent-onset depression by optimally integrating clinical, neurocognitive, neuroimaging, and genetic information with clinicians’ prognostic estimates?

    Findings  In this prognostic study of 334 patients and 334 control individuals, machine learning models sequentially combining clinical and biological data with clinicians’ estimates correctly predicted disease transitions in 85.9% of cases across geographically distinct patient populations. The clinicians’ lack of prognostic sensitivity, as measured by a false-negative rate of 38.5%, was reduced to 15.4% by the sequential prognostic model.

    Meaning  These findings suggest that an individualized prognostic workflow integrating artificial and human intelligence may facilitate the personalized prevention of psychosis in young patients with clinical high-risk syndromes or recent-onset depression.

    Abstract

    Importance  Diverse models have been developed to predict psychosis in patients with clinical high-risk (CHR) states. Whether prediction can be improved by efficiently combining clinical and biological models and by broadening the risk spectrum to young patients with depressive syndromes remains unclear.

    Objectives  To evaluate whether psychosis transition can be predicted in patients with CHR or recent-onset depression (ROD) using multimodal machine learning that optimally integrates clinical and neurocognitive data, structural magnetic resonance imaging (sMRI), and polygenic risk scores (PRS) for schizophrenia; to assess models’ geographic generalizability; to test and integrate clinicians’ predictions; and to maximize clinical utility by building a sequential prognostic system.

    Design, Setting, and Participants  This multisite, longitudinal prognostic study performed in 7 academic early recognition services in 5 European countries followed up patients with CHR syndromes or ROD and healthy volunteers. The referred sample of 167 patients with CHR syndromes and 167 with ROD was recruited from February 1, 2014, to May 31, 2017, of whom 26 (23 with CHR syndromes and 3 with ROD) developed psychosis. Patients with 18-month follow-up (n = 246) were used for model training and leave-one-site-out cross-validation. The remaining 88 patients with nontransition served as the validation of model specificity. Three hundred thirty-four healthy volunteers provided a normative sample for prognostic signature evaluation. Three independent Swiss projects contributed a further 45 cases with psychosis transition and 600 with nontransition for the external validation of clinical-neurocognitive, sMRI-based, and combined models. Data were analyzed from January 1, 2019, to March 31, 2020.

    Main Outcomes and Measures  Accuracy and generalizability of prognostic systems.

    Results  A total of 668 individuals (334 patients and 334 controls) were included in the analysis (mean [SD] age, 25.1 [5.8] years; 354 [53.0%] female and 314 [47.0%] male). Clinicians attained a balanced accuracy of 73.2% by effectively ruling out (specificity, 84.9%) but ineffectively ruling in (sensitivity, 61.5%) psychosis transition. In contrast, algorithms showed high sensitivity (76.0%-88.0%) but low specificity (53.5%-66.8%). A cybernetic risk calculator combining all algorithmic and human components predicted psychosis with a balanced accuracy of 85.5% (sensitivity, 84.6%; specificity, 86.4%). In comparison, an optimal prognostic workflow produced a balanced accuracy of 85.9% (sensitivity, 84.6%; specificity, 87.3%) at a much lower diagnostic burden by sequentially integrating clinical-neurocognitive, expert-based, PRS-based, and sMRI-based risk estimates as needed for the given patient. Findings were supported by good external validation results.

    Conclusions and Relevance  These findings suggest that psychosis transition can be predicted in a broader risk spectrum by sequentially integrating algorithms’ and clinicians’ risk estimates. For clinical translation, the proposed workflow should undergo large-scale international validation.

    Introduction

    The clinical high-risk (CHR) criteria for psychosis have been established to detect vulnerable individuals as early as possible to intercept disease development.1 These criteria identify a patient population with increased incidence compared with the general population,2 yet only 22% of patients with CHR as detected by ultra–high-risk criteria show a psychosis transition during a 3-year period.2 The clinical utility of the CHR designation may be further limited because its ascertainment is laborious and confined to specialized, well-equipped health care services that do not sufficiently cover the vulnerable population.3,4 Hence, improved prognostic accuracy and clinical scalability are needed to accurately identify patients truly at risk for psychosis.

    Prognostic accuracy may be increased using psychosis risk calculators for populations with CHR based on conventional statistics3,5-7 or machine learning,8-13 with studies finding that a first episode can be predicted with clinical data,3,14 combinations of clinical and cognitive data,6,15 neuroimaging,8 and, recently, with polygenic risk scores (PRS) for schizophrenia,16 among other measures.17 However, reviews18-20 have also highlighted methodological shortcomings, such as inadequate sample sizes and model validation strategies,21,22 that may have inflated accuracy. Moreover, studies suggested that psychosis does not only emerge from CHR states23,24 but occurs and can be predicted across a broader spectrum of comorbid conditions commencing in late adolescence and early adulthood.3,14 Hence, generalizable risk prediction models may require transdiagnostic discovery and validation populations, encompassing patients with CHR states and early-onset affective syndromes that share environmental, clinical, and neurobiological features.25-30 In addition, the growing diversity of risk prediction models originating from different data modalities has led to uncertainty about the minimum number of modalities needed to increase prognostic accuracy to a level justifying clinical implementation.2,31 Finally, algorithms should be compared and integrated with clinicians’ predictions of psychosis transition to determine their potential utility from public health and service provision perspectives.13,32

    Addressing these challenges, the European Union–funded PRONIA study (Personalised Prognostic Tools for Early Psychosis Management [https://www.pronia.eu]) collected multimodal longitudinal data from adolescents and young adults in CHR states, those with recent-onset depression (ROD), and healthy control individuals. We evaluated clinical, neuroanatomical, and genetic machine learning models trained to identify patients with CHR syndromes and ROD who undergo psychosis transition. We compared our models’ performance with our clinical raters’ ability to predict psychosis transition and explored whether the sequential integration of algorithmic and expert-based prognoses produces clinically efficient cybernetic workflows,33,34 that is, structured interactions between humans and machines that maximize prognostic accuracy while minimizing the examination burden in the given patient. We assessed potential confounders and moderators of prognostic performance and tested whether our models’ and raters’ estimates predicted not only psychosis transition but also distinct CHR syndromes and nonpsychotic disease trajectories. Finally, we explored whether a considerably condensed, and hence less burdensome, clinical model could be generalized to 3 independent patient cohorts with CHR syndromes and other mental conditions.35-37 This external validation step also benchmarked the neuroanatomical and combined models derived from the PRONIA cohort.

    Methods
    Study Design and Population

    The eMethods section in the Supplement details all of the methods for this prognostic study, which followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guideline.38 In summary, our analysis included 334 patients with CHR states (n = 167) or ROD (n = 167) recruited across 7 sites in Finland, Germany, Italy, Switzerland, and the United Kingdom from February 1, 2014, to May 31, 2017, using standardized inclusion and exclusion criteria (eTables 1 and 2 and CONSORT diagram in eFigure 1 in the Supplement). Follow-up for all patients ranged from 9 to 36 months,13 with visits every 3 months to the 18-month point and every 9 months thereafter. Furthermore, 334 healthy controls matched for age, sex, and site were included to evaluate prognostic patterns. Adult participants gave informed consent before study inclusion. Participants younger than 18 years and their guardians provided their written informed assent and consent. The PRONIA observational study was registered at the German Clinical Trials Register (DRKS00005042) and approved by all local research ethics committees.

    Sociodemographic and clinical variables were compared between diagnostic groups (eTable 3 in the Supplement), patients with psychosis transition and nontransition, and patients with 18-month or later follow-up data (discovery sample [n = 246]; PRONIA plus 18M) or earlier attrition (validation sample [n = 88]; PRONIA minus 18M) (Table 1). Psychosis transition was defined when at least 1 of the 5 positive symptom items in the Structured Interview for Psychosis–Risk Syndromes39 reached psychotic intensity daily for at least 7 days.40 Diagnoses of cases with psychosis transition are listed in eTable 4 in the Supplement.

    Prognostic Modeling Strategy

    Data were analyzed from January 1, 2019, to March 31, 2020. Using the machine learning software NeuroMiner, version 1.05 (GitHub [https://github.com/neurominer-git/NeuroMiner-1]), we constructed and tested unimodal, multimodal, and clinically scalable sequential risk calculators for transition prediction in the PRONIA plus 18M cohort using leave-one-site-out cross-validation (LOSOCV)21,41 (eMethods and eFigures 2-4 in the Supplement). We evaluated the risk calculators using baseline and longitudinal data and validated their specificity in the PRONIA minus 18M sample, which did not include cases with psychosis transition but provided all data modalities. In addition, 3 external data sets consisting of cases with psychosis transition and nontransition were available to test selected models.35-37

    First, unimodal risk calculators were trained with literature-based baseline predictors of psychosis transition,2,6,16 including prodromal symptoms,39,42 functioning,43,44 childhood adversity,45,46 and neurocognitive measures6,47 in the clinical-neurocognitive domain (eTable 5 in the Supplement); PRS for schizophrenia48 in the genetic domain16; and gray matter volume maps in the structural magnetic resonance imaging (sMRI) domain8,9 (eTable 6 in the Supplement). In addition, we evaluated our raters’ predictions, which, at the conclusion of baseline assessments, were provided as yes or no replies to the question, “Do you think the patient will likely transition to psychosis?” Then we assessed whether combining unimodal algorithms using stacked generalization49 improved prognostic accuracy (eFigures 2 and 3 in the Supplement and Table 2).13 Following the concept of expert-based machine learning,50 we integrated our raters’ estimates as additional predictors to produce a cybernetic model33 (eFigure 2 in the Supplement). Models’ predictive signatures were visualized in Figures 1 and 2 and eFigure 5 in the Supplement using measures of pattern element stability (cross-validation ratio) and pattern element significance (sign-based consistency; eMethods in the Supplement). In addition, the prognostic models were assessed using random-label permutations (Table 2). Raters’ and models’ performances were compared statistically at the omnibus level using the Quade test,51 an extension of the nonparametric Wilcoxon signed rank test, followed by post hoc pairwise mean differences tests using the t distribution.52 Statistical significance was determined at α = .05. Obtained P values were 2-sided; P values computed in the pairwise classifier comparisons were corrected using the false discovery rate (FDR) (Figure 3). Classifiers were visually compared in eFigure 6 in the Supplement).51,52

    In-Depth Model Evaluation

    We tested differences between the prognostic assignment groups and the matched healthy control group to determine whether the models’ predictive patterns represented a deviation from normality (eFigures 7-9 in the Supplement). Potential model confounders and moderators were systematically tested, including image quality (eFigure 10 in the Supplement), treatments (eTable 7 in the Supplement), follow-up frequency and duration (eFigure 11 and eTable 8 in the Supplement), site-related effects (eTables 9-11 in the Supplement), baseline study group membership (eTables 12 and 13 in the Supplement), and, specifically, the inclusion of patients with ROD (eTable 14 and eFigure 12 in the Supplement). Using the patients’ longitudinal data, we evaluated whether model predictions were not specific for the binary transition vs nontransition outcome, but we also separated transitions from nonremitting/de novo CHR symptom courses (P-CHR) and asymptomatic/nonpersisting trajectories (NP-CHR) (eTable 15 in the Supplement). To further explore a prognostic generalization effect,13 we used unsupervised machine learning (eMethods in the Supplement) to construct fine-grained CHR syndrome trajectories (eFigure 13A in the Supplement) and used linear mixed-effects modeling to compare trajectories between predicted and observed outcome groups (eFigure 13B [part 1] in the Supplement). Finally, we investigated whether assignments generalized to the prediction of nonpsychotic outcomes (eTable 16 in the Supplement).

    Optimization for Clinical Scalability

    To facilitate clinical implementation, we developed a sequential prediction method that optimizes the ordering and number of data modalities as well as the prognostic uncertainty thresholds to decide whether a patient needs further testing (eTable 17 and eMethods in the Supplement). We analyzed the identified optimal prognostic workflow (eFigure 14 in the Supplement) and tested whether it achieved similar performance as the fully stacked models at lower diagnostic burden for the patients. To further enhance clinical scalability, we condensed the clinical-neurocognitive model, which was the workflow’s entry point, from 141 to 7 (5.0%) variables (Figure 2) using sign-based consistency mapping (eMethods in the Supplement).53 We tested the condensed model and the respective workflow’s specificity in the PRONIA minus 18M sample (Table 2). Finally, we explored whether the prognostic sequence could be further trimmed using diagnostic parsimony regularization (eMethods in the Supplement and Table 2). Nonregularized and regularized workflows including the full or condensed clinical-neurocognitive models were compared in eFigure 15 in the Supplement.

    External Validation Experiments

    We validated the condensed clinical-neurocognitive model in 2 external cohorts: 146 patients with CHR (aged 15-35 years; 16 [11.0%] transitions) provided by the Zurich Early Recognition Program (ZInEP) (eTable 18 in the Supplement)36 and 462 patients with diverse mental conditions (aged 8-17 years; 13 [2.8%] transitions) drawn from the Bi-national Evaluation of At-Risk Symptoms in Children and Adolescents study (eTable 19 in the Supplement).37 Second, we validated the sMRI-based model in ZInEP and in 37 patients with CHR (16 [43.2%] transitions) from the Früherkennung von Psychosen study (Table 2).35,54 To validate the increased performance of multimodal risk calculators, we trained a stacked model using the condensed clinical-neurocognitive and sMRI-based models, tested it in the ZInEP data (Table 2), and used the Quade test51 to compare the 2 unimodal prediction models with the stacked classifier (Figure 3). Finally, we made our models available in the NeuroMiner Model Library (http://www.proniapredictors.eu) to facilitate their independent external validation.

    Results
    Group-Level Differences

    A total of 668 patients and controls were included in the analysis (mean [SD] age, 25.1 [5.8] years; 354 [53.0%] female and 314 [47.0%] male). Patients in the PRONIA plus 18M and PRONIA minus 18M groups were followed up for a mean (SD) of 842.7 (272.3) and 390.6 (99.6) days, respectively (Table 1). They did not differ in any examined variable (Table 1). Psychosis transition occurred after a mean (SD) of 246.9 (244.5) days in 26 cases and developed into schizophrenia in 8 (30.8%) (eTable 4 in the Supplement). Follow-up durations differed between sites but not time to psychosis transition (eTable 9 in the Supplement). Compared with nontransition, individuals with psychosis transition had more repeated school years (mean [SD], 0.67 [0.88] vs 0.26 [0.61] years) and more prevalent attenuated positive symptoms (APSs) at baseline (18 of 26 [69.2%] vs 88 of 308 [28.6%]) (Table 1). Of 167 patients with CHR, 23 (13.8%) developed psychosis, whereas 53 (31.7%) had remitted from CHR criteria at the 9-month visit. Major depression affected 103 patients with CHR syndromes (61.7%) but was not differentially associated with psychosis transition (Table 1). Nonremitting mood and anxiety disorders were present during follow-up in cases with psychosis transition and nontransition (eTable 16 in the Supplement). Compared with healthy controls, ROD was associated with low but significantly elevated CHR symptom scores (eg, mean [SD] Structured Interview for Psychosis–Risk Syndromes positive symptoms, 0.43 [0.44] vs 0.10 [0.21]; P < .001; mean [SD] Schizophrenia Proneness Instrument: Cognitive Disturbances symptoms, 0.24 [0.32] vs 0.02 [0.08]; P < .001) (eTable 3 in the Supplement). Functional-cognitive and interpersonal abnormalities were comparable between ROD and CHR groups. Of 167 patients with ROD, 32 (19.2%) developed psychosis-related outcomes, including CHR states in 29 (17.4%) and psychosis transitions in 3 (1.8%).

    Machine Learning Analyses

    The full clinical-neurocognitive model predicted psychosis transition with a balanced accuracy (BAC-LOSOCV) of 75.7% (sensitivity, 84.6%; specificity, 66.8%; P < .001) (Table 2). Significant predictors as determined by sign-based consistency mapping (z>3.28; P < .05 for FDR) were APS and motor disturbances, a nonsupportive family environment during childhood, and reduced facial emotion recognition (Figure 1B). Compared with healthy controls, those assigned to psychosis transition had elevated abnormality scores in these variables, whereas those with nontransition assignments showed an abnormality pattern focused on unusual thought content, suspiciousness, perceptual abnormalities, and childhood adversity, with higher visual working memory and semantic verbal fluency performance (P < .05 for FDR) (eFigure 7 in the Supplement).

    The PRS-based model achieved a BAC-LOSOCV of 66.1% (sensitivity, 76.0%; specificity, 56.2%; P < .001). Among the 10 tested genome-wide significance thresholds, only P = 1.0 reached significance (Figure 2). Compared with healthy controls, those with psychosis transition assignments had elevated PRS across all whole-genome P thresholds, whereas those with nontransition assignments expressed reduced PRS at P ≥ 5.7 × 10−4 (eFigure 8A in the Supplement). Patients with observed nontransition and those with ROD did not show reduced PRS (eFigure 8B-C in the Supplement).

    The sMRI-based model attained a BAC-LOSOCV of 70.7% (sensitivity, 88.0%; specificity, 53.5%; P < .001). At a stability threshold (cross-validation ratio) of at least 3, the brain pattern predicting psychosis transition involved reduced gray matter volume in the superior temporal, supramarginal, angular, orbitofrontal, inferior frontal, dorsomedial prefrontal, and occipital cortices. The predictive pattern also included areas of increased gray matter volume covering the dorsolateral prefrontal, precuneal, insular, hippocampal, and cerebellar brain regions (eFigure 5 in the Supplement). This brain signature differentiated psychosis transition-assigned patients from healthy controls, whereas nontransition-assigned patients showed a partial pattern inversion with increased temporo-occipital gray matter volume compared with healthy controls (see threshold-free cluster enhancement statistics thresholded at P < .05 for FDR) (eFigure 9 in the Supplement).

    Clinical raters achieved a BAC of 73.2% (sensitivity, 61.5%; specificity, 84.9%), which was independent of the length of their early recognition experience (mean [SD] length for correct predictions: 31.8 [46.8] months; mean [SD] length for wrong predictions, 29.4 [38.1] months; unpaired 2-tailed t326 = 0.35; P = .72). The stacked model combining unimodal algorithms produced a BAC-LOSOCV of 82.9% (sensitivity, 80.8%; specificity, 85.0%). Integration of raters’ prognoses into the stacked model increased BAC-LOSOCV to 85.5% (sensitivity, 84.6%; specificity, 86.4%), and they were the cybernetic model’s most relevant predictor (Figure 2C).

    Baseline Moderators of Prediction Performance

    Image quality (eFigure 10 in the Supplement), baseline treatments or previous hospitalizations (eTable 7 in the Supplement), follow-up duration and frequency (eTable 8 and eFigure 11 in the Supplement), and site effects (eTable 10 in the Supplement) did not influence model performance. Study group (CHR vs ROD) could be classified with a BAC-LOSOCV of 82.3% (sensitivity, 73.7%; specificity, 91.0%) using clinical-neurocognitive data and with a BAC of 55.7% (sensitivity, 49.7%; specificity, 61.7%) using PRS (eTable 12 in the Supplement). These diagnostic classifiers explained 49.7% and 18.3%, respectively, of the variance of the respective prognostic counterparts (both P < .001 for FDR) (eTable 13 in the Supplement). Raters’ prognoses were also significantly informed by baseline study group (BAC, 62.7%; sensitivity, 30.9%; specificity, 94.5%) (eTable 12 in the Supplement). The removal of the patients with ROD from the training samples or their substitution with healthy controls significantly reduced the balanced accuracy of all risk calculators by −2.8% to −11.7% in the CHR group (eTable 14 and eFigure 12 in the Supplement).

    Prognostic Generalization

    Prognostic assignments also delineated psychosis transition and P-CHR and NP-CHR courses irrespective of model type (eTable 15 in the Supplement). The separability of P-CHR from NP-CHR courses was lower and only significant for clinical-neurocognitive models and raters. The nonnegative matrix factorization and linear mixed-model analysis showed that CHR syndrome trajectories were stratified by the predictions of the clinical-neurocognitive classifier (factor F1 paranoid-perceptual disturbances, F1,832 = 136.35 [P < .001 for FDR]; factor F2 disturbances of volition and affect, F1,832 = 12.76 [P = .001 for FDR]; factor F3 functional disturbances, F1,832 = 24.34 [P < .001 for FDR]; factor F4 cognitive disturbances, F1,832 = 44.05 [P = .007 for FDR]) as well as by raters' outcome estimates (factor F1 paranoid-perceptual disturbances, F1,825 = 30.64 [P < .001 for FDR]; factor F2 disturbances of volition and affect, F1,825 = 6.15 [P = .03 for FDR]; factor F3 functional disturbances, F1,825 = 5.80 [P < .03 for FDR]; factor F4 cognitive disturbances, F1,825 = 9.00 [P = .007 for FDR]) (eFigure 13 in the Supplement). Nonpsychotic disease courses were not associated with prognostic assignments (eTable 16 in the Supplement).

    Clinical Scalability and External Validation of Predictive Models

    We identified a prognostic sequence that produced a BAC-LOSOCV of 85.9% (sensitivity, 84.6%; specificity, 87.3%) (Table 2) and started with the clinical-neurocognitive model, added raters, and finally integrated PRS- and sMRI-based models (eFigure 14 in the Supplement). Across this sequence, the positive likelihood ratio increased to 6.6, whereas the population requiring all prognostic assessments decreased to 41.1%. Regularization for diagnostic parsimony further reduced this population to 23.2% (regularization strength Γ = 0.5) and 0 (regularization strength Γ = 1.0) (PRONIA plus 18M sample, left panel of eFigure 15B in the Supplement), with the latter parsimony level significantly reducing BAC by −6.4% (PRONIA plus 18M sample, left panel of eFigure 15C, part 2 in the Supplement). Highly similar findings were obtained when analyzing the complete PRONIA cohort (right panels in eFigure 15B and C in the Supplement).

    Further clinical scalability experiments showed that the condensed clinical-neurocognitive model matched the full model in correctly predicting PRONIA minus 18M cases with nontransition (specificity, 65.9%). Furthermore, its performance in the external ZInEP-CHR cohort (BAC, 65.3%; sensitivity, 87.5%; specificity, 43.1%) and the Bi-national Evaluation of At-Risk Symptoms in Children and Adolescents sample (BAC, 70.4%; sensitivity, 76.9%; specificity, 63.9%) was similar to the full model in the PRONIA-CHR sample (BAC, 63.3%; sensitivity, 87.0%; specificity, 39.6%) (eTable 14 in the Supplement) and the complete PRONIA cohort (BAC, 72.8%; sensitivity, 88.0%; specificity, 57.5%) (Table 2). Replacing the full model with its condensed counterpart in the nonregularized or regularized (Γ = 0.5) workflows did not increase the false-positive rate in the PRONIA minus 18M sample or the complete PRONIA cohort (eFigure 14C in the Supplement).

    The validation of the unimodal sMRI-based workflow component in the ZInEP (BAC, 67.9%; sensitivity, 75.0%; specificity, 60.8%) and Früherkennung von Psychosen (BAC, 68.5%; sensitivity, 75.0%; specificity, 61.9%) samples approximated the PRONIA-CHR results (BAC, 70.8%; sensitivity, 86.4%; specificity, 55.3%) (eTable 14 in the Supplement). Finally, the stacked risk calculator composed of the condensed clinical-neurocognitive and sMRI-based models significantly outperformed these models in the ZInEP data (BAC, 71.3%; sensitivity, 75.0%; specificity, 67.7%) (Figure 3 and Table 2).

    Comparisons of Risk Calculators and Clinical Raters

    All risk calculators were significant in the permutation analysis (mean [SD] BAC, 77.3 [4.8]; mean [SD] sensitivity, 80.2% [6.5%]; mean [SD] specificity, 74.5% [13.8%]; P < .001 for FDR for all models) Table 2), but differences in BAC emerged (Figure 3). Multimodal risk calculators outperformed all unimodal counterparts (t range, 3.14-6.20; P < 3.19 × 10−8 to P = .002 for FDR), whereas the nonregularized prognostic sequence did not differ from the cybernetic model (mean [SD] BAC for nonregularized sequence, 83.7% [9.6%]; mean [SD] BAC for cybernetic model, 83.4% [9.6%]; t = 0.21; P = .44 for FDR). In addition, the stacked model (mean [SD] BAC, 79.7% [7.9%]) was outperformed by both the cybernetic model (mean [SD] BAC, 81.5% [9.6%]; t = 3.18; P = .001 for FDR) and the nonregularized sequential model (mean [SD] BAC, 82.0% [9.6%]; t = 3.82; P < .001 for FDR) in the complete PRONIA sample. Raters were comparable to unimodal risk calculators (mean [SD] BAC for raters, 68.2% [12.6%]; mean [SD] BAC range for unimodal predictors, 67.5% [16.4%] to 74.8% [6.8%]; t range, 0.002-1.56; P = .50 to P = .07 for FDR) but were outperformed by multimodal prediction algorithms in terms of higher BAC and reduced prognostic variability (stacked model vs raters, t = 4.64 [P < .001 for FDR]; cybernetic model vs raters, t = 7.82 [P < .001 for FDR]; sequence model vs raters, t = 8.46 [P < .001 for FDR]). Finally, the nonregularized sequential model reduced raters’ false-negative rate from 38.5% to 15.4% (PRONIA plus 18M sample) or 19.2% (complete PRONIA cohort) of cases.

    Discussion

    Using a thorough model discovery and validation approach,21 our study demonstrated geographic transportability of expert-based clinical and biological psychosis transition prediction approaches across a transdiagnostic, multinational risk population. We found that combined risk calculators outperformed all unimodal counterparts and clinical raters in terms of prognostic accuracy and cross-site stability. Importantly, our study revealed that the increased diagnostic burden arising from data fusion could be mitigated through optimized sequential testing that arranges clinicians and risk calculators into clinically scalable prognostic workflows. Based on this form of deferral learning,55 we showed that the complete assessment battery is only needed in 23.2% of the initial population (eFigure 15B in the Supplement). This subgroup was enriched for patients who received a prediction of psychosis transition in the initial clinical-neurocognitive examination, suggesting that biological markers of psychosis transition are useful for delineating true-positive from false-positive findings at the later steps of a multistep prognostic assessment.

    Examining the baseline heterogeneity of our transdiagnostic population, we found functional-neurocognitive impairments in the ROD group akin to the CHR group and low between-group neuroanatomical and genetic separability (eTable 12 in the Supplement), supporting the neurobiological proximity between early-onset affective and psychotic disorders.26,30,56 Although CHR syndromes expectedly separated CHR and ROD groups at the cross-sectional level, we observed that these syndromes emerged in 19.2% of patients with ROD during the follow-up period, which led to psychosis transition in 1.8% of cases.56 Strikingly, our analyses also showed that the models’ prognostic accuracy, particularly the sensitivity for psychosis transition in patients with CHR, depended on patients with ROD being part of the model discovery process, which further supports the pooling of both groups into a broader risk population (eTable 14 and eFigure 12 in the Supplement). Our finding of a transdiagnostic predictability of psychosis was corroborated by the generalizability of the clinical-neurocognitive and neuroanatomical models to external samples, which showed markedly different risk levels, age distributions, and diagnostic compositions.

    The in-depth analysis of the clinical-neurocognitive domain revealed that the presence of APSs facilitated a good baseline separability of CHR syndromes vs ROD and substantially informed the prediction of psychosis transition (eTable 13 in the Supplement). However, measures of childhood adversity,46 motor disturbances,57 and facial affect recognition58 did not overlap between diagnostic and prognostic models (eFigure 16 in the Supplement) and thus could be regarded as transdiagnostic markers59-62 of poor psychosis-related outcomes, including transition to psychosis. This interpretation was supported by the prognostic generalization of the clinical-neurocognitive model to the clinically relevant separation of patients with (1) nonremitting/de novo and nonsymptomatic CHR syndrome courses (eTable 15 in the Supplement) or (2) unfavorable perceptual, affective, functional, and basic symptom trajectories (eFigure 13 in the Supplement).63,64 Importantly, the model’s prognostic generalization capacity did not encompass nonpsychotic diagnoses (eTable 16 in the Supplement), and, thus, its prognostic pluripotency was confined to diverse CHR-specific symptom courses.65

    Furthermore, we confirmed the prognostic value of PRS for schizophenia,48 as reported recently,16 and extended those findings by showing that genetic information augments the performance of clinical-neurocognitive models and prognostic workflows in a broader risk population (Table 2 and eTable 20 in the Supplement). Within this transdiagnostic setting, we replicated group-level differences among patients with psychosis transition, patients with nontransition, and healthy controls16 but also found that PRS-based prognostic assignments specifically differentiated APS-related trajectories (eFigure 13 in the Supplement).66 They also delineated patient groups with abnormally high and low genetic risk compared with healthy controls (eFigure 8 in the Supplement)—a finding that may point to distinct environmental and/or neurobiological pathways conferring risk and resilience to psychosis.67

    The analysis of the structural neuroimaging data revealed a psychosis-predictive brain signature that generalized well across 3 independent cohorts. This signature overlapped with brain alterations previously reported to correlate with perceptual abnormalities, disorganization of speech and thought, and poor insight in early, subsyndromal, or prodromal stages of psychosis.9,68-70 Interestingly, nontransition-assigned patients showed reversed temporo-occipital volume reductions, which differentiated them from healthy controls (eFigure 9 in the Supplement). These findings may point to ongoing compensatory mechanisms of resilience to psychosis, as reported previously in a longitudinal sMRI study of adolescents with CHR.71 In this regard, our sMRI-based risk calculator may serve as a useful tool for enriching future observational studies and clinical trials with at-risk patients who express potential brain mechanisms of resilience to psychosis transition.

    We observed that our raters matched unimodal risk calculators in predicting psychosis as measured by their BAC. However, raters also showed a pronounced optimism bias (low sensitivity and high specificity) toward the true risk of poor clinical outcome.13 It is noteworthy that their prognoses were based on all information collected in an extended study-related assessment and likely would be less accurate in routine, time-restricted diagnostic settings. Because the algorithmic counterparts showed exactly the inverted bias (high sensitivity and low specificity), the integration of clinicians and risk calculators into the cybernetic model produced a superior predictive system.50 Furthermore, our prognostic workflows demonstrated that similar levels of prognostic accuracy can be achieved by reducing the false-positive rate through sequential model application in patients with an estimated higher risk for psychosis transition (eFigure 14 in the Supplement). In this subgroup, the removal of the final sMRI-based assessment step increased false-positive findings (eFigure 15 in the Supplement), suggesting that the cost-benefit ratio of expensive neuromarkers needs to be individually adjusted according to the patient’s predicted risk.72

    The finding that prognostic workflows always started with the clinical-neurocognitive model places the recognition of the clinical gestalt of emerging psychosis at the gateway of more precise early detection techniques.73 Our scalability experiments suggest that the laborious recognition of this pattern currently practiced in early recognition services could be effectively condensed to a few clinical-neurocognitive variables,6 thus enhancing the clinical utility of the proposed workflow. Nonetheless, future studies should revisit the validity of the selected 7 variables because they have been taken out of their original assessment context. Further studies also need to quantitatively explore the information patterns guiding clinicians’ gut-feeling estimates of psychosis transition and in turn foster more effective clinical early recognition strategies that integrate with cybernetic systems.

    Limitations

    Psychosis transitions were limited to 26 individuals in the PRONIA discovery sample. This sample size increased the risk of producing overly optimistic prediction results owing to an accidental collection of well-classifiable cases. We implemented a multistep model validation procedure to guard against this possibility, including label permutation testing, strict nested cross-validation of all processing steps,74 in-depth model analysis to assess possible prognostic confounds and moderators, specificity testing of all models in a completely held-back portion of the PRONIA sample, and model benchmarking in 3 independent data sets, which provided a further 45 individuals with psychosis transition and 600 with nontransition for external validation. Owing to limited data availability in these samples, only the condensed clinical-neurocognitive, sMRI-based, and a specific stacked risk calculator trained on the outputs of the former 2 models could be externally validated. However, our internal-external validation approach followed established guidelines for model construction and validation.21 In keeping with this literature, the similar performance levels observed in our LOSOCV and independent validation experiments support the validity of the models not tested in external samples.

    Conclusions

    In this prognostic study, we identified generalizable risk assessment tools that can be arranged into a multimodal prognostic workflow for a clinically viable, individualized prediction of psychosis in patients with CHR states and ROD. Our study showed for the first time, to our knowledge, that the augmentation of human prognostic abilities with algorithmic pattern recognition improves prognostic accuracy to margins that likely justify the clinical implementation of cybernetic decision-support tools. New international collaborations, such as the HARMONY (Harmonization of At Risk Multisite Observational Networks for Youth) initiative,75 may help to propel a reciprocal and iterative process of clinical validation and refinement of these prognostic tools in real-world early recognition services.

    Back to top
    Article Information

    Accepted for Publication: September 12, 2020.

    Published Online: December 2, 2020. doi:10.1001/jamapsychiatry.2020.3604

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Koutsouleris N et al. JAMA Psychiatry.

    Corresponding Author: Nikolaos Koutsouleris, MD, Department of Psychiatry and Psychotherapy, Ludwig-Maximilian-University, Nussbaumstrasse 7, D-80336 Munich, Germany (nikolaos.koutsouleris@med.uni-muenchen.de).

    Author Contributions: Drs Schultze-Lutter, Theodoridou, and Meisenzahl contributed equally to this work. Dr Koutsouleris had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Koutsouleris, Ruhrmann, Kambeitz, Hietala, Schirmer, Schimmelmann, Falkai, Salokangas, Borgwardt, Wood, Upthegrove, Theodoridou, Meisenzahl.

    Acquisition, analysis, or interpretation of data: Koutsouleris, Dwyer, Degenhardt, Maj, Urquijo-Castro, Sanfelici, Popovic, Oeztuerk, Haas, Weiske, Ruef, Kambeitz-Ilankovic, Antonucci, Neufang, Schmidt-Kraepelin, Ruhrmann, Penzel, Kambeitz, Haidl, Rosen, Chisholm, Riecher-Rössler, Egloff, Schmidt, Andreou, Hietala, Schirmer, Romer, Walger, Franscini, Traber-Walker, Schimmelmann, Flückiger, Michel, Rössler, Borisov, Krawitz, Heekeren, Buechler, Pantelis, Salokangas, Lencer, Bertolino, Borgwardt, Noethen, Brambilla, Wood, Upthegrove, Schultze-Lutter, Theodoridou, Meisenzahl.

    Drafting of the manuscript: Koutsouleris, Degenhardt, Kambeitz, Salokangas, Upthegrove, Meisenzahl.

    Critical revision of the manuscript for important intellectual content: Koutsouleris, Dwyer, Degenhardt, Maj, Urquijo-Castro, Sanfelici, Popovic, Oeztuerk, Haas, Weiske, Ruef, Kambeitz-Ilankovic, Antonucci, Neufang, Schmidt-Kraepelin, Ruhrmann, Penzel, Kambeitz, Haidl, Rosen, Chisholm, Riecher-Rössler, Egloff, Schmidt, Andreou, Hietala, Schirmer, Romer, Walger, Franscini, Traber-Walker, Schimmelmann, Flückiger, Michel, Rössler, Borisov, Krawitz, Heekeren, Buechler, Pantelis, Falkai, Salokangas, Lencer, Bertolino, Borgwardt, Noethen, Brambilla, Wood, Schultze-Lutter, Theodoridou, Meisenzahl.

    Statistical analysis: Koutsouleris, Maj, Popovic, Borisov, Krawitz, Meisenzahl.

    Obtained funding: Koutsouleris, Degenhardt, Ruhrmann, Riecher-Rössler, Schirmer, Pantelis, Salokangas, Borgwardt, Brambilla, Wood, Meisenzahl.

    Administrative, technical, or material support: Koutsouleris, Dwyer, Sanfelici, Oeztuerk, Haas, Ruef, Kambeitz-Ilankovic, Antonucci, Ruhrmann, Penzel, Kambeitz, Haidl, Rosen, Riecher-Rössler, Egloff, Schmidt, Andreou, Hietala, Romer, Flückiger, Michel, Buechler, Pantelis, Salokangas, Borgwardt, Noethen, Brambilla, Upthegrove, Theodoridou, Meisenzahl.

    Supervision: Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Kambeitz, Andreou, Hietala, Schirmer, Walger, Flückiger, Michel, Krawitz, Pantelis, Falkai, Lencer, Bertolino, Borgwardt, Noethen, Brambilla, Upthegrove, Schultze-Lutter, Theodoridou.

    Conflict of Interest Disclosures: Dr Koutsouleris reported receiving grants from the European Union (EU) during the conduct of the study and having a patent to US20160192889A1 issued. Ms Sanfelici reported receiving personal fees from H. Lundbeck A/S outside the submitted work. Dr Ruhrmann reported receiving grants from the European Commission during the conduct of the study. Dr Riecher-Rössler reported receiving grants from the EU during the conduct of the study. Dr Andreou reported receiving nonfinancial support from Sunovion Pharmaceuticals, Inc, and H. Lundbeck A/S outside the submitted work. Dr Hietala reported receiving personal fees from Orion Company, Ltd, Otsuka Pharmaceutical Co, Ltd, and H. Lundbeck A/S and European College of Neuropsychopharmacology Congress participation support from Takeda Pharmaceutical Company Limited during the conduct of the study. Dr Schirmer reported receiving personal fees from GE Healthcare GmbH outside the submitted work. Dr Romer reported receiving grants from the EU during the conduct of the study. Dr Schimmelmann reported receiving personal fees from Shire Deutschland GmbH outside the submitted work. Dr Flückiger reported receiving grants from the Swiss National Foundation during the conduct of the study. Dr Michel reported receiving grants from the Swiss National Foundation during the conduct of the study. Dr Rössler reported receiving grants from The Zurich Program for Sustainable Development of Mental Health Services (for Zurich Early Recognition Program [ZInEP]) and support by a private donation. Dr Heekeren reported receiving grants from The Zurich Program for Sustainable Development of Mental Health Services during the conduct of the study. Dr Pantelis reported receiving grants from Australian National Health and the Medical Research Council during the conduct of the study and personal fees from H. Lundbeck A/S and Australia Pty Ltd outside the submitted work. Dr Noethen reported receiving personal fees from the Lundbeck Foundation, Robert-Bosch-Stiftung GmbH, HMG Systems Engineering GmbH, Shire Deutschland GmbH, and Life & Brain GmbH outside the submitted work and having a patent to Means and Methods for Establishing a Clinical Prognosis of Diseases Associated With the Formation of Aggregates of Aß1-42 issued. Dr Upthegrove reported receiving personal fees from Sunovion Pharmaceuticals, Inc, outside the submitted work. Dr Meisenzahl reported having a patent to US20160192889A1 licensed. No other disclosures were reported.

    Funding/Support: PRONIA (Personalised Prognostic Tools for Early Psychosis Management) is a Collaboration Project funded by the EU under the 7th Framework Programme and grant agreement 602152. This study was also supported by grant COMMITMENT by the German Federal Ministry of Education and Research (BMBF) within the e:Med programme (Dr Degenhardt), COST Action EnGagE CA17130 from the EU COST Programme (Dr Degenhardt), and the Else-Kröner-Fresenius-Foundation through the Clinician Scientist Program Else-Kröner-Fresenius-Foundation-Translational Psychiatry (Drs Popovic and Oeztuerk).

    Role of the Funder/Sponsor: The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    The PRONIA Consortium: The following members of the PRONIA Consortium performed the screening, recruitment, rating, examination, and follow-up of the study participants and were involved in implementing the examination protocols of the study, setting up its information technology infrastructure, and organizing the flow and quality control of the data analyzed in this study between the local study sites and the central study database: Shalaila Haas, Alkomiet Hasan, Claudius Hoff, Ifrah Khanyaree, Aylin Melo, Susanna Muckenhuber-Sternbauer, Yanis Köhler, Ömer Öztürk, Nora Penzel, David Popovic, Adrian Rangnick, Sebastian von Saldern, Rachele Sanfelici, Moritz Spangemacher, Ana Tupac, Maria Fernanda Urquijo-Castro, Johanna Weiske, Antonia Wosgien, and Camilla Krämer (Department of Psychiatry and Psychotherapy, Ludwig-Maximilian-University); Karsten Blume, Dennis Hedderich, Dominika Julkowski, Nathalie Kaiser, Thorsten Lichtenstein, Ruth Milz, Alexandra Nikolaides, Tanja Pilgram, Mauro Seves, and Martina Wassen (Department of Psychiatry and Psychotherapy, University of Cologne); Christina Andreou, Laura Egloff, Fabienne Harrisberger, Ulrike Heitz, Claudia Lenz, Letizia Leanza, Amatya Mackintosh, Renata Smieskova, Erich Studerus, Anna Walter, and Sonja Widmayer (Department of Psychiatry, Psychiatric University Hospital, University of Basel); Chris Day, Sian Lowri Griffiths, Mariam Iqbal, Mirabel Pelton, Pavan Mallikarjun, Alexandra Stainton, and Ashleigh Lin (Institute for Mental Health and School of Psychology, University of Birmingham); Alexander Denissoff, Anu Ellilä, Tiina From, Markus Heinimaa, Tuula Ilonen, Päivi Jalo, Heikki Laurikainen, Antti Luutonen, Akseli Mäkela, Janina Paju, Henri Pesonen, Reetta-Liina Säilä, Anna Toivonen, and Otto Turtonen (Department of Psychiatry, University of Turku); Sonja Botterweck, Norman Kluthausen, Gerald Antoch, Julian Caspers, and Hans-Jörg Wittsack (Department of Psychiatry, Psychiatric University Hospital LVR/Heinrich-Heine-University Düsseldorf, University of Düsseldorf); Giuseppe Blasi, Giulio Pergola, Grazia Caforio, Leonardo Fazio, Tiziana Quarto, Barbara Gelao, Raffaella Romano, Ileana Andriola, Andrea Falsetti, Marina Barone, Roberta Passiatore, and Marina Sangiuliano (Department of Basic Medical Science, Neuroscience and Sense Organs, University of Bari Aldo Moro); Marian Surmann, Olga Bienek, and Udo Dannlowski (Department of Psychiatry and Psychotherapy, University of Münster); Ana Beatriz Solana, Manuela Abraham, and Timo Schirmer (GE Global Research, Inc); Carlo Altamura, Marika Belleri, Francesca Bottinelli, Adele Ferro, and Marta Re (Department of Neuroscience and Mental Health, Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico, Workgroup of Paolo Brambilla, University of Milan); Emiliano Monzani and Maurizio Sberna (Programma 2000, Niguarda Hospital, Workgroup of Paolo Brambilla, University of Milan); Giampaolo Perna, Maria Nobile, and Alessandra Alciati (San Paolo Hospital, Workgroup of Paolo Brambilla, University of Milan); Armando D’Agostino and Lorenzo Del Fabro (Villa San Benedetto Menni, Albese con Cassano, Workgroup of Paolo Brambilla, University of Milan); Matteo Balestrieri, Carolina Bonivento, Giuseppe Cabras, and Franco Fabbro (Department of Medical Area, Workgroup of Paolo Brambilla, University of Udine); and Marco Garzitto and Sara Piccin (IRCCS Scientific Institute E. Medea, Polo FVG, Workgroup of Paolo Brambilla, University of Udine).

    References
    1.
    McGorry  PD, Mei  C.  Ultra-high-risk paradigm: lessons learnt and new directions.   Evid Based Ment Health. 2018;21(4):131-133. doi:10.1136/ebmental-2018-300061 PubMedGoogle ScholarCrossref
    2.
    Fusar-Poli  P, Salazar de Pablo  G, Correll  CU,  et al.  Prevention of psychosis: advances in detection, prognosis, and intervention.   JAMA Psychiatry. 2020;77(7):755-765. doi:10.1001/jamapsychiatry.2019.4779 PubMedGoogle ScholarCrossref
    3.
    Fusar-Poli  P, Rutigliano  G, Stahl  D,  et al.  Development and validation of a clinically based risk calculator for the transdiagnostic prediction of psychosis.   JAMA Psychiatry. 2017;74(5):493-500. doi:10.1001/jamapsychiatry.2017.0284 PubMedGoogle ScholarCrossref
    4.
    Schultze-Lutter  F, Michel  C, Schmidt  SJ,  et al. EPA guidance on the early detection of clinical high risk states of psychoses.  Eur Psychiatry. 2015;30(3):405-416. doi:10.1016/j.eurpsy.2015.01.010
    5.
    Carrión  RE, McLaughlin  D, Goldberg  TE,  et al.  Prediction of functional outcome in individuals at clinical high risk for psychosis.   JAMA Psychiatry. 2013;70(11):1133-1142. doi:10.1001/jamapsychiatry.2013.1909 PubMedGoogle ScholarCrossref
    6.
    Cannon  TD, Yu  C, Addington  J,  et al.  An individualized risk calculator for research in prodromal psychosis.   Am J Psychiatry. 2016;173(10):980-988. doi:10.1176/appi.ajp.2016.15070890 PubMedGoogle ScholarCrossref
    7.
    Zhang  T, Xu  L, Tang  Y,  et al; SHARP (ShangHai At Risk for Psychosis) Study Group.  Prediction of psychosis in prodrome: development and validation of a simple, personalized risk calculator.   Psychol Med. 2019;49(12):1990-1998. doi:10.1017/S0033291718002738PubMedGoogle ScholarCrossref
    8.
    Koutsouleris  N, Meisenzahl  EM, Davatzikos  C,  et al.  Use of neuroanatomical pattern classification to identify subjects in at-risk mental states of psychosis and predict disease transition.   Arch Gen Psychiatry. 2009;66(7):700-712. doi:10.1001/archgenpsychiatry.2009.62 PubMedGoogle ScholarCrossref
    9.
    Koutsouleris  N, Riecher-Rössler  A, Meisenzahl  EM,  et al.  Detecting the psychosis prodrome across high-risk populations using neuroanatomical biomarkers.   Schizophr Bull. 2015;41(2):471-482. doi:10.1093/schbul/sbu078 PubMedGoogle ScholarCrossref
    10.
    Das  T, Borgwardt  S, Hauke  DJ,  et al.  Disorganized gyrification network properties during the transition to psychosis.   JAMA Psychiatry. 2018;75(6):613-622. doi:10.1001/jamapsychiatry.2018.0391 PubMedGoogle ScholarCrossref
    11.
    Mechelli  A, Lin  A, Wood  S,  et al.  Using clinical information to make individualized prognostic predictions in people at ultra high risk for psychosis.   Schizophr Res. 2017;184:32-38. doi:10.1016/j.schres.2016.11.047 PubMedGoogle ScholarCrossref
    12.
    Clark  SR, Baune  BT, Schubert  KO,  et al.  Prediction of transition from ultra-high risk to first-episode psychosis using a probabilistic model combining history, clinical assessment and fatty-acid biomarkers.   Transl Psychiatry. 2016;6(9):e897. doi:10.1038/tp.2016.170 PubMedGoogle Scholar
    13.
    Koutsouleris  N, Kambeitz-Ilankovic  L, Ruhrmann  S,  et al; PRONIA Consortium.  Prediction models of functional outcomes for individuals in the clinical high-risk state for psychosis or with recent-onset depression: a multimodal, multisite machine learning analysis.   JAMA Psychiatry. 2018;75(11):1156-1172. doi:10.1001/jamapsychiatry.2018.2165 PubMedGoogle ScholarCrossref
    14.
    Fusar-Poli  P, Werbeloff  N, Rutigliano  G,  et al.  Transdiagnostic risk calculator for the automatic detection of individuals at risk and the prediction of psychosis: second replication in an independent national health service trust.   Schizophr Bull. 2019;45(3):562-570. doi:10.1093/schbul/sby070PubMedGoogle ScholarCrossref
    15.
    Carrión  RE, Cornblatt  BA, Burton  CZ,  et al.  Personalized prediction of psychosis: external validation of the NAPLS-2 psychosis risk calculator with the EDIPPP project.   Am J Psychiatry. 2016;173(10):989-996. doi:10.1176/appi.ajp.2016.15121565 PubMedGoogle ScholarCrossref
    16.
    Perkins  DO, Olde Loohuis  L, Barbee  J,  et al.  Polygenic risk score contribution to psychosis prediction in a target population of persons at clinical high risk.   Am J Psychiatry. 2020;177(2):155-163. doi:10.1176/appi.ajp.2019.18060721 PubMedGoogle ScholarCrossref
    17.
    Worthington  MA, Cao  H, Cannon  TD.  Discovery and validation of prediction algorithms for psychosis in youth at clinical high risk.   Biol Psychiatry Cogn Neurosci Neuroimaging. 2020;5(8):738-747. doi:10.1016/j.bpsc.2019.10.006PubMedGoogle Scholar
    18.
    Studerus  E, Ramyead  A, Riecher-Rössler  A.  Prediction of transition to psychosis in patients with a clinical high risk for psychosis: a systematic review of methodology and reporting.   Psychol Med. 2017;47(7):1163-1178. doi:10.1017/S0033291716003494 PubMedGoogle ScholarCrossref
    19.
    Sanfelici  R, Dwyer  DB, Antonucci  LA, Koutsouleris  N.  Individualized diagnostic and prognostic models for patients with psychosis risk syndromes: a meta-analytic view on the state-of-the-art.   Biol Psychiatry. 2020;88(4):349-360. doi:10.1016/j.biopsych.2020.02.009 PubMedGoogle ScholarCrossref
    20.
    Yarkoni  T, Westfall  J.  Choosing prediction over explanation in psychology: lessons from machine learning.   Perspect Psychol Sci. 2017;12(6):1100-1122. doi:10.1177/1745691617693393 PubMedGoogle ScholarCrossref
    21.
    Steyerberg  EW, Harrell  FE  Jr.  Prediction models need appropriate internal, internal-external, and external validation.   J Clin Epidemiol. 2016;69:245-247. doi:10.1016/j.jclinepi.2015.04.005 PubMedGoogle ScholarCrossref
    22.
    Fusar-Poli  P, Hijazi  Z, Stahl  D, Steyerberg  EW.  The science of prognosis in psychiatry: a review.   JAMA Psychiatry. 2018;75(12):1289-1297. doi:10.1001/jamapsychiatry.2018.2530 PubMedGoogle ScholarCrossref
    23.
    Lee  TY, Lee  J, Kim  M, Choe  E, Kwon  JS.  Can we predict psychosis outside the clinical high-risk state? a systematic review of non-psychotic risk syndromes for mental disorders.   Schizophr Bull. 2018;44(2):276-285. doi:10.1093/schbul/sbx173 PubMedGoogle ScholarCrossref
    24.
    McGorry  PD, Hartmann  JA, Spooner  R, Nelson  B.  Beyond the “at risk mental state” concept: transitioning to transdiagnostic psychiatry.   World Psychiatry. 2018;17(2):133-142. doi:10.1002/wps.20514 PubMedGoogle ScholarCrossref
    25.
    Fusar-Poli  P, Nelson  B, Valmaggia  L, Yung  AR, McGuire  PK.  Comorbid depressive and anxiety disorders in 509 individuals with an at-risk mental state: impact on psychopathology and transition to psychosis.   Schizophr Bull. 2014;40(1):120-131. doi:10.1093/schbul/sbs136 PubMedGoogle ScholarCrossref
    26.
    Koutsouleris  N, Meisenzahl  EM, Borgwardt  S,  et al.  Individualized differential diagnosis of schizophrenia and mood disorders using neuroanatomical biomarkers.   Brain. 2015;138(pt 7):2059-2073. doi:10.1093/brain/awv111 PubMedGoogle ScholarCrossref
    27.
    Häfner  H, Maurer  K, Trendler  G, an der Heiden  W, Schmidt  M, Könnecke  R.  Schizophrenia and depression: challenging the paradigm of two separate diseases—a controlled study of schizophrenia, depression and healthy controls.   Schizophr Res. 2005;77(1):11-24. doi:10.1016/j.schres.2005.01.004 PubMedGoogle ScholarCrossref
    28.
    Wigman  JTW, van Nierop  M, Vollebergh  WA,  et al.  Evidence that psychotic symptoms are prevalent in disorders of anxiety and depression, impacting on illness onset, risk, and severity—implications for diagnosis and ultra-high risk research.   Schizophr Bull. 2012;38(2):247-257. doi:10.1093/schbul/sbr196 PubMedGoogle ScholarCrossref
    29.
    Siebald  C, Khandaker  GM, Zammit  S, Lewis  G, Jones  PB.  Association between childhood psychiatric disorders and psychotic experiences in adolescence: a population-based longitudinal study.   Compr Psychiatry. 2016;69:45-52. doi:10.1016/j.comppsych.2016.05.004 PubMedGoogle ScholarCrossref
    30.
    Musliner  KL, Mortensen  PB, McGrath  JJ,  et al; Bipolar Disorder Working Group of the Psychiatric Genomics Consortium.  Association of polygenic liabilities for major depression, bipolar disorder, and schizophrenia with risk for depression in the Danish population.   JAMA Psychiatry. 2019;76(5):516-525. doi:10.1001/jamapsychiatry.2018.4166 PubMedGoogle ScholarCrossref
    31.
    Schmidt  A, Cappucciati  M, Radua  J,  et al.  Improving prognostic accuracy in subjects at clinical high risk for psychosis: systematic review of predictive models and meta-analytical sequential testing simulation.   Schizophr Bull. 2017;43(2):375-388. doi:10.1093/schbul/sbw098 PubMedGoogle Scholar
    32.
    Rajkomar  A, Dean  J, Kohane  I.  Machine learning in medicine.   N Engl J Med. 2019;380(14):1347-1358. doi:10.1056/NEJMra1814259 PubMedGoogle ScholarCrossref
    33.
    Wiener  N.  The Human Use of Human Beings. Houghton Mifflin; 1950.
    34.
    Clark  SR, Schubert  KO, Baune  BT.  Towards indicated prevention of psychosis: using probabilistic assessments of transition risk in psychosis prodrome.   J Neural Transm (Vienna). 2015;122(1):155-169. doi:10.1007/s00702-014-1325-9 PubMedGoogle ScholarCrossref
    35.
    Riecher-Rössler  A, Gschwandtner  U, Aston  J,  et al.  The Basel early-detection-of-psychosis (FEPSY)-study—design and preliminary results.   Acta Psychiatr Scand. 2007;115(2):114-125. doi:10.1111/j.1600-0447.2006.00854.x PubMedGoogle ScholarCrossref
    36.
    Theodoridou  A, Heekeren  K, Dvorsky  D,  et al.  Early recognition of high risk of bipolar disorder and psychosis: an overview of the ZInEP “early recognition” study.   Front Public Health. 2014;2:166. doi:10.3389/fpubh.2014.00166 PubMedGoogle ScholarCrossref
    37.
    Schimmelmann  BG, Michel  C, Martz-Irngartinger  A, Linder  C, Schultze-Lutter  F.  Age matters in the prevalence and clinical significance of ultra-high-risk for psychosis symptoms and criteria in the general population: findings from the BEAR and BEARS-Kid studies.   World Psychiatry. 2015;14(2):189-197. doi:10.1002/wps.20216 PubMedGoogle ScholarCrossref
    38.
    Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement.   Br J Surg. 2015;102(3):148-158. doi:10.1002/bjs.9736 PubMedGoogle ScholarCrossref
    39.
    McGlashan  T, Walsh  B, Woods  S.  The Psychosis-Risk Syndrome: Handbook for Diagnosis and Follow-up. Oxford University Press; 2010.
    40.
    Yung  AR, Nelson  B, Thompson  A, Wood  SJ.  The psychosis threshold in ultra high risk (prodromal) research: is it valid?   Schizophr Res. 2010;120(1-3):1-6. doi:10.1016/j.schres.2010.03.014 PubMedGoogle ScholarCrossref
    41.
    Dwyer  DB, Falkai  P, Koutsouleris  N.  Machine learning approaches for clinical psychology and psychiatry.   Annu Rev Clin Psychol. 2018;14:91-118. doi:10.1146/annurev-clinpsy-032816-045037 PubMedGoogle ScholarCrossref
    42.
    Schultze-Lutter  F, Addington  J, Ruhrmann  S, Klosterkötter  J.  Schizophrenia Proneness Instrument, Adult Version. Giovanni Fioriti Editore; 2007.
    43.
    Cornblatt  BA, Auther  AM, Niendam  T,  et al.  Preliminary findings for two new measures of social and role functioning in the prodromal phase of schizophrenia.   Schizophr Bull. 2007;33(3):688-702. doi:10.1093/schbul/sbm029 PubMedGoogle ScholarCrossref
    44.
    Pedersen  G, Hagtvet  KA, Karterud  S.  Generalizability studies of the Global Assessment of Functioning–Split version.   Compr Psychiatry. 2007;48(1):88-94. doi:10.1016/j.comppsych.2006.03.008 PubMedGoogle ScholarCrossref
    45.
    Bernstein  D, Fink  L.  CTQ: Childhood Trauma Questionnaire: A Retrospective Self-report. Psychological Corp; 1998.
    46.
    Redman  SL, Corcoran  CM, Kimhy  D, Malaspina  D.  Effects of early trauma on psychosis development in clinical high-risk individuals and stability of trauma assessment across studies: a review.   Arch Psychol (Chic). 2017;1(3):28.PubMedGoogle Scholar
    47.
    Koutsouleris  N, Davatzikos  C, Bottlender  R,  et al.  Early recognition and disease prediction in the at-risk mental states for psychosis using neurocognitive pattern classification.   Schizophr Bull. 2012;38(6):1200-1215. doi:10.1093/schbul/sbr037 PubMedGoogle ScholarCrossref
    48.
    Ripke  S, Neale  BM, Corin  A,  et al; Schizophrenia Working Group of the Psychiatric Genomics Consortium.  Biological insights from 108 schizophrenia-associated genetic loci.   Nature. 2014;511(7510):421-427. doi:10.1038/nature13595 PubMedGoogle ScholarCrossref
    49.
    Karrer  TM, Bassett  DS, Derntl  B,  et al.  Brain-based ranking of cognitive domains to predict schizophrenia.   Hum Brain Mapp. 2019;40(15):4487-4507. doi:10.1002/hbm.24716 PubMedGoogle ScholarCrossref
    50.
    Gennatas  ED, Friedman  JH, Ungar  LH,  et al.  Expert-augmented machine learning.   Proc Natl Acad Sci U S A. 2020;117(9):4571-4577. doi:10.1073/pnas.1906831117 PubMedGoogle ScholarCrossref
    51.
    Quade  D.  Using weighted rankings in the analysis of complete blocks with additive block effects.   J Am Stat Assoc. 1979;74(367):680-683. doi:10.1080/01621459.1979.10481670 Google ScholarCrossref
    52.
    Heckert  NA, Filliben  JJ. Dataplot Reference Manual, Volume 2: LET Subcommands and Library Functions. Published 2003. Accessed November 2, 2020. https://www.itl.nist.gov/div898/handbook/eda/section3/eda3672.htm
    53.
    Gómez-Verdejo  V, Parrado-Hernández  E, Tohka  J; Alzheimer’s Disease Neuroimaging Initiative.  Sign-consistency based variable importance for machine learning in brain imaging.   Neuroinformatics. 2019;17(4):593-609. doi:10.1007/s12021-019-9415-3 PubMedGoogle ScholarCrossref
    54.
    Koutsouleris  N, Borgwardt  S, Meisenzahl  EM, Bottlender  R, Möller  HJ, Riecher-Rössler  A.  Disease prediction in the at-risk mental state for psychosis using neuroanatomical biomarkers: results from the FePsy study.   Schizophr Bull. 2012;38(6):1234-1246. doi:10.1093/schbul/sbr145 PubMedGoogle ScholarCrossref
    55.
    Madras  D, Pitassi  T, Zemel  R.  Predict responsibly: improving fairness and accuracy by learning to defer.  In: Bengio S, Wallach H, Larochelle H, Grauman K, CesaBianchi N, Garnett R, eds.  Advances in Neural Information Processing Systems. Volume 31. Curran Associates Inc; 2018:6147–6157.Google Scholar
    56.
    Zavos  HMS, Eley  TC, McGuire  P,  et al.  Shared etiology of psychotic experiences and depressive symptoms in adolescence: a longitudinal twin study.   Schizophr Bull. 2016;42(5):1197-1206. doi:10.1093/schbul/sbw021 PubMedGoogle ScholarCrossref
    57.
    Callaway  DA, Perkins  DO, Woods  SW, Liu  L, Addington  J.  Movement abnormalities predict transitioning to psychosis in individuals at clinical high risk for psychosis.   Schizophr Res. 2014;159(2-3):263-266. doi:10.1016/j.schres.2014.09.031 PubMedGoogle ScholarCrossref
    58.
    Green  MF, Horan  WP, Lee  J.  Nonsocial and social cognition in schizophrenia: current evidence and future directions.   World Psychiatry. 2019;18(2):146-161. doi:10.1002/wps.20624 PubMedGoogle ScholarCrossref
    59.
    Thompson  A, Papas  A, Bartholomeusz  C,  et al.  Social cognition in clinical “at risk” for psychosis and first episode psychosis populations.   Schizophr Res. 2012;141(2-3):204-209. doi:10.1016/j.schres.2012.08.007 PubMedGoogle ScholarCrossref
    60.
    Barbato  M, Liu  L, Cadenhead  KS,  et al.  Theory of mind, emotion recognition and social perception in individuals at clinical high risk for psychosis: findings from the NAPLS-2 cohort.   Schizophr Res Cogn. 2015;2(3):133-139. doi:10.1016/j.scog.2015.04.004 PubMedGoogle ScholarCrossref
    61.
    Schiffman  J.  Motor issues in the clinical high risk phase of psychosis.   Schizophr Bull. 2017;43(5):937-938. doi:10.1093/schbul/sbx086 PubMedGoogle ScholarCrossref
    62.
    Bora  E, Berk  M.  Theory of mind in major depressive disorder: a meta-analysis.   J Affect Disord. 2016;191:49-55. doi:10.1016/j.jad.2015.11.023 PubMedGoogle ScholarCrossref
    63.
    Lindgren  M, Jonninen  M, Jokela  M, Therman  S.  Adolescent psychosis risk symptoms predicting persistent psychiatric service use: a 7-year follow-up study.   Eur Psychiatry. 2019;55:102-108. doi:10.1016/j.eurpsy.2018.10.004 PubMedGoogle ScholarCrossref
    64.
    Beck  K, Studerus  E, Andreou  C,  et al.  Clinical and functional ultra-long-term outcome of patients with a clinical high risk (CHR) for psychosis.   Eur Psychiatry. 2019;62:30-37. doi:10.1016/j.eurpsy.2019.08.005 PubMedGoogle ScholarCrossref
    65.
    Woods  SW, Powers  AR  III, Taylor  JH,  et al.  Lack of diagnostic pluripotentiality in patients at clinical high risk for psychosis: specificity of comorbidity persistence and search for pluripotential subgroups.   Schizophr Bull. 2018;44(2):254-263. doi:10.1093/schbul/sbx138 PubMedGoogle ScholarCrossref
    66.
    Jonas  KG, Lencz  T, Li  K,  et al.  Schizophrenia polygenic risk score and 20-year course of illness in psychotic disorders.   Transl Psychiatry. 2019;9(1):300. doi:10.1038/s41398-019-0612-5 PubMedGoogle ScholarCrossref
    67.
    Hess  JL, Tylee  DS, Mattheisen  M,  et al; Schizophrenia Working Group of the Psychiatric Genomics Consortium; Lundbeck Foundation Initiative for Integrative Psychiatric Research (iPSYCH).  A polygenic resilience score moderates the genetic risk for schizophrenia.  Published online September 6, 2019.  Mol Psychiatry. doi:10.1038/s41380-019-0463-8 PubMedGoogle Scholar
    68.
    Takahashi  T, Suzuki  M.  Brain morphologic changes in early stages of psychosis: implications for clinical application and early intervention.   Psychiatry Clin Neurosci. 2018;72(8):556-571. doi:10.1111/pcn.12670 PubMedGoogle ScholarCrossref
    69.
    Sasabayashi  D, Takayanagi  Y, Takahashi  T,  et al.  Increased occipital gyrification and development of psychotic disorders in individuals with an at-risk mental state: a multicenter study.   Biol Psychiatry. 2017;82(10):737-745. doi:10.1016/j.biopsych.2017.05.018 PubMedGoogle ScholarCrossref
    70.
    Okada  N, Yahata  N, Koshiyama  D,  et al.  Abnormal asymmetries in subcortical brain volume in early adolescents with subclinical psychotic experiences.   Transl Psychiatry. 2018;8(1):254. doi:10.1038/s41398-018-0312-6 PubMedGoogle ScholarCrossref
    71.
    de Wit  S, Wierenga  LM, Oranje  B,  et al.  Brain development in adolescents at ultra-high risk for psychosis: longitudinal changes related to resilience.   Neuroimage Clin. 2016;12:542-549. doi:10.1016/j.nicl.2016.08.013 PubMedGoogle ScholarCrossref
    72.
    Escudero  J, Ifeachor  E, Zajicek  JP, Green  C, Shearer  J, Pearson  S; Alzheimer’s Disease Neuroimaging Initiative.  Machine learning-based method for personalized and cost-effective detection of Alzheimer’s disease.   IEEE Trans Biomed Eng. 2013;60(1):164-168. doi:10.1109/TBME.2012.2212278 PubMedGoogle ScholarCrossref
    73.
    Schultze-Lutter  F, Schmidt  SJ, Theodoridou  A.  Psychopathology: a precision tool in need of re-sharpening.   Front Psychiatry. 2018;9:446. doi:10.3389/fpsyt.2018.00446 PubMedGoogle ScholarCrossref
    74.
    Varma  S, Simon  R.  Bias in error estimation when using cross-validation for model selection.   BMC Bioinformatics. 2006;7:91. doi:10.1186/1471-2105-7-91 PubMedGoogle ScholarCrossref
    75.
    Brady  LS, Potter  WZ, Gordon  JA.  Redirecting the revolution: new developments in drug development for psychiatry.   Expert Opin Drug Discov. 2019;14(12):1213-1219. doi:10.1080/17460441.2019.1666102 PubMedGoogle ScholarCrossref
    ×