The reliability of predictive pattern elements was evaluated using cross-validation ratio mapping (A). In addition, the significance of predictive features used by the clinical-neurocognitive model was assessed by means of sign-based consistency mapping (B). Both visualization methods are detailed in the eMethods in the Supplement. CTQ indicates Childhood Trauma Questionnaire; DANVA, Diagnostic Analysis of Non-Verbal Accuracy; DSST, Digit-Symbol Substitution Test; FDR, false discovery rate; PVF, Phonetic Verbal Fluency; ROCF, Rey-Osterreith Figure; SIPS, Structured Interview for Psychosis-Risk Syndromes; SOPT, Self-Ordered Pointing Task; SVF, Semantic Verbal Fluency; TMT, Trail Making Test; and WAIS, Wechsler Adult Intelligence Scale.
The reliability of predictive pattern elements was evaluated using cross-validation ratio (CVR) mapping (A). In addition, the significance of predictive features used by the PRS-based model was assessed by means of sign-based consistency mapping (B) (as described in the eMethods in the Supplement). The cybernetic model combines all algorithmic and human components (C). FDR indicates false discovery rate; and sMRI, structural magnetic resonance imaging.
Cohorts include patients with follow-up of 18 months or longer (PRONIA plus 18M), the complete PRONIA cohort, and the Zurich Early Recognition Program (ZInEP). Data points indicate median. The Quade test51 was used to compare the models’ median balanced accuracy (BAC) computed across the cross-validation cycle (CV2) test data partitions. The BAC measures obtained for the ZInEP cohort (C) were produced by applying the condensed clinical-neurocognitive (Clin-NC), structural magnetic resonance imaging (sMRI)–based, and respective stacked risk calculators of the complete PRONIA sample (B) to this external sample (eFigure 2 in the Supplement). Post hoc comparisons were performed using the t distribution approximation described by Heckert and Filliben.52 P values were corrected for multiple comparisons using the false discovery rate (FDR). The upper graphs represent the median BAC for each risk calculator in analyses A, B, and C along with the lower and upper quartiles of the BAC distributions (whiskers of the error bars). The lower figures show the logarithmized, FDR-corrected P matrix for the pairwise post hoc classifier comparisons. For an in-depth analysis of the prognostic sequence included in the risk classifier comparison, see eFigures 14 and 15 in the Supplement. The cybernetic risk calculator analyzed the combined predictions of raters, Clin-NC, polygenic risk score (PRS)–based, and sMRI-based risk calculators; the stacked risk calculator, the combined predictions of Clin-NC, PRS-based, and sMRI-based risk calculators.
aIndicates risk calculator encompassing the condensed Clin-NCs and sMRI-based models and specifically trained to externally validate the effect of stacking on prognostic performance in the ZInEP cohort.
eMethods. Participants and Analysis
eTable 1. Study Inclusion/Exclusion Criteria of the Study
eTable 2. Antipsychotic Medication Thresholds Based on the Previous Version of the S3 Guidelines of the German Association for Psychiatry, Psychotherapy and Psychosomatics
eTable 3. Group-Level Comparison of CHR, ROD and HC Individuals
eTable 4. Diagnostic Breakdown of Psychotic Disorders in PRONIA Cases With a Disease Transition
eTable 5. Sociodemographic, Clinical and Neurocognitive Variables Used in the Clinical Prediction Models
eTable 6. MR Scanner Systems and Structural MRI Sequence Parameters Used at the Respective PRONIA Sites
eTable 7. Effects of Baseline Treatments on the Decision Scores Generated by 5 Different Risk Calculators
eTable 8. Correlations Between Decision Scores of Unimodal and Cybernetic Risk Calculators and Patients’ Maximum Follow-up Intervals and Number of Follow-up Examinations
eTable 9. Analysis of the Site-Related Variation in Follow-up Duration, Time to Transition, Age and Sex Distributions in the PRONIA Cohort
eTable 10. Effects of the Factor ‘Site’ on Predictive Performance of Raters, Unimodal, Stacked, and Cybernetic Risk Calculators
eTable 11. Heterogeneity of Raters’ and Models’ Predictive Performance
eTable 12. Differential Diagnostic Performance of Classification Models Trained to Separate Between CHR and ROD Patients Using Clinical-Neurocognitive, PRS-Based, and sMRI-Based Data Domain
eTable 13. Explained Variances of Pairwise Prognostic, Diagnostic, and Prognostic-Diagnostic Classifier Combinations
eTable 14. ROD Depletion and Substitution Analyses Assessing the Performance Effects Induced by the ROD Group in the Prediction of Psychosis Transitions in the CHR Sample
eTable 15. Discriminative Performance of Raters and Risk Calculators in Distinguishing Between Transition Cases, Cases With Nonremitting/De novo CHR States and Cases Developing Asymptomatic CHR Trajectories
eTable 16. Comparison of Nonpsychotic Diagnostic Outcomes Between Patients With a Predicted Transition vs Predicted Nontransition to Psychosis During the Follow-up Period of the Study
eTable 17. Prognostic Sequences Tested in the Sequence Optimization Algorithm
eTable 18. Study-Related, Sociodemographic, Physical, Functional, and Clinical Differences in the ZInEP Cohort
eTable 19. Study-Related, Sociodemographic, Physical, Functional, and Clinical Differences in the BEARS-Kid Cohort
eTable 20. Performance Gains Produced by the Stacking of the Clinical-Neurocognitive and PRS-Based Models
eFigure 1. CONSORT Chart and Follow-up Protocol of the PRONIA Study for the Clinical Participants
eFigure 2. Schematic Analysis Workflow of the Study
eFigure 3. Experimental Design of the Machine Learning Pipelines Used to Train and Cross-validate the Unimodal and Stacked Risk Calculators
eFigure 4. Schematic Representation of the NeuroMiner Model Optimization Process Used to Train the Structural MRI Predictors
eFigure 5. Predictive Signature Underlying the sMRI-Based Risk Calculator
eFigure 6. Comparison of Predictive Performance of Expert-Based, Unimodal, Stacked, Cybernetic and Sequentially Stacked Risk Calculators Trained and Cross-validated Using the PRONIA-18M and Complete PRONIA Cohorts
eFigure 7. Comparison of Standardized Clinical-Neurocognitive Variables Between Healthy Volunteers and CHR/ROD Patients Who Were Labeled With a Transition or Nontransition to Psychosis by the Clinical-Neurocognitive Risk Calculator
eFigure 8. Comparison of Polygenic Risk Scores (PRS) Between Healthy Volunteers and CHR/ROD Patients Who Were Labeled With a Transition or Nontransition to Psychosis by the PRS-Based Risk Calculator
eFigure 9. Univariate Volumetric Comparisons Between Healthy Volunteers and Patients Labeled With a Transition or Nontransition to Psychosis by the sMRI-Based Risk Calculator
eFigure 10. Image Quality Assessment of T1-Weighted Images Analyzed in the Study Using the Quality Assessment Tools of the CAT12 Toolbox
eFigure 11. Interaction Analysis Assessing the Effects of the Number of Examinations Available and the Longest Interval Duration on the Predictions of Four Different Risk Calculators
eFigure 12. Statistical Analysis of Prognostic Performance Effects Induced by the ROD Depletion and Substitution Strategies in the Three Unimodal Risk Calculators and the Stacked Model
eFigure 13. Prognostic Stratification Effects of Raters’ Predictions, Unimodal and Cybernetic Risk Calculators on the Trajectories of CHR Syndromes and Functioning as Determined by Nonnegative Matrix Factorization and Linear Mixed-Effects Modeling
eFigure 14. Detailed Analysis of the Optimal Sequential Prediction Algorithm
eFigure 15. Regularization of the Prognostic Workflow for DIFFERENT LEVELS of Examination Sparsity and Analysis of Effects of Regularization on Prognostic Performance
eFigure 16. Comparison of Clinical-Neurocognitive Signatures Used by the Prognostic Risk Calculator and the Differential Diagnostic Classifier (CHR vs ROD)
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Koutsouleris N, Dwyer DB, Degenhardt F, et al. Multimodal Machine Learning Workflows for Prediction of Psychosis in Patients With Clinical High-Risk Syndromes and Recent-Onset Depression. JAMA Psychiatry. 2021;78(2):195–209. doi:10.1001/jamapsychiatry.2020.3604
Can a transition to psychosis be predicted in patients with clinical high-risk states or recent-onset depression by optimally integrating clinical, neurocognitive, neuroimaging, and genetic information with clinicians’ prognostic estimates?
In this prognostic study of 334 patients and 334 control individuals, machine learning models sequentially combining clinical and biological data with clinicians’ estimates correctly predicted disease transitions in 85.9% of cases across geographically distinct patient populations. The clinicians’ lack of prognostic sensitivity, as measured by a false-negative rate of 38.5%, was reduced to 15.4% by the sequential prognostic model.
These findings suggest that an individualized prognostic workflow integrating artificial and human intelligence may facilitate the personalized prevention of psychosis in young patients with clinical high-risk syndromes or recent-onset depression.
Diverse models have been developed to predict psychosis in patients with clinical high-risk (CHR) states. Whether prediction can be improved by efficiently combining clinical and biological models and by broadening the risk spectrum to young patients with depressive syndromes remains unclear.
To evaluate whether psychosis transition can be predicted in patients with CHR or recent-onset depression (ROD) using multimodal machine learning that optimally integrates clinical and neurocognitive data, structural magnetic resonance imaging (sMRI), and polygenic risk scores (PRS) for schizophrenia; to assess models’ geographic generalizability; to test and integrate clinicians’ predictions; and to maximize clinical utility by building a sequential prognostic system.
Design, Setting, and Participants
This multisite, longitudinal prognostic study performed in 7 academic early recognition services in 5 European countries followed up patients with CHR syndromes or ROD and healthy volunteers. The referred sample of 167 patients with CHR syndromes and 167 with ROD was recruited from February 1, 2014, to May 31, 2017, of whom 26 (23 with CHR syndromes and 3 with ROD) developed psychosis. Patients with 18-month follow-up (n = 246) were used for model training and leave-one-site-out cross-validation. The remaining 88 patients with nontransition served as the validation of model specificity. Three hundred thirty-four healthy volunteers provided a normative sample for prognostic signature evaluation. Three independent Swiss projects contributed a further 45 cases with psychosis transition and 600 with nontransition for the external validation of clinical-neurocognitive, sMRI-based, and combined models. Data were analyzed from January 1, 2019, to March 31, 2020.
Main Outcomes and Measures
Accuracy and generalizability of prognostic systems.
A total of 668 individuals (334 patients and 334 controls) were included in the analysis (mean [SD] age, 25.1 [5.8] years; 354 [53.0%] female and 314 [47.0%] male). Clinicians attained a balanced accuracy of 73.2% by effectively ruling out (specificity, 84.9%) but ineffectively ruling in (sensitivity, 61.5%) psychosis transition. In contrast, algorithms showed high sensitivity (76.0%-88.0%) but low specificity (53.5%-66.8%). A cybernetic risk calculator combining all algorithmic and human components predicted psychosis with a balanced accuracy of 85.5% (sensitivity, 84.6%; specificity, 86.4%). In comparison, an optimal prognostic workflow produced a balanced accuracy of 85.9% (sensitivity, 84.6%; specificity, 87.3%) at a much lower diagnostic burden by sequentially integrating clinical-neurocognitive, expert-based, PRS-based, and sMRI-based risk estimates as needed for the given patient. Findings were supported by good external validation results.
Conclusions and Relevance
These findings suggest that psychosis transition can be predicted in a broader risk spectrum by sequentially integrating algorithms’ and clinicians’ risk estimates. For clinical translation, the proposed workflow should undergo large-scale international validation.
The clinical high-risk (CHR) criteria for psychosis have been established to detect vulnerable individuals as early as possible to intercept disease development.1 These criteria identify a patient population with increased incidence compared with the general population,2 yet only 22% of patients with CHR as detected by ultra–high-risk criteria show a psychosis transition during a 3-year period.2 The clinical utility of the CHR designation may be further limited because its ascertainment is laborious and confined to specialized, well-equipped health care services that do not sufficiently cover the vulnerable population.3,4 Hence, improved prognostic accuracy and clinical scalability are needed to accurately identify patients truly at risk for psychosis.
Prognostic accuracy may be increased using psychosis risk calculators for populations with CHR based on conventional statistics3,5-7 or machine learning,8-13 with studies finding that a first episode can be predicted with clinical data,3,14 combinations of clinical and cognitive data,6,15 neuroimaging,8 and, recently, with polygenic risk scores (PRS) for schizophrenia,16 among other measures.17 However, reviews18-20 have also highlighted methodological shortcomings, such as inadequate sample sizes and model validation strategies,21,22 that may have inflated accuracy. Moreover, studies suggested that psychosis does not only emerge from CHR states23,24 but occurs and can be predicted across a broader spectrum of comorbid conditions commencing in late adolescence and early adulthood.3,14 Hence, generalizable risk prediction models may require transdiagnostic discovery and validation populations, encompassing patients with CHR states and early-onset affective syndromes that share environmental, clinical, and neurobiological features.25-30 In addition, the growing diversity of risk prediction models originating from different data modalities has led to uncertainty about the minimum number of modalities needed to increase prognostic accuracy to a level justifying clinical implementation.2,31 Finally, algorithms should be compared and integrated with clinicians’ predictions of psychosis transition to determine their potential utility from public health and service provision perspectives.13,32
Addressing these challenges, the European Union–funded PRONIA study (Personalised Prognostic Tools for Early Psychosis Management [https://www.pronia.eu]) collected multimodal longitudinal data from adolescents and young adults in CHR states, those with recent-onset depression (ROD), and healthy control individuals. We evaluated clinical, neuroanatomical, and genetic machine learning models trained to identify patients with CHR syndromes and ROD who undergo psychosis transition. We compared our models’ performance with our clinical raters’ ability to predict psychosis transition and explored whether the sequential integration of algorithmic and expert-based prognoses produces clinically efficient cybernetic workflows,33,34 that is, structured interactions between humans and machines that maximize prognostic accuracy while minimizing the examination burden in the given patient. We assessed potential confounders and moderators of prognostic performance and tested whether our models’ and raters’ estimates predicted not only psychosis transition but also distinct CHR syndromes and nonpsychotic disease trajectories. Finally, we explored whether a considerably condensed, and hence less burdensome, clinical model could be generalized to 3 independent patient cohorts with CHR syndromes and other mental conditions.35-37 This external validation step also benchmarked the neuroanatomical and combined models derived from the PRONIA cohort.
The eMethods section in the Supplement details all of the methods for this prognostic study, which followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guideline.38 In summary, our analysis included 334 patients with CHR states (n = 167) or ROD (n = 167) recruited across 7 sites in Finland, Germany, Italy, Switzerland, and the United Kingdom from February 1, 2014, to May 31, 2017, using standardized inclusion and exclusion criteria (eTables 1 and 2 and CONSORT diagram in eFigure 1 in the Supplement). Follow-up for all patients ranged from 9 to 36 months,13 with visits every 3 months to the 18-month point and every 9 months thereafter. Furthermore, 334 healthy controls matched for age, sex, and site were included to evaluate prognostic patterns. Adult participants gave informed consent before study inclusion. Participants younger than 18 years and their guardians provided their written informed assent and consent. The PRONIA observational study was registered at the German Clinical Trials Register (DRKS00005042) and approved by all local research ethics committees.
Sociodemographic and clinical variables were compared between diagnostic groups (eTable 3 in the Supplement), patients with psychosis transition and nontransition, and patients with 18-month or later follow-up data (discovery sample [n = 246]; PRONIA plus 18M) or earlier attrition (validation sample [n = 88]; PRONIA minus 18M) (Table 1). Psychosis transition was defined when at least 1 of the 5 positive symptom items in the Structured Interview for Psychosis–Risk Syndromes39 reached psychotic intensity daily for at least 7 days.40 Diagnoses of cases with psychosis transition are listed in eTable 4 in the Supplement.
Data were analyzed from January 1, 2019, to March 31, 2020. Using the machine learning software NeuroMiner, version 1.05 (GitHub [https://github.com/neurominer-git/NeuroMiner-1]), we constructed and tested unimodal, multimodal, and clinically scalable sequential risk calculators for transition prediction in the PRONIA plus 18M cohort using leave-one-site-out cross-validation (LOSOCV)21,41 (eMethods and eFigures 2-4 in the Supplement). We evaluated the risk calculators using baseline and longitudinal data and validated their specificity in the PRONIA minus 18M sample, which did not include cases with psychosis transition but provided all data modalities. In addition, 3 external data sets consisting of cases with psychosis transition and nontransition were available to test selected models.35-37
First, unimodal risk calculators were trained with literature-based baseline predictors of psychosis transition,2,6,16 including prodromal symptoms,39,42 functioning,43,44 childhood adversity,45,46 and neurocognitive measures6,47 in the clinical-neurocognitive domain (eTable 5 in the Supplement); PRS for schizophrenia48 in the genetic domain16; and gray matter volume maps in the structural magnetic resonance imaging (sMRI) domain8,9 (eTable 6 in the Supplement). In addition, we evaluated our raters’ predictions, which, at the conclusion of baseline assessments, were provided as yes or no replies to the question, “Do you think the patient will likely transition to psychosis?” Then we assessed whether combining unimodal algorithms using stacked generalization49 improved prognostic accuracy (eFigures 2 and 3 in the Supplement and Table 2).13 Following the concept of expert-based machine learning,50 we integrated our raters’ estimates as additional predictors to produce a cybernetic model33 (eFigure 2 in the Supplement). Models’ predictive signatures were visualized in Figures 1 and 2 and eFigure 5 in the Supplement using measures of pattern element stability (cross-validation ratio) and pattern element significance (sign-based consistency; eMethods in the Supplement). In addition, the prognostic models were assessed using random-label permutations (Table 2). Raters’ and models’ performances were compared statistically at the omnibus level using the Quade test,51 an extension of the nonparametric Wilcoxon signed rank test, followed by post hoc pairwise mean differences tests using the t distribution.52 Statistical significance was determined at α = .05. Obtained P values were 2-sided; P values computed in the pairwise classifier comparisons were corrected using the false discovery rate (FDR) (Figure 3). Classifiers were visually compared in eFigure 6 in the Supplement).51,52
We tested differences between the prognostic assignment groups and the matched healthy control group to determine whether the models’ predictive patterns represented a deviation from normality (eFigures 7-9 in the Supplement). Potential model confounders and moderators were systematically tested, including image quality (eFigure 10 in the Supplement), treatments (eTable 7 in the Supplement), follow-up frequency and duration (eFigure 11 and eTable 8 in the Supplement), site-related effects (eTables 9-11 in the Supplement), baseline study group membership (eTables 12 and 13 in the Supplement), and, specifically, the inclusion of patients with ROD (eTable 14 and eFigure 12 in the Supplement). Using the patients’ longitudinal data, we evaluated whether model predictions were not specific for the binary transition vs nontransition outcome, but we also separated transitions from nonremitting/de novo CHR symptom courses (P-CHR) and asymptomatic/nonpersisting trajectories (NP-CHR) (eTable 15 in the Supplement). To further explore a prognostic generalization effect,13 we used unsupervised machine learning (eMethods in the Supplement) to construct fine-grained CHR syndrome trajectories (eFigure 13A in the Supplement) and used linear mixed-effects modeling to compare trajectories between predicted and observed outcome groups (eFigure 13B [part 1] in the Supplement). Finally, we investigated whether assignments generalized to the prediction of nonpsychotic outcomes (eTable 16 in the Supplement).
To facilitate clinical implementation, we developed a sequential prediction method that optimizes the ordering and number of data modalities as well as the prognostic uncertainty thresholds to decide whether a patient needs further testing (eTable 17 and eMethods in the Supplement). We analyzed the identified optimal prognostic workflow (eFigure 14 in the Supplement) and tested whether it achieved similar performance as the fully stacked models at lower diagnostic burden for the patients. To further enhance clinical scalability, we condensed the clinical-neurocognitive model, which was the workflow’s entry point, from 141 to 7 (5.0%) variables (Figure 2) using sign-based consistency mapping (eMethods in the Supplement).53 We tested the condensed model and the respective workflow’s specificity in the PRONIA minus 18M sample (Table 2). Finally, we explored whether the prognostic sequence could be further trimmed using diagnostic parsimony regularization (eMethods in the Supplement and Table 2). Nonregularized and regularized workflows including the full or condensed clinical-neurocognitive models were compared in eFigure 15 in the Supplement.
We validated the condensed clinical-neurocognitive model in 2 external cohorts: 146 patients with CHR (aged 15-35 years; 16 [11.0%] transitions) provided by the Zurich Early Recognition Program (ZInEP) (eTable 18 in the Supplement)36 and 462 patients with diverse mental conditions (aged 8-17 years; 13 [2.8%] transitions) drawn from the Bi-national Evaluation of At-Risk Symptoms in Children and Adolescents study (eTable 19 in the Supplement).37 Second, we validated the sMRI-based model in ZInEP and in 37 patients with CHR (16 [43.2%] transitions) from the Früherkennung von Psychosen study (Table 2).35,54 To validate the increased performance of multimodal risk calculators, we trained a stacked model using the condensed clinical-neurocognitive and sMRI-based models, tested it in the ZInEP data (Table 2), and used the Quade test51 to compare the 2 unimodal prediction models with the stacked classifier (Figure 3). Finally, we made our models available in the NeuroMiner Model Library (http://www.proniapredictors.eu) to facilitate their independent external validation.
A total of 668 patients and controls were included in the analysis (mean [SD] age, 25.1 [5.8] years; 354 [53.0%] female and 314 [47.0%] male). Patients in the PRONIA plus 18M and PRONIA minus 18M groups were followed up for a mean (SD) of 842.7 (272.3) and 390.6 (99.6) days, respectively (Table 1). They did not differ in any examined variable (Table 1). Psychosis transition occurred after a mean (SD) of 246.9 (244.5) days in 26 cases and developed into schizophrenia in 8 (30.8%) (eTable 4 in the Supplement). Follow-up durations differed between sites but not time to psychosis transition (eTable 9 in the Supplement). Compared with nontransition, individuals with psychosis transition had more repeated school years (mean [SD], 0.67 [0.88] vs 0.26 [0.61] years) and more prevalent attenuated positive symptoms (APSs) at baseline (18 of 26 [69.2%] vs 88 of 308 [28.6%]) (Table 1). Of 167 patients with CHR, 23 (13.8%) developed psychosis, whereas 53 (31.7%) had remitted from CHR criteria at the 9-month visit. Major depression affected 103 patients with CHR syndromes (61.7%) but was not differentially associated with psychosis transition (Table 1). Nonremitting mood and anxiety disorders were present during follow-up in cases with psychosis transition and nontransition (eTable 16 in the Supplement). Compared with healthy controls, ROD was associated with low but significantly elevated CHR symptom scores (eg, mean [SD] Structured Interview for Psychosis–Risk Syndromes positive symptoms, 0.43 [0.44] vs 0.10 [0.21]; P < .001; mean [SD] Schizophrenia Proneness Instrument: Cognitive Disturbances symptoms, 0.24 [0.32] vs 0.02 [0.08]; P < .001) (eTable 3 in the Supplement). Functional-cognitive and interpersonal abnormalities were comparable between ROD and CHR groups. Of 167 patients with ROD, 32 (19.2%) developed psychosis-related outcomes, including CHR states in 29 (17.4%) and psychosis transitions in 3 (1.8%).
The full clinical-neurocognitive model predicted psychosis transition with a balanced accuracy (BAC-LOSOCV) of 75.7% (sensitivity, 84.6%; specificity, 66.8%; P < .001) (Table 2). Significant predictors as determined by sign-based consistency mapping (z>3.28; P < .05 for FDR) were APS and motor disturbances, a nonsupportive family environment during childhood, and reduced facial emotion recognition (Figure 1B). Compared with healthy controls, those assigned to psychosis transition had elevated abnormality scores in these variables, whereas those with nontransition assignments showed an abnormality pattern focused on unusual thought content, suspiciousness, perceptual abnormalities, and childhood adversity, with higher visual working memory and semantic verbal fluency performance (P < .05 for FDR) (eFigure 7 in the Supplement).
The PRS-based model achieved a BAC-LOSOCV of 66.1% (sensitivity, 76.0%; specificity, 56.2%; P < .001). Among the 10 tested genome-wide significance thresholds, only P = 1.0 reached significance (Figure 2). Compared with healthy controls, those with psychosis transition assignments had elevated PRS across all whole-genome P thresholds, whereas those with nontransition assignments expressed reduced PRS at P ≥ 5.7 × 10−4 (eFigure 8A in the Supplement). Patients with observed nontransition and those with ROD did not show reduced PRS (eFigure 8B-C in the Supplement).
The sMRI-based model attained a BAC-LOSOCV of 70.7% (sensitivity, 88.0%; specificity, 53.5%; P < .001). At a stability threshold (cross-validation ratio) of at least 3, the brain pattern predicting psychosis transition involved reduced gray matter volume in the superior temporal, supramarginal, angular, orbitofrontal, inferior frontal, dorsomedial prefrontal, and occipital cortices. The predictive pattern also included areas of increased gray matter volume covering the dorsolateral prefrontal, precuneal, insular, hippocampal, and cerebellar brain regions (eFigure 5 in the Supplement). This brain signature differentiated psychosis transition-assigned patients from healthy controls, whereas nontransition-assigned patients showed a partial pattern inversion with increased temporo-occipital gray matter volume compared with healthy controls (see threshold-free cluster enhancement statistics thresholded at P < .05 for FDR) (eFigure 9 in the Supplement).
Clinical raters achieved a BAC of 73.2% (sensitivity, 61.5%; specificity, 84.9%), which was independent of the length of their early recognition experience (mean [SD] length for correct predictions: 31.8 [46.8] months; mean [SD] length for wrong predictions, 29.4 [38.1] months; unpaired 2-tailed t326 = 0.35; P = .72). The stacked model combining unimodal algorithms produced a BAC-LOSOCV of 82.9% (sensitivity, 80.8%; specificity, 85.0%). Integration of raters’ prognoses into the stacked model increased BAC-LOSOCV to 85.5% (sensitivity, 84.6%; specificity, 86.4%), and they were the cybernetic model’s most relevant predictor (Figure 2C).
Image quality (eFigure 10 in the Supplement), baseline treatments or previous hospitalizations (eTable 7 in the Supplement), follow-up duration and frequency (eTable 8 and eFigure 11 in the Supplement), and site effects (eTable 10 in the Supplement) did not influence model performance. Study group (CHR vs ROD) could be classified with a BAC-LOSOCV of 82.3% (sensitivity, 73.7%; specificity, 91.0%) using clinical-neurocognitive data and with a BAC of 55.7% (sensitivity, 49.7%; specificity, 61.7%) using PRS (eTable 12 in the Supplement). These diagnostic classifiers explained 49.7% and 18.3%, respectively, of the variance of the respective prognostic counterparts (both P < .001 for FDR) (eTable 13 in the Supplement). Raters’ prognoses were also significantly informed by baseline study group (BAC, 62.7%; sensitivity, 30.9%; specificity, 94.5%) (eTable 12 in the Supplement). The removal of the patients with ROD from the training samples or their substitution with healthy controls significantly reduced the balanced accuracy of all risk calculators by −2.8% to −11.7% in the CHR group (eTable 14 and eFigure 12 in the Supplement).
Prognostic assignments also delineated psychosis transition and P-CHR and NP-CHR courses irrespective of model type (eTable 15 in the Supplement). The separability of P-CHR from NP-CHR courses was lower and only significant for clinical-neurocognitive models and raters. The nonnegative matrix factorization and linear mixed-model analysis showed that CHR syndrome trajectories were stratified by the predictions of the clinical-neurocognitive classifier (factor F1 paranoid-perceptual disturbances, F1,832 = 136.35 [P < .001 for FDR]; factor F2 disturbances of volition and affect, F1,832 = 12.76 [P = .001 for FDR]; factor F3 functional disturbances, F1,832 = 24.34 [P < .001 for FDR]; factor F4 cognitive disturbances, F1,832 = 44.05 [P = .007 for FDR]) as well as by raters' outcome estimates (factor F1 paranoid-perceptual disturbances, F1,825 = 30.64 [P < .001 for FDR]; factor F2 disturbances of volition and affect, F1,825 = 6.15 [P = .03 for FDR]; factor F3 functional disturbances, F1,825 = 5.80 [P < .03 for FDR]; factor F4 cognitive disturbances, F1,825 = 9.00 [P = .007 for FDR]) (eFigure 13 in the Supplement). Nonpsychotic disease courses were not associated with prognostic assignments (eTable 16 in the Supplement).
We identified a prognostic sequence that produced a BAC-LOSOCV of 85.9% (sensitivity, 84.6%; specificity, 87.3%) (Table 2) and started with the clinical-neurocognitive model, added raters, and finally integrated PRS- and sMRI-based models (eFigure 14 in the Supplement). Across this sequence, the positive likelihood ratio increased to 6.6, whereas the population requiring all prognostic assessments decreased to 41.1%. Regularization for diagnostic parsimony further reduced this population to 23.2% (regularization strength Γ = 0.5) and 0 (regularization strength Γ = 1.0) (PRONIA plus 18M sample, left panel of eFigure 15B in the Supplement), with the latter parsimony level significantly reducing BAC by −6.4% (PRONIA plus 18M sample, left panel of eFigure 15C, part 2 in the Supplement). Highly similar findings were obtained when analyzing the complete PRONIA cohort (right panels in eFigure 15B and C in the Supplement).
Further clinical scalability experiments showed that the condensed clinical-neurocognitive model matched the full model in correctly predicting PRONIA minus 18M cases with nontransition (specificity, 65.9%). Furthermore, its performance in the external ZInEP-CHR cohort (BAC, 65.3%; sensitivity, 87.5%; specificity, 43.1%) and the Bi-national Evaluation of At-Risk Symptoms in Children and Adolescents sample (BAC, 70.4%; sensitivity, 76.9%; specificity, 63.9%) was similar to the full model in the PRONIA-CHR sample (BAC, 63.3%; sensitivity, 87.0%; specificity, 39.6%) (eTable 14 in the Supplement) and the complete PRONIA cohort (BAC, 72.8%; sensitivity, 88.0%; specificity, 57.5%) (Table 2). Replacing the full model with its condensed counterpart in the nonregularized or regularized (Γ = 0.5) workflows did not increase the false-positive rate in the PRONIA minus 18M sample or the complete PRONIA cohort (eFigure 14C in the Supplement).
The validation of the unimodal sMRI-based workflow component in the ZInEP (BAC, 67.9%; sensitivity, 75.0%; specificity, 60.8%) and Früherkennung von Psychosen (BAC, 68.5%; sensitivity, 75.0%; specificity, 61.9%) samples approximated the PRONIA-CHR results (BAC, 70.8%; sensitivity, 86.4%; specificity, 55.3%) (eTable 14 in the Supplement). Finally, the stacked risk calculator composed of the condensed clinical-neurocognitive and sMRI-based models significantly outperformed these models in the ZInEP data (BAC, 71.3%; sensitivity, 75.0%; specificity, 67.7%) (Figure 3 and Table 2).
All risk calculators were significant in the permutation analysis (mean [SD] BAC, 77.3 [4.8]; mean [SD] sensitivity, 80.2% [6.5%]; mean [SD] specificity, 74.5% [13.8%]; P < .001 for FDR for all models) Table 2), but differences in BAC emerged (Figure 3). Multimodal risk calculators outperformed all unimodal counterparts (t range, 3.14-6.20; P < 3.19 × 10−8 to P = .002 for FDR), whereas the nonregularized prognostic sequence did not differ from the cybernetic model (mean [SD] BAC for nonregularized sequence, 83.7% [9.6%]; mean [SD] BAC for cybernetic model, 83.4% [9.6%]; t = 0.21; P = .44 for FDR). In addition, the stacked model (mean [SD] BAC, 79.7% [7.9%]) was outperformed by both the cybernetic model (mean [SD] BAC, 81.5% [9.6%]; t = 3.18; P = .001 for FDR) and the nonregularized sequential model (mean [SD] BAC, 82.0% [9.6%]; t = 3.82; P < .001 for FDR) in the complete PRONIA sample. Raters were comparable to unimodal risk calculators (mean [SD] BAC for raters, 68.2% [12.6%]; mean [SD] BAC range for unimodal predictors, 67.5% [16.4%] to 74.8% [6.8%]; t range, 0.002-1.56; P = .50 to P = .07 for FDR) but were outperformed by multimodal prediction algorithms in terms of higher BAC and reduced prognostic variability (stacked model vs raters, t = 4.64 [P < .001 for FDR]; cybernetic model vs raters, t = 7.82 [P < .001 for FDR]; sequence model vs raters, t = 8.46 [P < .001 for FDR]). Finally, the nonregularized sequential model reduced raters’ false-negative rate from 38.5% to 15.4% (PRONIA plus 18M sample) or 19.2% (complete PRONIA cohort) of cases.
Using a thorough model discovery and validation approach,21 our study demonstrated geographic transportability of expert-based clinical and biological psychosis transition prediction approaches across a transdiagnostic, multinational risk population. We found that combined risk calculators outperformed all unimodal counterparts and clinical raters in terms of prognostic accuracy and cross-site stability. Importantly, our study revealed that the increased diagnostic burden arising from data fusion could be mitigated through optimized sequential testing that arranges clinicians and risk calculators into clinically scalable prognostic workflows. Based on this form of deferral learning,55 we showed that the complete assessment battery is only needed in 23.2% of the initial population (eFigure 15B in the Supplement). This subgroup was enriched for patients who received a prediction of psychosis transition in the initial clinical-neurocognitive examination, suggesting that biological markers of psychosis transition are useful for delineating true-positive from false-positive findings at the later steps of a multistep prognostic assessment.
Examining the baseline heterogeneity of our transdiagnostic population, we found functional-neurocognitive impairments in the ROD group akin to the CHR group and low between-group neuroanatomical and genetic separability (eTable 12 in the Supplement), supporting the neurobiological proximity between early-onset affective and psychotic disorders.26,30,56 Although CHR syndromes expectedly separated CHR and ROD groups at the cross-sectional level, we observed that these syndromes emerged in 19.2% of patients with ROD during the follow-up period, which led to psychosis transition in 1.8% of cases.56 Strikingly, our analyses also showed that the models’ prognostic accuracy, particularly the sensitivity for psychosis transition in patients with CHR, depended on patients with ROD being part of the model discovery process, which further supports the pooling of both groups into a broader risk population (eTable 14 and eFigure 12 in the Supplement). Our finding of a transdiagnostic predictability of psychosis was corroborated by the generalizability of the clinical-neurocognitive and neuroanatomical models to external samples, which showed markedly different risk levels, age distributions, and diagnostic compositions.
The in-depth analysis of the clinical-neurocognitive domain revealed that the presence of APSs facilitated a good baseline separability of CHR syndromes vs ROD and substantially informed the prediction of psychosis transition (eTable 13 in the Supplement). However, measures of childhood adversity,46 motor disturbances,57 and facial affect recognition58 did not overlap between diagnostic and prognostic models (eFigure 16 in the Supplement) and thus could be regarded as transdiagnostic markers59-62 of poor psychosis-related outcomes, including transition to psychosis. This interpretation was supported by the prognostic generalization of the clinical-neurocognitive model to the clinically relevant separation of patients with (1) nonremitting/de novo and nonsymptomatic CHR syndrome courses (eTable 15 in the Supplement) or (2) unfavorable perceptual, affective, functional, and basic symptom trajectories (eFigure 13 in the Supplement).63,64 Importantly, the model’s prognostic generalization capacity did not encompass nonpsychotic diagnoses (eTable 16 in the Supplement), and, thus, its prognostic pluripotency was confined to diverse CHR-specific symptom courses.65
Furthermore, we confirmed the prognostic value of PRS for schizophenia,48 as reported recently,16 and extended those findings by showing that genetic information augments the performance of clinical-neurocognitive models and prognostic workflows in a broader risk population (Table 2 and eTable 20 in the Supplement). Within this transdiagnostic setting, we replicated group-level differences among patients with psychosis transition, patients with nontransition, and healthy controls16 but also found that PRS-based prognostic assignments specifically differentiated APS-related trajectories (eFigure 13 in the Supplement).66 They also delineated patient groups with abnormally high and low genetic risk compared with healthy controls (eFigure 8 in the Supplement)—a finding that may point to distinct environmental and/or neurobiological pathways conferring risk and resilience to psychosis.67
The analysis of the structural neuroimaging data revealed a psychosis-predictive brain signature that generalized well across 3 independent cohorts. This signature overlapped with brain alterations previously reported to correlate with perceptual abnormalities, disorganization of speech and thought, and poor insight in early, subsyndromal, or prodromal stages of psychosis.9,68-70 Interestingly, nontransition-assigned patients showed reversed temporo-occipital volume reductions, which differentiated them from healthy controls (eFigure 9 in the Supplement). These findings may point to ongoing compensatory mechanisms of resilience to psychosis, as reported previously in a longitudinal sMRI study of adolescents with CHR.71 In this regard, our sMRI-based risk calculator may serve as a useful tool for enriching future observational studies and clinical trials with at-risk patients who express potential brain mechanisms of resilience to psychosis transition.
We observed that our raters matched unimodal risk calculators in predicting psychosis as measured by their BAC. However, raters also showed a pronounced optimism bias (low sensitivity and high specificity) toward the true risk of poor clinical outcome.13 It is noteworthy that their prognoses were based on all information collected in an extended study-related assessment and likely would be less accurate in routine, time-restricted diagnostic settings. Because the algorithmic counterparts showed exactly the inverted bias (high sensitivity and low specificity), the integration of clinicians and risk calculators into the cybernetic model produced a superior predictive system.50 Furthermore, our prognostic workflows demonstrated that similar levels of prognostic accuracy can be achieved by reducing the false-positive rate through sequential model application in patients with an estimated higher risk for psychosis transition (eFigure 14 in the Supplement). In this subgroup, the removal of the final sMRI-based assessment step increased false-positive findings (eFigure 15 in the Supplement), suggesting that the cost-benefit ratio of expensive neuromarkers needs to be individually adjusted according to the patient’s predicted risk.72
The finding that prognostic workflows always started with the clinical-neurocognitive model places the recognition of the clinical gestalt of emerging psychosis at the gateway of more precise early detection techniques.73 Our scalability experiments suggest that the laborious recognition of this pattern currently practiced in early recognition services could be effectively condensed to a few clinical-neurocognitive variables,6 thus enhancing the clinical utility of the proposed workflow. Nonetheless, future studies should revisit the validity of the selected 7 variables because they have been taken out of their original assessment context. Further studies also need to quantitatively explore the information patterns guiding clinicians’ gut-feeling estimates of psychosis transition and in turn foster more effective clinical early recognition strategies that integrate with cybernetic systems.
Psychosis transitions were limited to 26 individuals in the PRONIA discovery sample. This sample size increased the risk of producing overly optimistic prediction results owing to an accidental collection of well-classifiable cases. We implemented a multistep model validation procedure to guard against this possibility, including label permutation testing, strict nested cross-validation of all processing steps,74 in-depth model analysis to assess possible prognostic confounds and moderators, specificity testing of all models in a completely held-back portion of the PRONIA sample, and model benchmarking in 3 independent data sets, which provided a further 45 individuals with psychosis transition and 600 with nontransition for external validation. Owing to limited data availability in these samples, only the condensed clinical-neurocognitive, sMRI-based, and a specific stacked risk calculator trained on the outputs of the former 2 models could be externally validated. However, our internal-external validation approach followed established guidelines for model construction and validation.21 In keeping with this literature, the similar performance levels observed in our LOSOCV and independent validation experiments support the validity of the models not tested in external samples.
In this prognostic study, we identified generalizable risk assessment tools that can be arranged into a multimodal prognostic workflow for a clinically viable, individualized prediction of psychosis in patients with CHR states and ROD. Our study showed for the first time, to our knowledge, that the augmentation of human prognostic abilities with algorithmic pattern recognition improves prognostic accuracy to margins that likely justify the clinical implementation of cybernetic decision-support tools. New international collaborations, such as the HARMONY (Harmonization of At Risk Multisite Observational Networks for Youth) initiative,75 may help to propel a reciprocal and iterative process of clinical validation and refinement of these prognostic tools in real-world early recognition services.
Accepted for Publication: September 12, 2020.
Published Online: December 2, 2020. doi:10.1001/jamapsychiatry.2020.3604
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Koutsouleris N et al. JAMA Psychiatry.
Corresponding Author: Nikolaos Koutsouleris, MD, Department of Psychiatry and Psychotherapy, Ludwig-Maximilian-University, Nussbaumstrasse 7, D-80336 Munich, Germany (email@example.com).
Author Contributions: Drs Schultze-Lutter, Theodoridou, and Meisenzahl contributed equally to this work. Dr Koutsouleris had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Koutsouleris, Ruhrmann, Kambeitz, Hietala, Schirmer, Schimmelmann, Falkai, Salokangas, Borgwardt, Wood, Upthegrove, Theodoridou, Meisenzahl.
Acquisition, analysis, or interpretation of data: Koutsouleris, Dwyer, Degenhardt, Maj, Urquijo-Castro, Sanfelici, Popovic, Oeztuerk, Haas, Weiske, Ruef, Kambeitz-Ilankovic, Antonucci, Neufang, Schmidt-Kraepelin, Ruhrmann, Penzel, Kambeitz, Haidl, Rosen, Chisholm, Riecher-Rössler, Egloff, Schmidt, Andreou, Hietala, Schirmer, Romer, Walger, Franscini, Traber-Walker, Schimmelmann, Flückiger, Michel, Rössler, Borisov, Krawitz, Heekeren, Buechler, Pantelis, Salokangas, Lencer, Bertolino, Borgwardt, Noethen, Brambilla, Wood, Upthegrove, Schultze-Lutter, Theodoridou, Meisenzahl.
Drafting of the manuscript: Koutsouleris, Degenhardt, Kambeitz, Salokangas, Upthegrove, Meisenzahl.
Critical revision of the manuscript for important intellectual content: Koutsouleris, Dwyer, Degenhardt, Maj, Urquijo-Castro, Sanfelici, Popovic, Oeztuerk, Haas, Weiske, Ruef, Kambeitz-Ilankovic, Antonucci, Neufang, Schmidt-Kraepelin, Ruhrmann, Penzel, Kambeitz, Haidl, Rosen, Chisholm, Riecher-Rössler, Egloff, Schmidt, Andreou, Hietala, Schirmer, Romer, Walger, Franscini, Traber-Walker, Schimmelmann, Flückiger, Michel, Rössler, Borisov, Krawitz, Heekeren, Buechler, Pantelis, Falkai, Salokangas, Lencer, Bertolino, Borgwardt, Noethen, Brambilla, Wood, Schultze-Lutter, Theodoridou, Meisenzahl.
Statistical analysis: Koutsouleris, Maj, Popovic, Borisov, Krawitz, Meisenzahl.
Obtained funding: Koutsouleris, Degenhardt, Ruhrmann, Riecher-Rössler, Schirmer, Pantelis, Salokangas, Borgwardt, Brambilla, Wood, Meisenzahl.
Administrative, technical, or material support: Koutsouleris, Dwyer, Sanfelici, Oeztuerk, Haas, Ruef, Kambeitz-Ilankovic, Antonucci, Ruhrmann, Penzel, Kambeitz, Haidl, Rosen, Riecher-Rössler, Egloff, Schmidt, Andreou, Hietala, Romer, Flückiger, Michel, Buechler, Pantelis, Salokangas, Borgwardt, Noethen, Brambilla, Upthegrove, Theodoridou, Meisenzahl.
Supervision: Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Kambeitz, Andreou, Hietala, Schirmer, Walger, Flückiger, Michel, Krawitz, Pantelis, Falkai, Lencer, Bertolino, Borgwardt, Noethen, Brambilla, Upthegrove, Schultze-Lutter, Theodoridou.
Conflict of Interest Disclosures: Dr Koutsouleris reported receiving grants from the European Union (EU) during the conduct of the study and having a patent to US20160192889A1 issued. Ms Sanfelici reported receiving personal fees from H. Lundbeck A/S outside the submitted work. Dr Ruhrmann reported receiving grants from the European Commission during the conduct of the study. Dr Riecher-Rössler reported receiving grants from the EU during the conduct of the study. Dr Andreou reported receiving nonfinancial support from Sunovion Pharmaceuticals, Inc, and H. Lundbeck A/S outside the submitted work. Dr Hietala reported receiving personal fees from Orion Company, Ltd, Otsuka Pharmaceutical Co, Ltd, and H. Lundbeck A/S and European College of Neuropsychopharmacology Congress participation support from Takeda Pharmaceutical Company Limited during the conduct of the study. Dr Schirmer reported receiving personal fees from GE Healthcare GmbH outside the submitted work. Dr Romer reported receiving grants from the EU during the conduct of the study. Dr Schimmelmann reported receiving personal fees from Shire Deutschland GmbH outside the submitted work. Dr Flückiger reported receiving grants from the Swiss National Foundation during the conduct of the study. Dr Michel reported receiving grants from the Swiss National Foundation during the conduct of the study. Dr Rössler reported receiving grants from The Zurich Program for Sustainable Development of Mental Health Services (for Zurich Early Recognition Program [ZInEP]) and support by a private donation. Dr Heekeren reported receiving grants from The Zurich Program for Sustainable Development of Mental Health Services during the conduct of the study. Dr Pantelis reported receiving grants from Australian National Health and the Medical Research Council during the conduct of the study and personal fees from H. Lundbeck A/S and Australia Pty Ltd outside the submitted work. Dr Noethen reported receiving personal fees from the Lundbeck Foundation, Robert-Bosch-Stiftung GmbH, HMG Systems Engineering GmbH, Shire Deutschland GmbH, and Life & Brain GmbH outside the submitted work and having a patent to Means and Methods for Establishing a Clinical Prognosis of Diseases Associated With the Formation of Aggregates of Aß1-42 issued. Dr Upthegrove reported receiving personal fees from Sunovion Pharmaceuticals, Inc, outside the submitted work. Dr Meisenzahl reported having a patent to US20160192889A1 licensed. No other disclosures were reported.
Funding/Support: PRONIA (Personalised Prognostic Tools for Early Psychosis Management) is a Collaboration Project funded by the EU under the 7th Framework Programme and grant agreement 602152. This study was also supported by grant COMMITMENT by the German Federal Ministry of Education and Research (BMBF) within the e:Med programme (Dr Degenhardt), COST Action EnGagE CA17130 from the EU COST Programme (Dr Degenhardt), and the Else-Kröner-Fresenius-Foundation through the Clinician Scientist Program Else-Kröner-Fresenius-Foundation-Translational Psychiatry (Drs Popovic and Oeztuerk).
Role of the Funder/Sponsor: The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
The PRONIA Consortium: The following members of the PRONIA Consortium performed the screening, recruitment, rating, examination, and follow-up of the study participants and were involved in implementing the examination protocols of the study, setting up its information technology infrastructure, and organizing the flow and quality control of the data analyzed in this study between the local study sites and the central study database: Shalaila Haas, Alkomiet Hasan, Claudius Hoff, Ifrah Khanyaree, Aylin Melo, Susanna Muckenhuber-Sternbauer, Yanis Köhler, Ömer Öztürk, Nora Penzel, David Popovic, Adrian Rangnick, Sebastian von Saldern, Rachele Sanfelici, Moritz Spangemacher, Ana Tupac, Maria Fernanda Urquijo-Castro, Johanna Weiske, Antonia Wosgien, and Camilla Krämer (Department of Psychiatry and Psychotherapy, Ludwig-Maximilian-University); Karsten Blume, Dennis Hedderich, Dominika Julkowski, Nathalie Kaiser, Thorsten Lichtenstein, Ruth Milz, Alexandra Nikolaides, Tanja Pilgram, Mauro Seves, and Martina Wassen (Department of Psychiatry and Psychotherapy, University of Cologne); Christina Andreou, Laura Egloff, Fabienne Harrisberger, Ulrike Heitz, Claudia Lenz, Letizia Leanza, Amatya Mackintosh, Renata Smieskova, Erich Studerus, Anna Walter, and Sonja Widmayer (Department of Psychiatry, Psychiatric University Hospital, University of Basel); Chris Day, Sian Lowri Griffiths, Mariam Iqbal, Mirabel Pelton, Pavan Mallikarjun, Alexandra Stainton, and Ashleigh Lin (Institute for Mental Health and School of Psychology, University of Birmingham); Alexander Denissoff, Anu Ellilä, Tiina From, Markus Heinimaa, Tuula Ilonen, Päivi Jalo, Heikki Laurikainen, Antti Luutonen, Akseli Mäkela, Janina Paju, Henri Pesonen, Reetta-Liina Säilä, Anna Toivonen, and Otto Turtonen (Department of Psychiatry, University of Turku); Sonja Botterweck, Norman Kluthausen, Gerald Antoch, Julian Caspers, and Hans-Jörg Wittsack (Department of Psychiatry, Psychiatric University Hospital LVR/Heinrich-Heine-University Düsseldorf, University of Düsseldorf); Giuseppe Blasi, Giulio Pergola, Grazia Caforio, Leonardo Fazio, Tiziana Quarto, Barbara Gelao, Raffaella Romano, Ileana Andriola, Andrea Falsetti, Marina Barone, Roberta Passiatore, and Marina Sangiuliano (Department of Basic Medical Science, Neuroscience and Sense Organs, University of Bari Aldo Moro); Marian Surmann, Olga Bienek, and Udo Dannlowski (Department of Psychiatry and Psychotherapy, University of Münster); Ana Beatriz Solana, Manuela Abraham, and Timo Schirmer (GE Global Research, Inc); Carlo Altamura, Marika Belleri, Francesca Bottinelli, Adele Ferro, and Marta Re (Department of Neuroscience and Mental Health, Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico, Workgroup of Paolo Brambilla, University of Milan); Emiliano Monzani and Maurizio Sberna (Programma 2000, Niguarda Hospital, Workgroup of Paolo Brambilla, University of Milan); Giampaolo Perna, Maria Nobile, and Alessandra Alciati (San Paolo Hospital, Workgroup of Paolo Brambilla, University of Milan); Armando D’Agostino and Lorenzo Del Fabro (Villa San Benedetto Menni, Albese con Cassano, Workgroup of Paolo Brambilla, University of Milan); Matteo Balestrieri, Carolina Bonivento, Giuseppe Cabras, and Franco Fabbro (Department of Medical Area, Workgroup of Paolo Brambilla, University of Udine); and Marco Garzitto and Sara Piccin (IRCCS Scientific Institute E. Medea, Polo FVG, Workgroup of Paolo Brambilla, University of Udine).
Create a personal account or sign in to: