Assessment of a Person-Level Risk Calculator to Predict New-Onset Bipolar Spectrum Disorder in Youth at Familial Risk

Key Points

Question  Can a risk calculator be developed to predict, on the individual level, the risk of developing bipolar spectrum disorder in youth at familial risk for the disorder?

Findings  In an ongoing cohort study that included 412 offspring of parents with bipolar disorder (54 of whom developed bipolar spectrum disorder during follow-up), predictors from the literature were used to construct a risk calculator to distinguish those who would develop bipolar spectrum disorder in the next 5 years vs those who would not. The model (which included mood and anxiety symptoms, general psychosocial functioning, and parental age at mood disorder onset) discriminated with an area under the curve of 0.76, indicating good discrimination, comparable to risk calculators used clinically in other areas of medicine.

Meaning  This risk calculator is an important practical tool to inform clinical decisions (eg, frequency of monitoring) and research studies (eg, to help identify an ultra–high-risk group for studies of biomarkers and prevention).

Importance  Early identification of individuals at high risk for the onset of bipolar spectrum disorder (BPSD) is key from both a clinical and research perspective. While previous work has identified the presence of a bipolar prodrome, the predictive implications for the individual have not been assessed, to date.

Objective  To build a risk calculator to predict the 5-year onset of BPSD in youth at familial risk for BPSD.

Design, Setting, and Participants  The Pittsburgh Bipolar Offspring Study is an ongoing community-based longitudinal cohort investigation of offspring of parents with bipolar I or II (and community controls), recruited between November 2001 and July 2007, with a median follow-up period of more than 9 years. Recruitment has ended, but follow-up is ongoing. The present analysis included offspring of parents with bipolar I or II (aged 6-17 years) who had not yet developed BPSD at baseline.

Main Outcomes and Measures  This study tested the degree to which a time-to-event model, including measures of mood and anxiety, general psychosocial functioning, age at mood disorder onset in the bipolar parent, and age at each visit, predicted new-onset BPSD. To fully use longitudinal data, the study assessed each visit separately, clustering within individuals. Discrimination was measured using the time-dependent area under the curve (AUC), predicting 5-year risk; internal validation was performed using 1000 bootstrapped resamples. Calibration was assessed by comparing observed vs predicted probability of new-onset BPSD.

Results  There were 412 at-risk offspring (202 [49.0%] female), with a mean (SD) visit age of 12.0 (3.5) years and a mean (SD) age at new-onset BPSD of 14.2 (4.5) years. Among them, 54 (13.1%) developed BPSD during follow-up (18 with BD I or II); these participants contributed a total of 1058 visits, 67 (6.3%) of which preceded new-onset BPSD within the next 5 years. Using internal validation to account for overfitting, the model provided good discrimination between converting vs nonconverting visits (AUC, 0.76; bootstrapped 95% CI, 0.71-0.82). Important univariate predictors of outcome (AUC range, 0.66-0.70) were dimensional measures of mania, depression, anxiety, and mood lability; psychosocial functioning; and parental age at mood disorder.

Conclusions and Relevance  This risk calculator provides a practical tool for assessing the probability that a youth at familial risk for BPSD will develop new-onset BPSD within the next 5 years. Such a tool may be used by clinicians to inform frequency of monitoring and treatment options and for research studies to better identify potential participants at ultra high risk of conversion.

Offspring of parents with bipolar disorder (BD) are at risk for a broad range of psychopathology, including bipolar spectrum disorder (BPSD).^1,2 Research indicates that they also have a higher rate of anxiety, unipolar mood disorders, and behavioral disorders,^2-5 along with a range of difficulties, including mood lability, anxiety, and attention problems.^6-9 However, identifying which offspring will go on to develop BPSD remains a major challenge. Multiple studies have pointed to a prodrome that precedes the onset of BD, lasting at least 2 years¹⁰ and up to 10 years.¹¹ The most consistent symptom predictors of BD (in youth and adults) are subthreshold manic symptoms, subthreshold and full-threshold depressive symptoms, mood lability, and anxiety,^1,10,12,13 which are not necessarily specific for BD. Despite this progress, it remains unclear how these factors combine to influence the prognosis of an individual at familial risk.

The challenge of risk prediction is not unique to psychiatry. Over the past few decades, other fields of medicine have used risk predictive models to assess how a combination of risk and protective factors influence the risk of developing disorders.¹⁴ From these models, risk calculators can be constructed, which allow the clinician to enter in relevant variables and estimate the probability that a particular outcome will occur during a given period.¹⁵ The most well-known is probably the Framingham Risk Score for cardiovascular disease, which is used regularly in primary care to stratify risk of future myocardial infarction and thus appropriately intervene to decrease risk.¹⁶

While risk calculators have been used widely in nonpsychiatric medical disorders, the development of such models in psychiatry has been limited. A recent review article identified 43 adult studies that generated risk predictive models.¹⁷ Most of these models were for depression and psychosis, with only one for BD; the use of internal and external validation within these studies was somewhat limited. Adding significantly to this literature, the authors of the North American Prodrome Longitudinal Study (NAPLS) generated a risk calculator from their data to predict the 2-year onset of psychosis in a sample of very high-risk adolescents and young adults.¹⁸ They tested a model based on predictors from the literature and found adequate levels of discrimination (able to distinguish those who converted vs those who did not) and calibration (predicted and observed risk matched within each stratum of the risk calculator). Furthermore, they were able to test this model on a completely independent data set and found that the discrimination and calibration were very good.¹⁹ More recently, a risk calculator was developed to predict psychosis in adults initially seen for treatment of nonpsychotic psychiatric disorders, which also yielded very good discrimination.²⁰

Herein, we build a risk calculator to determine the risk of conversion to BPSD in youth at familial risk for the disorder using data from the ongoing community-based longitudinal Pittsburgh Bipolar Offspring Study (BIOS).^1,2 The BIOS is a cohort investigation of offspring of parents with bipolar I or II (and community controls), recruited between November 2001 and July 2007, with a median follow-up period of more than 9 years. Recruitment has ended, but follow-up is ongoing. Such a risk calculator would have both clinical and research utility. From a research perspective, a risk score could be used to identify individuals at ultra high risk of conversion, which would allow for more efficient testing of biomarkers that precede conversion and therapies that might prevent or delay onset. From a clinical perspective, this risk score could be used to monitor the risk of conversion over time and might also help clinicians weigh risks and benefits of using certain psychotherapy or pharmacological interventions.

The methods of BIOS have been described in detail in prior studies^13,21 and are described in detail in the eMethods in the Supplement. All procedures were approved by the University of Pittsburgh Institutional Review Board.

Parents with bipolar I or II were recruited via advertisement, research studies, and outpatient clinics. The study recruited offspring who were originally aged 6 to 18 years, unless the child had an illness that interfered with participation in the study (eg, mental retardation or autism). We used any visit where the offspring was aged 6 to 17 years. For the present analysis, we excluded youth with BPSD at baseline (n = 33), leaving 480 offspring, 412 of whom were younger than 18 years at baseline and had at least one follow-up visit.

Written informed consent from the parents and written assent from the children were obtained. Participating parents were assessed by direct interview using the Structured Clinical Interview for DSM-IV. At baseline and during follow-up visits, parents and their offspring were interviewed using the Kiddie Schedule for Affective Disorders and Schizophrenia for School-Age Children–Present and Lifetime Version (K-SADS-PL) for nonmood disorders, the K-SADS Mania Rating Scale (KMRS) and the depression items from the KSADS–Present Version (KDRS), which assess symptoms (both subthreshold and threshold) during the worst week over the past month.^22,23 Summary scores were obtained using clinical consensus, integrating parent and offspring interviews. Parents and offspring completed several rating scales (at baseline and follow-up visits) covering a range of psychopathology, including (among others) the Children’s Affective Lability Scale (CALS)²⁴ and the Screen for Child Anxiety Related Emotional Disorders (SCARED).²⁵ We used the Children’s Global Assessment Scale (CGAS)²⁶ as a basic quantification of functioning at home and school for children and adolescents.

Follow-up evaluations were performed every 2 years to assess for the onset of DSM-IV disorders. Date of bipolar onset was set to be the first time the participant met criteria for BD not otherwise specified (BD-NOS) or DSM-IV criteria for a manic, mixed, or hypomanic episode. As detailed elsewhere, operationalized criteria were used for BD-NOS.²⁷ Youth with this diagnosis have a family history of BD, suicidality, risk for substance abuse, and psychosocial impairment comparable to those with BD I or II,^21,27-29 and 50% progress to BD I or II within 5 years.^1,30

To make use of the full extent of longitudinal data, we used assessment as the unit of analysis, allowing us to use presenting symptoms at both baseline and relevant follow-up visits and to model the time to new-onset BPSD (or censoring) separately from each assessment. Inclusion of data from follow-up visits allows us to incorporate symptoms that might occur closer to BPSD conversion, which is especially important because some prodromal symptoms seem to emerge proximal to conversion.¹³ We included all index assessments that were (1) before the onset of BPSD, (2) before age 18 years, and (3) followed by at least one additional assessment. Index assessments for participants after age 18 years were excluded because different self-report scales were used in youth vs adults.

We used baseline-resetting Cox proportional hazards regression to model the time to event (conversion or right censoring) from each index assessment using a frailty model parameterization to account for clustering of visits within individual. Because of the wide range of follow-up from each index assessment, we used the model to assess the predicted cumulative hazard (ie, risk) at 5 years. The median follow-up time for baseline-resetting Cox proportional hazards regression was 5.9 years, thus allowing for sufficient data to test cumulative hazard within a 5-year window.

To avoid the circular logic of testing the prognostic power of variables that have previously shown to be predictive within the BIOS sample, we used the results from a recent meta-analysis¹⁰ that identified prodromal symptoms in children and adults who later develop BD. This meta-analysis generated an objective list of 26 items found to be fairly common (>25%) in individuals before conversion, including subsyndromal manic symptoms, subsyndromal depressive symptoms, mood lability, general psychosocial functioning, and anxiety. To specifically capture these symptoms, we modified the KMRS and KDRS (only including items found to be common in the meta-analysis); CALS, SCARED, CGAS, child age, and parental age at mood disorder onset (in the bipolar proband) were also entered as predictors (Box). We use child-reported CALS and SCARED in the primary analysis. Parental age at mood disorder onset was entered because several investigations have shown that BD with earlier onset is more likely to be familial.³¹

Box Section Ref ID

To avoid overfitting, training and testing were performed and internally validated via the algorithm by Harrell et al³² for bootstrap optimism correction using 1000 bootstrapped resamples (eMethods in the Supplement). Discrimination and calibration were evaluated within this bootstrap procedure; discrimination was measured using the time-dependent area under the curve (AUC), predicting the 5-year risk of an event.³³ Calibration was assessed by (1) plotting observed vs predicted probability of conversion to BPSD and (2) using the Hosmer-Lemeshow test. Sensitivity, specificity, and positive predictive value were assessed at a range of thresholds. To test the predictive importance of each variable, we used (1) the 5-year AUC of a model with only that variable and (2) the decrement in the AUC with removal of that variable (or subset of variables) from the full model. Bootstrapped 95% CIs were calculated for both measures to assess statistical significance. To ensure that the findings were not driven by youth who developed BD-NOS (but not BD I or II), we ran a sensitivity analysis removing youth who had a diagnosis of BD-NOS at their most recent visit. Further supplemental analyses were used to test additional potential demographic and clinical predictors of BPSD. Specifically, we tested the change in AUC with the addition of each variable and calculated bootstrapped 95% CIs to assess statistical significance.

Previous studies^2,34 have described the baseline characteristics of the BIOS sample. Offspring of bipolar parents (n = 412) were followed up for a median of 9.5 years; during that time, 54 (13.1%) developed new-onset BPSD (9 BD I, 9 BD II, and 36 BD-NOS). Our sample consisted of 1058 visits (with a median of 2.0 years between visits). Of these visits, 104 (9.8%) were “converting,” meaning that they preceded the onset of BPSD over the remaining follow-up period (median, 5.9 years); 67 visits (6.3%) were followed by conversion to BPSD within the next 5 years. The mean (SD) visit age was 12.0 (3.5) years, and the mean (SD) age at new-onset BPSD was 14.2 (4.5) years.

After bootstrapping internal validation, the risk calculator discriminated between converting vs nonconverting visits with a 5-year AUC of 0.76 (bootstrapped 95% CI, 0.71-0.82), indicating good discrimination. A model using parent-report scales (CALS and SCARED), in lieu of child report, showed similar discrimination, with a 5-year AUC of 0.77 (bootstrapped 95% CI, 0.72-0.83). The distribution of risk scores for participants who developed new-onset BPSD vs those who did not indicated clinically relevant discrimination between these 2 groups (Figure 1). The calibration plot shows that the predicted and observed risks of new-onset BPSD were consistent throughout the range of risk scores (Figure 2). The median predicted risk score of 0.05 matched closely to the 5-year probability of conversion of 0.06, and the Hosmer-Lemeshow test result was not significant (χ² = 6.19, P = .63), indicating good calibration.

Table 1 lists the mean values for each predictor within converting vs nonconverting visits and the degree to which each variable contributed to model prediction. Based on this risk calculator, an individual with more symptoms (anxiety, mood lability, depressive, and manic symptoms), lower general psychosocial functioning, and whose parent had a younger age at mood disorder onset is at greater risk for developing new-onset BPSD. Univariate AUC values indicate that all predictors except offspring age at visit discriminated moderately well (AUC range, 0.66-0.70) and that discrimination is unlikely to be due to chance. Removing individual variables did not lead to a significant decrement in the AUC. Removing pairs of variables also did not lead to a significant decrease in the AUC. The decrement associated with removing the parental age at onset and CGAS was 0.06 (bootstrapped 95% CI, −0.00 to 0.12) (eTable 1 in the Supplement). Four combinations of variable triplets were associated with significant decrements in the AUC when removed, ranging between 0.06 and 0.07. All triplets associated with a significant decrement included the parental age at onset variable, indicating the importance of this variable to the risk calculator (eTable 2 in the Supplement).

Table 2 lists model performance characteristics at a range of thresholds. For example, a less stringent threshold of 0.05 would capture 82% of cases, but only 15% of the selected sample would be expected to develop new-onset BPSD within 5 years. Using a more stringent threshold of 0.15, there would be a higher frequency of 5-year conversion (30%) but would only capture 37% of cases.

We reran our analysis excluding any participant who developed BD-NOS (but not BD I or II) by the end of follow-up. Within this smaller sample (979 visits, 19 of which preceded conversion by ≤5 years), the 5-year AUC was 0.76 (95% CI, 0.69-0.83), indicating good discrimination. Therefore, our findings were not driven by BD-NOS. We also conducted exploratory analyses to assess whether other predictors, not found in the meta-analysis,¹⁰ might improve discrimination of the risk calculator. Addition of demographic characteristics (socioeconomic status and living with both biological parents), abuse, and a previous depression diagnosis did not significantly improve model discrimination (AUC range, 0.76-0.77) (eTable 3 in the Supplement).

Using data from a longitudinal cohort study of offspring of parents with BD, we have built a risk calculator to predict the 5-year risk of new-onset BPSD (available at http://www.pediatricbipolar.pitt.edu). This model provides clinically relevant discrimination between those who will develop BPSD within 5 years vs those who will not. The AUC of 0.76 is comparable to previous risk calculators used in medicine (eg, cardiovascular disease [AUC range, 0.76-0.79]¹⁶ and colorectal cancer [AUC, 0.68]³⁵); it is also comparable to discrimination achieved by the NAPLS risk calculator¹⁸ for new-onset psychosis (0.71 in the initial study¹⁸ and 0.79 in the validation study¹⁹). Anxiety, manic symptoms, depressive symptoms, mood lability, poor general psychosocial functioning, and earlier parental age at onset individually and collectively predicted new-onset BPSD. Additional analyses excluding participants who developed BD-NOS (but not BD I or II) by the end of follow-up indicated that the findings were not driven by youth with BD-NOS.

This work builds on previous studies that identified predictors of BPSD at a population level. Specifically, we chose predictors (including a modified selection of depressive and manic symptoms) based on a recent meta-analysis¹⁰ that did not include the findings from the BIOS sample. Notably, there was much overlap between the findings of this meta-analysis and previous results from BIOS,¹³ which indicated that subthreshold manic symptoms and mood lability were important predictors of new-onset BPSD. In this way, we build on the results from our group’s previous analysis, while avoiding model selection based on these findings, which could induce circular logic and limit generalizability.¹³

Building a risk calculator to estimate person-level risk has important utility for both clinicians and research studies. For clinicians, this risk calculator represents a practical tool that can be used to assess risk that a patient will develop BPSD within the next 5 years; such information can be used to provide prognostic information to the patient and his or her family, as well as guide frequency of monitoring and early intervention. For researchers, this risk calculator provides a metric for identifying an ultra–high-risk population with a high chance of developing BPSD over the next 5 years, which may be useful for assessing biomarkers (eg, neuroimaging) and for testing preventive measures and early intervention. This risk score will also change and can be monitored over time by both clinicians and researchers, thus providing some indication of the risk trajectory that might shed light on the efficacy of a particular intervention (ie, an intermediate outcome). We provide a range of thresholds that might be used when deciding whether to classify a particular individual as being at ultra high risk of conversion; positive predictive values provide an indication of the proportion above that threshold who would be expected to develop new-onset BPSD. The optimal threshold will depend on several factors, such as the goal of a particular study or the risk profile of a given intervention.

While the primary aim of this study was not to assess the association of individual variables, we assessed the influence of removing each variable (and variable pairs and triplets) to provide some indication of which predictors were most central to the risk calculator. Removal of individual variables and variable pairs did not lead to significant decrements in the AUC, reflecting redundancy in the model. This finding is not surprising given that each scale was highly correlated with at least one other scale in the model (Spearman rank correlation range, ≥0.58) and that even individual variables discriminated well (AUC range, >0.65). In addition, there was limited power to answer this secondary question, as evidenced by the wide 95% CIs around the decrement estimates. Nonetheless, consistent with the literature,^31,36,37 parental age at mood disorder onset emerges as the most important independent predictor within the model because only combinations that included this variable led to a significant decrement in the AUC.

The findings of our study should be considered in light of some limitations. First, our sample was not clinically recruited but rather was selected without regard to symptoms in the offspring. While the predictors in the risk calculator were chosen based on a meta-analysis¹⁰ that includes clinical samples, the specific estimates of risk are related to the base rate of the sample and might thus underestimate risk in a clinical population. Future work should externally validate this risk calculator in a clinical setting. Second, our risk calculator was purposefully derived in a population of offspring of parents with BD because these offspring are at elevated risk of BD; it is presently unknown how this risk calculator may perform among individuals without such genetic loading. Third, we used state-of-the-art methods to internally validate our analysis, but we did not have a sample available for external validation. The latter is the criterion standard and is the next step to establishing the clinical utility of this risk calculator. Fourth, while we had adequate numbers of new-onset BPSD to build a risk calculator, we had few youth with BD I or II. However, we had adequate power to conduct a sensitivity analysis, which revealed findings consistent with those of the primary model. Fifth, follow-up visits were scheduled every 2 years. Therefore, we do not know the precise timing of BPSD onset, and our analyses might have missed transient symptoms.

Despite these limitations, this study developed the first risk calculator to predict the onset of BPSD in youth at familial risk, to our knowledge, and one of the first risk calculators for use in psychiatry. We built our predictive model using the results from a recent meta-analysis¹⁰ and found that dimensional mood and anxiety symptoms, general psychosocial functioning, and parental age at mood disorder onset provide clinically relevant discrimination between those who will develop BPSD within a 5-year follow-up vs those who will not. We recognize that replication of these findings is warranted before the risk calculator can be confidently used for clinical decision making. In the interim, this risk calculator provides a practical tool for assessing the prognosis and guiding monitoring and early intervention for offspring of parents with BD.

Accepted for Publication: May 16, 2017.

Corresponding Author: Danella M. Hafeman, MD, PhD, Department of Psychiatry, University of Pittsburgh, 3811 O’Hara St, Pittsburgh, PA 15213 (hafemand@upmc.edu).

Published Online: July 5, 2017. doi:10.1001/jamapsychiatry.2017.1763

Author Contributions: Drs Hafeman and Birmaher had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: T. R. Goldstein, Axelson, B. Goldstein, Sakolsky, Diler, Brent, Kupfer, Birmaher.

Acquisition, analysis, or interpretation of data: Hafeman, Merranko, T. R. Goldstein, Axelson, Monk, Hickey, Diler, Iyengar, Brent, Kupfer, Kattan, Birmaher.

Drafting of the manuscript: Hafeman, Merranko, T. R. Goldstein, Hickey, Kupfer, Birmaher.

Critical revision of the manuscript for important intellectual content: Hafeman, Merranko, T. R. Goldstein, Axelson, B. Goldstein, Monk, Sakolsky, Diler, Iyengar, Brent, Kupfer, Kattan, Birmaher.

Statistical analysis: Hafeman, Merranko, T. R. Goldstein, Hickey, Diler, Iyengar, Brent, Kattan, Birmaher.

Obtained funding: Birmaher.

Administrative, technical, or material support: Axelson, Monk, Hickey, Sakolsky, Diler, Birmaher.

Study supervision: T. R. Goldstein, Axelson, Diler, Birmaher.

Conflict of Interest Disclosures: Dr Hafeman reported receiving funding from The Klingenstein Third Generation Foundation. Dr T. R. Goldstein reported receiving grant funding from the National Institute of Mental Health (NIMH), the Brain & Behavior Research Foundation, and the American Foundation for Suicide Prevention and reported receiving royalties from Guilford Press. Dr Axelson reported serving as a consultant to Janssen Research, reported receiving grant support from Neuronetics, and reported receiving royalties from UpToDate. Dr Sakolsky reported receiving research support from the NIMH and the National Alliance for Research on Schizophrenia and Depression, reported receiving an honorarium from the American Academy of Child and Adolescent Psychiatry for continuing medical education work, reported serving as an editorial board member of Child and Adolescent Psychopharmacology News, reported serving as a specialty consultant for Prescriber’s Letter, and reported serving as a paid consultant to L.E.K. Consulting. Dr Brent reported receiving royalties from Guilford Press; reported receiving royalties from the electronic self-rated version of the Columbia–Suicide Severity Rating Scale (C-SSRS) from eResearch Technology, Inc (ERT); reported receiving consulting fees from Lundbeck; and reported serving as an UpToDate psychiatry section editor. Dr Kupfer reported serving as a consultant to the American Psychiatric Association (as chair of the DSM-5 Task Force), reported having joint ownership of copyright for the Pittsburgh Sleep Quality Index, reported being a member of the Valdoxan Advisory Board of Servier International, and reported being a stockholder in AliphCom, HealthRhythms, Inc, and Psychiatric Assessments, Inc. Dr Birmaher reported receiving royalties from American Psychiatric Publishing, Random House, Lippincott Williams & Wilkins, and UpToDate and reported serving as a consultant to Janssen Research. No other disclosures were reported.

Funding/Support: This study was supported by grant R01 MH60952 from the National Institute of Mental Health (Dr Birmaher).

Role of the Funder/Sponsor: The funding organization did not have a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Additional Contributions: Eric A. Youngstrom, PhD (The University of North Carolina at Chapel Hill) provided input on earlier versions of this analysis. He was a paid consultant on the above grant.