Identification of Suicide Attempt Risk Factors in a National US Survey Using Machine Learning

Key Points

Question  Can survey data identify risk factors of nonfatal suicide attempt in the general population?

Findings  This study used a large, nationally representative longitudinal survey of US adults to create a suicide attempt model addressing risk factors of suicide. The most important factors included previous suicidal ideation or behavior, feeling downhearted, doing activities less carefully or accomplishing less because of emotional problems, younger age, lower educational achievement, and recent financial crisis.

Meaning  By using an algorithmic approach to analyze survey data and identify new risk factors, this study offers new avenues to guide future clinical assessment and development of suicide risk scales in the general population.

Importance  Because more than one-third of people making nonfatal suicide attempts do not receive mental health treatment, it is essential to extend suicide attempt risk factors beyond high-risk clinical populations to the general adult population.

Objective  To identify future suicide attempt risk factors in the general population using a data-driven machine learning approach including more than 2500 questions from a large, nationally representative survey of US adults.

Design, Setting, and Participants  Data came from wave 1 (2001 to 2002) and wave 2 (2004 to 2005) of the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC). NESARC is a face-to-face longitudinal survey conducted with a national representative sample of noninstitutionalized civilian population 18 years and older in the US. The cumulative response rate across both waves was 70.2% resulting in 34 653 wave 2 interviews. A balanced random forest was trained using cross-validation to develop a suicide attempt risk model. Out-of-fold model prediction was used to assess model performance, including the area under the receiver operator curve, sensitivity, and specificity. Survey design and nonresponse weights allowed estimates to be representative of the US civilian population based on the 2000 census. Analyses were performed between May 15, 2019, and June 10, 2020.

Main Outcomes and Measures  Attempted suicide in the 3 years between wave 1 and wave 2 interviews.

Results  Of 34 653 participants, 20 089 were female (weighted proportion, 52.1%). The weighted mean (SD) age was 45.1 (17.3) years at wave 1 and 48.2 (17.3) years at wave 2. Attempted suicide during the 3 years between wave 1 and wave 2 interviews was self-reported by 222 of 34 653 participants (0.6%). Using survey questions measured at wave 1, the suicide attempt risk model yielded a cross-validated area under the receiver operator characteristic curve of 0.857 with a sensitivity of 85.3% (95% CI, 79.8-89.7) and a specificity of 73.3% (95% CI, 72.8-73.8) at an optimized threshold. The model identified 1.8% of the US population to be at a 10% or greater risk of suicide attempt. The most important risk factors were 3 questions about previous suicidal ideation or behavior; 3 items from the 12-Item Short Form Health Survey, namely feeling downhearted, doing activities less carefully, or accomplishing less because of emotional problems; younger age; lower educational achievement; and recent financial crisis.

Conclusions and Relevance  In this study, after searching through more than 2500 survey questions, several well-known risk factors of suicide attempt were confirmed, such as previous suicidal behaviors and ideation, and new risks were identified, including functional impairment resulting from mental disorders and socioeconomic disadvantage. These results may help guide future clinical assessment and the development of new suicide risk scales.

Between 2001 and 2017, suicide mortality in the US increased by 31% from 10.7 to 14.0 cases per 100 000 population.¹ Previous studies estimate that between 8.5% and 13% of all suicide attempts are fatal^2-4 and that around 3% of index attempts lead to death.⁵ Roughly half of suicide deaths do not occur during a first attempt.^5,6 Thus, preventing nonfatal attempts presents an opportunity for early intervention in a substantial number of people at high risk of suicide⁷ and for decreasing the public health burden of suicide behaviors.

Despite extensive work over the last 50 years to improve prediction of suicide attempt, a meta-analysis of 365 studies concluded that using known suicide risk factors leads to only slightly better than chance prediction (weighted area under the receiver operating characteristic curve [AUC], 0.58).⁸ Machine learning methods and big data sources, such as electronic health records and social media text monitoring, have led to substantial improvements in predicting suicide attempt in clinical samples (AUC, 0.71-0.93).^9-14 However, most of the published literature on nonfatal suicide attempt prediction has focused on high-risk patients who have received mental health treatment.^15,16 More than one-third of people making nonfatal suicide attempts do not receive mental health treatment,^17,18 and those that engage in mental health treatment only represent one-third of all fatal suicide attempts in the US.^{11,15,17,19-21} These findings underscore the importance of extending suicide attempt prediction models beyond high-risk populations to the general adult population.^22,23

In the present study, we aimed to identify important risk factors of future suicide attempt in the general population by taking advantage of the richness of the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC) data set using an explanatory machine learning model. We extended prior research in 3 important directions. First, we used a large, nationally representative longitudinal sample to identify risk factors of suicide attempt in the general population. Second, we used an extensive assessment instrument that includes detailed evaluation of substance use, psychiatric disorders, and symptoms that are not routinely available in electronic health records or administrative data. Third, we incorporated class imbalance as a feature in our model to address the limitations of more generic algorithms, as few studies have previously done this.¹⁵ Overall, we expected to confirm previously identified risk factors found in clinical samples and, more importantly, identify new risk factors to expand our understanding of the etiology of suicide attempts.

Data were drawn from NESARC, a face-to-face survey conducted with a nationally representative sample of the US adult population by the National Institute on Alcoholism and Alcohol Abuse.²⁴ The target population included the noninstitutionalized civilian population 18 years and older in the US. Wave 1 NESARC survey data (2001 to 2002) and self-reported nonfatal suicide attempts at follow-up 3 years later (wave 2, 2004 to 2005)²⁵ were used to build a suicide attempt risk model. The cumulative response rate at wave 2 was 70.2%, resulting in 34 653 wave 2 interviews. Survey design and nonresponse weights allowed estimates to be representative of the US civilian population based on the 2000 Census.²⁶ Data were analyzed from May 15, 2019, to June 10, 2020. The research protocol received full human subjects review and approval from the US Census Bureau and the Office of Management and Budget. All participants provided written informed consent. The study followed Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

At wave 1, participants were assessed using the Alcohol Use Disorder and Associated Disabilities Interview Schedule DSM-IV (AUDADIS-IV),^27,28 a lay-administered structured interview to assess alcohol use, drug use, and mental health disorders according to DSM-IV criteria. Axis I disorders evaluated in the past 12 months included substance use disorders (alcohol use, drug use, and nicotine dependence), mood disorders (major depressive disorder, dysthymic disorder, and bipolar disorder), anxiety disorders (panic disorder, social anxiety disorder, specific phobia, and generalized anxiety disorder), and pathological gambling. Axis II disorders included avoidant, dependent, obsessive-compulsive, histrionic, paranoid, schizoid, and antisocial personality disorders assessed on a lifetime basis. Demographic and background information was collected. Response patterns for each of the 14 sections of the survey are summarized in eTable 1 in the Supplement. The test-retest reliability of AUDADIS-IV and its validity for measuring DSM-IV mental disorders is good to excellent for substance use (κ = 0.51-0.74) and fair to good for other disorders (κ = 0.40-0.67).^27,29-31

The wave 1 survey contained 2805 separate questions. To reduce interview burden, participants skipped entire sections based on their responses to gate questions. Additionally, there were 180 derived variables for DSM-IV past-year, prior-to-past-year, and lifetime diagnoses of mental disorders, including personality disorders. For each wave 1 participant, there were between 643 and 2985 available features.

At wave 2, a similar face-to-face structured interview follow-up was conducted. The primary outcome was retrospective and was defined as having attempted suicide at any point in the 3 years prior to the wave 2 interview. This variable was derived by combining responses to the wave 2 questions: “In your entire life, did you ever attempt suicide?” and, if affirmative, “How old were you the first time?” and “How old were you the most recent time?”³² If the most recent suicide attempt occurred within the last 3 years, the participant was considered to have met the primary outcome; otherwise the participant was not considered to have met the primary outcome. At wave 2, a total of 222 participants confirmed having attempted suicide since the wave 1 interview.

We performed an initial data analysis³³ and addressed the survey structural missingness by using the missing-indicator method^34,35 described in the eMethods in the Supplement. We used balanced random forest (BRF) to build a model to identify factors associated with suicide attempts by taking the processed 2978 wave 1 features to classify dichotomous suicide attempt at wave 2. BRF has better performance than regular random forest plotting for classification models with class-imbalanced data.^36,37 As detailed in the eMethods in the Supplement, we tuned the BRF parameters by using 10-fold cross-validation and further validated our classification model by using nested cross-validation.³⁸

We summarized the final model’s performance by aggregating the out-of-fold classifications of our optimal model and used this aggregated probability (threshold) to calculate an out-of-fold AUC. We weighted our results based on design and nonresponse weights to allow our estimates to be representative of the US civilian population based on the 2000 Census. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), alarms per 100 evaluations, and number needed to evaluate (NNE) to find 1 new suicide attempt case were examined against the threshold value.

To quantify model variable importance, we calculated the decrease in classification ability after any individual feature was permuted (ie, no longer used in the suicide attempt model) across the data set.^39,40 The importance measure was scaled between 0 and 100 by subtracting the smallest importance from all observations and dividing by the largest importance. To facilitate interpretation of suicide attempt risk, we defined 4 interpretable risk severity groups that could be used as a reference of suicide attempt subgroups in the US adult population based on the BRF model as defined in the eMethods in the Supplement. The first risk severity group was based on the Youden J statistic (sensitivity + specificity − 1), which was used to determine low risk vs anything higher than low risk. The cut point for this risk severity group was defined at a sensitivity of 85.3% and a specificity of 73.3%. This cut point placed 73.1% of the US population in the low-risk category. The remaining 26.9% of the population was categorized according to 2 meaningful population benchmarks: the cut point corresponding to the top decile of risk across the sample (differentiating the medium-risk group from the high-risk group) and the cut point corresponding to a PPV of 10% or greater (designating the very high-risk group). Sample weighting identified 7.5% of the US population as high risk and 1.8% as very high risk. The PPV of the very high-risk group was 10.4% (Table 1). We calculated summary statistics of suicide attempt broken down by risk groups. Finally, we quantified the risk associated with each of the top-performing risk factors by generating response plots of the distribution of probabilities for all observations in the 4 empirically derived risk groups.

We further validated our model in 3 ways. First, we calculated classification performance stratified by time-to-suicide attempt from the first interview. Second, we stratified classification performance across sex, age, self-reported race/ethnicity (White vs non-White), and income to test the robustness against demographic characteristics. Lastly, we examined erosion in model accuracy with fewer features by running additional BRFs using only the top 5 and 10 risk factors selected from the random forest importance measure.

Of 34 653 participants, 20 089 were female. The weighted mean (SD) age was 45.1 (17.3) years at wave 1 and 48.2 (17.3) years at wave 2. We found that 222 participants (0.6%) attempted suicide. The out-of-sample AUC for the best model, including all wave 1 features, was 0.857 (range, 0.803-0.909) with a sensitivity of 85.3% (95% CI, 79.8-89.7) and a specificity of 73.3% (95% CI, 72.8-73.8) at an optimized threshold. The optimal cross-validated number of variables to sample at each fold was 1700, representing 57.1% of all features. The out-of-sample generalizability, defined as the correlation between our final model and our nested cross-validated model, was 0.997. Figure 1 presents the distribution of model-calculated risk scores across the whole sample, stratified by whether participants reported a suicide attempt at wave 2. Suicide attempt risk strata (low, medium, high, very high) are summarized in Table 1. Based on our model, 73.1% of the US population are at low risk, 17.5% at medium risk, 7.6% at high risk, and 1.8% at very high risk of suicide attempt. Within these categories, 138 of 222 individuals (62.2%) who attempted suicide between waves 1 and 2 were in the high-risk or very high-risk groups based on their wave 1 survey responses and 32 (14.4%) were classified as being low risk.

Figure 2 displays sensitivity, specificity, PPV, NPV, alarms per 100 evaluations, and NNE to find 1 new suicide attempt case across various classification thresholds. For each plot, we labeled the 3 thresholds used to classify the 4 risk groups. If using the very high-risk group as a classification threshold, there would be a PPV of 10.4%, an NPV of 99.6%, 2 alarms, and an NNE of 10. If using the high-risk group, or the top decile of risk, as a threshold for classification, there would be a PPV of 3.9%, an NPV of 99.7%, 10 alarms, and an NNE of 26. Using the threshold that optimizes the Youden statistic (the low-risk group), there would be a PPV of 2.0%, an NPV of 99.9%, 27 alarms, and an NNE of 51.

Table 2 shows the 20 most important variables from the BRF model. The 3 most important risk factors were whether the individual felt at any point like they wanted to die, whether they thought about committing suicide, and previous suicide attempt. Several of the most important variables were associated with past month low energy and mood periods, such as feeling downhearted, feeling less accomplished, or paying less attention to work or other activities. Other features identified as important were age, family income and financial crisis, marital status, education level, paternal alcohol misuse, and parental separation. The eFigure in the Supplement shows the distribution of model-calculated scores as a function of the top 10 most important variables identified by the BRF algorithm allowing interpretation for how each variable was associated with suicide attempt.

We conducted a series of sensitivity and complementary analyses. eTable 2 in the Supplement shows that the classification ability of the model decreased as time-to-suicide attempt from the first interview increased. Among participants who attempted suicide within the first year, 21 (50.5%) were classified as very high risk. Among participants who attempted suicide between the first and second year, 16 were classified as very high risk (33.1%), while among participants who attempted suicide between the second and third year, 21 (30.3%) were classified as very high risk. Finally, among participants who attempted suicide between the third year and follow-up, 11 (16.48%) were classified as very high risk.

Second, we examined the classification ability of our model across demographic characteristics. As shown in eTable 3 in the Supplement, the AUC was 0.808 (95% CI, 0.765-0.851) for participants aged 18 to 36 years, 0.867 (95% CI, 0.827-0.906) for those aged 37 to 53 years, and 0.872 (95% CI, 0.800-0.945) for those 54 years and older. We found that the AUC was 0.850 (95% CI, 0.813-0.886) for female individuals (weighted proportion, 52.1%) and 0.872 (95% CI, 0.840-0.904) for male individuals. The AUC was 0.877 (95% CI, 0.845-0.909) for White participants and 0.831 (95% CI, 0.788-0.873) for non-White participants. Finally, the AUC was 0.845 (95% CI, 0.813-0.877) for participants with an income lower than $20 000, 0.848 (95% CI, 0.786-0.910) for those with an income between $20 000 and $34 999, 0.794 (95% CI, 0.676-0.911) for those with incomes between $35 000 and $69 999, and 0.944 (95% CI, 0.893-0.994) for those with incomes higher than $70 000.

Finally, we measured the decrease in model accuracy when fewer features were included by building new separate BRF models with the top 5 and 10 most important variables previously found using the entire feature set. The out-of-sample AUCs (SE) for these models were 0.818 (0.017) and 0.845 (0.016), respectively.

We built a model to classify nonfatal suicide attempts using a large, nationally representative sample of US adults. It confirmed several well-known risk factors of suicide attempt and identified several new ones. When tested outside the training set, our model performed at levels similar to models restricted to data from high-risk mental health patients^8,9 for the full sample and when stratified by demographic characteristics, indicating its robustness. Its classification power decreased with time elapsed from the baseline interview, providing an indirect measure of its validity. These results are encouraging given the recent emphasis on models in the general adult population using rich data sets and their usefulness to develop precision treatment rules for individuals who attempt suicide.^15,22,23

We found significant conceptual overlap of our most important risk factors with items commonly used in suicide risk scales. In accord with previous findings,^41-45 the strongest risk factors of future suicide attempts were related to previous suicidal behaviors. For example, whether the individual felt at any point like they wanted to die is covered in the Patient Health Questionnaire-9,⁴⁶ the Columbia Suicide Severity Rating Scale,⁴⁷ and Beck’s Scale of Suicide Ideation.⁴⁸ Previous suicide ideation is covered in Beck’s Scale of Suicide Ideation, the SAD PERSONS Scale,⁴⁹ and the Suicide Assessment Scale,⁵⁰ while previous suicide attempt is covered in the SAD PERSONS scale and the Columbia Suicide Severity Rating Scale. Feeling downhearted and depressed is covered in the Patient Health Questionnaire-9⁴⁶ (item 2) as well as the Beck Depression Inventory (item 1).^51,52

Our results extend prior work by revealing the predictive value of variables related to functional impairment resulting from mental disorders, which are not generally covered in screening tools for suicide risk assessment. The questions identified from the impairment construct, including accomplishing less than you like and not performing work or other activities as carefully as usual are covered in the 12-Item Short Form Health Survey.^53-55 These findings may offer new avenues to improve suicidal behavior prediction through functional assessment. We note that sex was not one of the most important variables, suggesting that sex differences in other risk factors are likely to mediate the difference in suicide attempt prevalence across sexes.^7,56,57

Other important novel risk factors identified were related to socioeconomic disadvantage. Lower educational level and experiencing a financial crisis in the last year were among the 10 most important variables. Seeking to alleviate the economic and emotional effects of financial crises might be an important aspect of suicide risk prevention, particularly in the context of deaths of despair formulations of suicide risk.^58,59 This is of particular contemporary relevance, given increased unemployment and economic stress in the US related to the coronavirus disease 2019 pandemic.⁶⁰ Our study identifies an individual-level association between economic strain with suicide attempt risk that extends beyond the findings of previous studies, showing a population-level association between economic recessions and increased suicide rates,^61-63 a link between financial debt and suicide ideation,⁶⁴ and case-control research linking unemployment and personal debt to suicide risk.⁶⁵ Although this association has previously been reported in NESARC,⁶⁶ our data-driven results highlight this risk factor as one of the most important for suicide attempt in the general population.

We incorporated technical advances in the modeling by using BRF to address the extreme class imbalance and by using the missing-indicator approach to address gate questions and skip patterns common to survey data. We ensured population-level generalizability by incorporating a complex survey design and sampling weights. The algorithms in this study may be useful for the analysis of other large survey data sets. Our methods may have wide applications given the NIH’s recent decision to link research samples to the National Death Index⁶⁷ and the greater availability of longitudinal mortality outcomes for cross-sectional surveys.

This study had some limitations. First, we only had data from participants who were 18 years and older, and some of the risk factors identified, such as financial crisis, might only be relevant to adult populations. Furthermore, suicide risk is highest for people aged 15 to 25 years.⁶⁸ Second, we did not have information about suicide attempts among participants lost to follow-up (ie, wave 2 nonresponders, including participants who died of suicide), which would have enhanced our ability to detect differences between fatal and nonfatal suicide attempts. Nevertheless, we found lower rates of prior suicidal behaviors and ideation at wave 1 among wave 2 nonresponders, suggesting that selection bias related to suicide attempts is likely small (eTable 4 in the Supplement). Third, there is potential for misclassification of suicide attempt. The reliability of self-reported suicide attempt over such a long recall period may be uncertain and may be affected by participants’ willingness to disclose previous attempts in a face-to-face interview.^69,70 However, that our findings confirm previous risk factors adds validity to our results. Fourth, our study examined occurrence of suicide attempts within 3 years of assessment. Exploration of shorter and more clinically relevant time horizons should also be evaluated. Furthermore, the association between risk factors and future suicide attempts may vary over time. Fifth, the data were collected from 2000 to 2005, and there may have been recent secular changes in risk factors of suicide attempts. Sixth, the survey was not collected to study suicide, and some important covariates, such as stress and adjustment disorders, were not included. Furthermore, wave 1 suicide symptoms were only asked of participants who endorsed depressed mood or anhedonia. Given the important role this item assumed as a risk factor of future suicide attempt, an item that was asked of everyone might have increased the accuracy of the models.

Our study demonstrates the ability of machine learning methods to generate powerful and parsimonious suicide attempt models in general adult population samples that build on and complement knowledge derived from clinical and high-risk samples. We confirmed several well-known risk factors of suicide attempts, such as previous suicidal behaviors and depression, while identifying new important risks. Specifically, functional impairment and socioeconomic disadvantage emerged as novel important factors of suicide attempt in the general population with lower educational level and recent financial crisis as an individual-level risk of future suicide attempts. We hope that these results deepen our understanding of the etiology of suicide attempts in adults and improve suicidal behavior prediction by identifying new risk variables to guide clinical assessment and development of suicide risk scales.

Accepted for Publication: November 6, 2020.

Published Online: January 6, 2021. doi:10.1001/jamapsychiatry.2020.4165

Corresponding Author: Ángel García de la Garza, BA, Department of Biostatistics, Columbia Mailman School of Public Health, 722 W 168th St, Room 656, New York, NY 10032 (ag3914@cumc.columbia.edu).

Author Contributions: Mr García de la Garza had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: García de la Garza, Blanco, Wall.

Drafting of the manuscript: García de la Garza, Wall.

Critical revision of the manuscript for important intellectual content: García de la Garza, Blanco, Olfson.

Statistical analysis: García de la Garza, Wall.

Conflict of Interest Disclosures: None reported.

Funding/Support: The National Epidemiologic Survey on Alcohol and Related Conditions was sponsored by the National Institute on Alcohol Abuse and Alcoholism and funded in part by the National Institute on Alcohol Abuse and Alcoholism Intramural Program. Drs Olfson and Wall were supported by the New York State Psychiatric Institute.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The views and opinions expressed here are those of the authors and should not be construed to represent the views of any of the sponsoring organizations or agencies or the US government.