Could implementation of suicide prediction models reinforce and worsen racial/ethnic disparities in care?
In this diagnostic/prognostic study, 2 prediction models for suicide within 90 days were developed and validated in a retrospective study of 13 980 570 outpatient mental health visits. Both models accurately predicted suicide risk for visits by White, Hispanic, and Asian patients, but performance was poor for visits by Black and American Indian/Alaskan Native patients and patients without race/ethnicity reported.
This study suggests that implementation of either suicide prediction model would disproportionately benefit White, Hispanic, and Asian patients compared with Black and American Indian/Alaskan Native patients and patients with unrecorded race/ ethnicity.
Clinical prediction models estimated with health records data may perpetuate inequities.
To evaluate racial/ethnic differences in the performance of statistical models that predict suicide.
Design, Setting, and Participants
In this diagnostic/prognostic study, performed from January 1, 2009, to September 30, 2017, with follow-up through December 31, 2017, all outpatient mental health visits to 7 large integrated health care systems by patients 13 years or older were evaluated. Prediction models were estimated using logistic regression with LASSO variable selection and random forest in a training set that contained all visits from a 50% random sample of patients (6 984 184 visits). Performance was evaluated in the remaining 6 996 386 visits, including visits from White (4 031 135 visits), Hispanic (1 664 166 visits), Black (578 508 visits), Asian (313 011 visits), and American Indian/Alaskan Native (48 025 visits) patients and patients without race/ethnicity recorded (274 702 visits). Data analysis was performed from January 1, 2019, to February 1, 2021.
Demographic, diagnosis, prescription, and utilization variables and Patient Health Questionnaire 9 responses.
Main Outcomes and Measures
Suicide death in the 90 days after a visit.
This study included 13 980 570 visits by 1 433 543 patients (64% female; mean [SD] age, 42  years. A total of 768 suicide deaths were observed within 90 days after 3143 visits. Suicide rates were highest for visits by patients with no race/ethnicity recorded (n = 313 visits followed by suicide within 90 days, rate = 5.71 per 10 000 visits), followed by visits by Asian (n = 187 visits followed by suicide within 90 days, rate = 2.99 per 10 000 visits), White (n = 2134 visits followed by suicide within 90 days, rate = 2.65 per 10 000 visits), American Indian/Alaskan Native (n = 21 visits followed by suicide within 90 days, rate = 2.18 per 10 000 visits), Hispanic (n = 392 visits followed by suicide within 90 days, rate = 1.18 per 10 000 visits), and Black (n = 65 visits followed by suicide within 90 days, rate = 0.56 per 10 000 visits) patients. The area under the curve (AUC) and sensitivity of both models were high for White, Hispanic, and Asian patients and poor for Black and American Indian/Alaskan Native patients and patients without race/ethnicity recorded. For example, the AUC for the logistic regression model was 0.828 (95% CI, 0.815-0.840) for White patients compared with 0.640 (95% CI, 0.598-0.681) for patients with unrecorded race/ethnicity and 0.599 (95% CI, 0.513-0.686) for American Indian/Alaskan Native patients. Sensitivity at the 90th percentile was 62.2% (95% CI, 59.2%-65.0%) for White patients compared with 27.5% (95% CI, 21.0%-34.7%) for patients with unrecorded race/ethnicity and 10.0% (95% CI, 0%-23.0%) for Black patients. Results were similar for random forest models, with an AUC of 0.812 (95% CI, 0.800-0.826) for White patients compared with 0.676 (95% CI, 0.638-0.714) for patients with unrecorded race/ethnicity and 0.642 (95% CI, 0.579-0.710) for American Indian/Alaskan Native patients and sensitivities at the 90th percentile of 52.8% (95% CI, 50.0%-55.8%) for White patients, 29.3% (95% CI, 22.8%-36.5%) for patients with unrecorded race/ethnicity, and 6.7% (95% CI, 0%-16.7%) for Black patients.
Conclusions and Relevance
These suicide prediction models may provide fewer benefits and more potential harms to American Indian/Alaskan Native or Black patients or those with undrecorded race/ethnicity compared with White, Hispanic, and Asian patients. Improving predictive performance in disadvantaged populations should be prioritized to improve, rather than exacerbate, health disparities.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Coley RY, Johnson E, Simon GE, Cruz M, Shortreed SM. Racial/Ethnic Disparities in the Performance of Prediction Models for Death by Suicide After Mental Health Visits. JAMA Psychiatry. Published online April 28, 2021. doi:10.1001/jamapsychiatry.2021.0493
Coronavirus Resource Center
Customize your JAMA Network experience by selecting one or more topics from the list below.
Create a personal account or sign in to: