Predicting Firearm Suicide—Small Steps Forward With Big Data

Suicide remains a leading cause of death in the United States, with half of all suicide deaths by firearm. Reducing firearm access during times of suicide risk is a recommended approach for suicide prevention, but identification of those at risk remains difficult. Many who die by firearm suicide may not self-disclose or seek help from a medical professional. Laqueur et al¹ look instead to the question of whether firearm suicide risk might be predicted—and those at risk helped—at or around the time of firearm purchase. Using California’s database of nearly 5 million handgun purchases, they applied machine learning techniques to predict subsequent firearm suicide deaths among purchasers. They found that firearm suicide occurred within a year for only 0.066% of handgun transactions, but 40% of suicides were among those with the highest risk score. Factors associated with subsequent firearm suicide death included older age at first firearm purchase, month of purchase, and shorter distance between home and point of firearm sale.

Machine learning models can provide insights where clinical trials or meta-analyses may be lacking. Laqueur et al¹ chose a supervised machine learning approach using a random forest model that has previously been applied in prior risk prediction of suicide or suicide behavior.² Benefits of such models are that they provide both classification and regression, helping identify those at risk while also showing important variables.³ The broad number of predictor variables here, including transaction data and purchaser demographic characteristics and geocoding, provide a rich number of features to evaluate. However, transaction history or death records may not provide the full story or context, and adding other data sources—mental health disorders, substance abuse, hospitalizations, or natural language processing of clinical notes—could prove valuable and improve risk prediction.⁴ Lastly, consideration of time frame and risk of suicide may be relevant to the type of intervention recommended.

It is also imperative to consider the potential uses of this model in relation to its accuracy. The model by Laqueur et al¹ shows that, depending on the random forest threshold, sensitivity varied from 0.39 to 0.75, specificity varied from 0.70 to 0.95, the negative predictive value was steady at 0.99, and the positive predictive value (PPV) was low at 0.002 to 0.145 (area under the curve, 0.81). A low PPV does not mean that the model is not useful, as this is common among most suicide prevention models, but a low PPV does warrant careful practice considerations to avoid inappropriate overidentification of risk. Laqueur et al¹ focus on the proportion of suicides that fall within high-risk ventiles, but this ignores the size of the pool of purchasers within each ventile. A feasible and cost-effective prevention program using the proposed model would need to incorporate information on the number of purchases within each ventile. A low-cost program, like a flyer or an email, could easily be delivered to a large group, but a higher-cost—and likely more effective—program could be cost prohibitive except for the highest risk group.

There are important caveats to the current approach, including that the analysis was only for handgun transactions and that the gun purchased may not have been the one used for the suicide. Most importantly, though, is the caveat to proceed cautiously in how such a model might be applied, beyond the considerations about PPV mentioned above. Specifically, the approach does not currently have strong enough predictive abilities to require its use at the point of sale. There is growing evidence that many firearm retailers are supportive of suicide prevention efforts and willing to engage in voluntary programs for customer education or temporary firearm storage.^5,6 However, retailers have expressed concerns about liability in the case of a suicide or homicide after return of stored firearms, suggesting that they may also have concerns about being required to use a predictive model prior to approving a transaction. Importantly, the low PPV and other data quality limitations (eg, in matching deaths to purchases) would indicate that this model should not be used as the main factor in recommending extreme risk protection orders or other legal interventions. Furthermore, it is unclear whether the data used in developing this model are available in real time around the time of sale (or within the 10-day waiting period) such that they could be used for individual level predictions, as the discussion suggests.

These concerns aside, creative approaches to firearm suicide prevention are urgently needed. Harnessing the power of big data for prediction, coupled with a thoughtful, collaborative approach to real-world application, may help prevent future firearm suicides.

Published: July 11, 2022. doi:10.1001/jamanetworkopen.2022.23758

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2022 Betz ME et al. JAMA Network Open.

Corresponding Author: Marian E. Betz, MD, MPH, School of Medicine, Department of Emergency Medicine, University of Colorado, 12401 E 17th Ave B215, Aurora, CO 80045 (marian.betz@ucdenver.edu).

Conflict of Interest Disclosures: Dr Goss reported receiving funding from the Agency for Healthcare Research and Quality during the conduct of the study and receiving consulting fees from Dispatch Health. Dr Goss’s financial interests have been reviewed by University of Colorado Hospital and University of Colorado School of Medicine in accordance with their institutional policies. No other disclosures were reported.