Accuracy Requirements for Cost-effective Suicide Risk Prediction Among Primary Care Patients in the US

Key Points

Question  How accurate must suicide risk prediction models be to be cost-effective in targeting interventions to high-risk individuals in a US primary care population?

Findings  This economic evaluation found that suicide risk prediction could be cost-effective for targeting a safety planning and telephone call intervention if its specificity was 95% or higher and its sensitivity was 17% or higher, corresponding to a positive predictive value of 1% or greater. For a more expensive cognitive behavioral therapy intervention, the required positive predictive value was 2% or greater.

Meaning  Existing suicide risk prediction models may be accurate enough to be cost-effective in US health care settings.

Importance  Several statistical models for predicting suicide risk have been developed, but how accurate such models must be to warrant implementation in clinical practice is not known.

Objective  To identify threshold values of sensitivity, specificity, and positive predictive value that a suicide risk prediction method must attain to cost-effectively target a suicide risk reduction intervention to high-risk individuals.

Design, Setting, and Participants  This economic evaluation incorporated published data on suicide epidemiology, the health care and societal costs of suicide, and the costs and efficacy of suicide risk reduction interventions into a novel decision analytic model. The model projected suicide-related health economic outcomes over a lifetime horizon among a population of US adults with a primary care physician. Data analysis was performed from September 19, 2019, to July 5, 2020.

Interventions  Two possible interventions were delivered to individuals at high predicted risk: active contact and follow-up (ACF; relative risk of suicide attempt, 0.83; annual health care cost, $96) and cognitive behavioral therapy (CBT; relative risk of suicide attempt, 0.47; annual health care cost, $1088).

Main Outcomes and Measures  Fatal and nonfatal suicide attempts, quality-adjusted life-years (QALYs), health care sector costs and societal costs (in 2016 US dollars), and incremental cost-effectiveness ratios (ICERs) (with ICERs ≤$150 000 per QALY designated cost-effective).

Results  With a specificity of 95% and a sensitivity of 25%, primary care–based suicide risk prediction could reduce suicide death rates by 0.5 per 100 000 person-years (if used to target ACF) or 1.6 per 100 000 person-years (if used to target CBT) from a baseline of 15.3 per 100 000 person-years. To be cost-effective from a health care sector perspective at a specificity of 95%, a risk prediction method would need to have a sensitivity of 17.0% or greater (95% CI, 7.4%-37.3%) if used to target ACF and 35.7% or greater (95% CI, 23.1%-60.3%) if used to target CBT. To achieve cost-effectiveness, ACF required positive predictive values of 0.8% for predicting suicide attempt and 0.07% for predicting suicide death; CBT required values of 1.7% for suicide attempt and 0.2% for suicide death.

Conclusions and Relevance  These findings suggest that with sufficient accuracy, statistical suicide risk prediction models can provide good health economic value in the US. Several existing suicide risk prediction models exceed the accuracy thresholds identified in this analysis and thus may warrant pilot implementation in US health care systems.

Suicide is the 10th most common cause of death in the US, with 48 344 deaths by suicide in 2018.¹ Despite substantial efforts at public education and prevention, suicide rates continue to increase; between 2006 and 2015, the estimated incidence of suicide attempt increased by 10%.²

Multiple interventions have evidence of efficacy in reducing suicide risk, including cognitive behavioral therapy,^3,4 dialectical behavioral therapy,³ and active contact and follow-up interventions.^5,6 However, identifying people at risk for suicide who would benefit from these interventions is challenging: fewer than half of people who die by suicide in the US have a previously identified mental health condition,⁷ and only one-third are seen by a specialized mental health care practitioner in the year preceding their suicide.^8,9

To address this challenge, multiple research groups have developed suicide risk prediction models using electronic health record data.^10-13 A recent systematic review¹⁰ identified 17 such predictive models that have been described in the literature. In reviewing the accuracy of these models, the authors highlighted their low positive predictive value (PPV) (most models had a PPV <1% for predicting suicide mortality) and suggested that this attribute might preclude any practical application of such models. Subsequent commentators have challenged this conclusion, noting that the health economic value of risk prediction depends not only on its accuracy but also on the consequences of the condition it predicts and the efficacy of interventions for that condition; within this cost-benefit framework, what constitutes an adequate accuracy for suicide risk prediction is unknown.¹⁴

To address this evidence gap, we developed an economic evaluation that incorporated published data on suicide epidemiology, the health care and societal costs of suicide, and the costs and efficacy of suicide risk reduction interventions into a novel decision analytic model. We used this model to estimate threshold values of sensitivity, specificity, and PPV that a suicide risk prediction method must attain to be cost-effective among primary care patients in the US.

We used a decision analytic model to simulate use of a suicide risk prediction method to identify high-risk individuals who would be offered 1 of 2 interventions to reduce suicide risk: an active contact and follow-up (ACF) intervention or a cognitive behavioral therapy (CBT) intervention. We chose these 2 options to capture the diversity of evidence-based interventions for suicide risk, with one (CBT) being intensive and costly and the other (ACF) being less intensive and less costly.^4,5 For each possible intervention, we varied the test characteristics of the risk prediction across a wide range of sensitivity and specificity values. For each set of test characteristics, we then used the model to simulate the clinical and economic effects of suicide risk prediction and intervention over a lifetime horizon.

For each simulation, we tracked the following outcomes: fatal and nonfatal suicide attempts, quality-adjusted life-years (QALYs), and health care sector and societal costs in 2016 US dollars. To determine their present value, future costs and QALYs were discounted at 3% annually.¹⁵ Using these outcomes, we calculated the incremental cost-effectiveness ratio (ICER) of suicide risk prediction and intervention as the ratio of its incremental cost to its incremental QALYs (compared with usual care).¹⁶ In the US, previous authors have advocated using ICER thresholds between $50 000 per QALY and $150 000 per QALY to define cost-effective health care interventions^17,18; for our primary analysis, we designated predictive methods with an ICER of $150 000 or less as cost-effective and explored a range of alternative cost-effectiveness thresholds. Data analysis was performed from September 19, 2019, to July 5, 2020.

On the basis of recommendations of the Second Panel on Cost-effectiveness in Health and Medicine, we report cost-effectiveness from both a health care sector perspective (including only health care–related costs) and a societal perspective (including additional costs, such as patient time and lost productivity); an impact inventory detailing each perspective is provided in eTable 1 in the Supplement.¹⁵ In describing our methods and results, we adhere to the 2013 Consolidated Health Economic Evaluation Reporting Standards (CHEERS) reporting guideline.¹⁹ As secondary research using only published data, this study was not regulated by the Massachusetts General Hospital Institutional Review Board and did not require consent from any participants.

We developed a state-transition model of suicide using Microsoft Excel; its structure is shown in eFigure 1 in the Supplement. The model divides the population into 1000 strata of true suicide risk, with 1/1000th of the population in each; the probability of suicide within a stratum during each 1-year model cycle is determined based on a logit-normal distribution of suicide risk. Those who make a suicide attempt have a specified probability of dying from the attempt, which does not vary by risk stratum. Finally, patients in all strata have an annual probability of dying of other causes, which increases as the population ages.

To simulate a risk prediction method, we directly specify its predictive accuracy rather than simulating the various clinical and demographic factors that real-world risk prediction methods incorporate.^11,12 Accuracy is specified using 2 variables: a risk threshold (eg, 95th percentile) and an accuracy reduction; the derivation and implementation of these variables are described in the eAppendix in the Supplement. By adjusting these variables, a wide range of sensitivity and specificity values can be simulated. For clarity, in reporting our results, we provide only sensitivity and specificity estimates rather than the underlying model variables used to produce those values.

Under a risk prediction and intervention strategy, any patient deemed high risk (ie, positive) is offered a risk reduction intervention; a specified fraction of patients will accept this intervention. Those who accept the intervention incur an intervention-specific cost and have their risk of suicide reduced according to a specified relative risk, which is constant across risk strata.

Model input values, uncertainty intervals, and sources are given in Table 1^4,15,20-31; their derivation is described below.

We calibrated the population suicide risk distribution to reproduce the following outcomes: suicide attempt and suicide death rates of 175 per 100 000 person-years and 15 per 100 000 person-years, derived from individuals 15 years or older in national emergency department and inpatient databases,²⁰ and fraction of suicide attempts preceded by a prior attempt of 54%, drawn from the National Epidemiologic Survey on Alcohol and Related Conditions.²¹ The calibrated risk distribution is given in Table 1.^4,15,20-31 We specified an 8.81% probability of death per suicide attempt, derived from the aforementioned national database study.²⁰

We simulated a population with mean (SD) age of 48.8 (17.2) years based on the age distribution of primary care patients in a large US health system.²² We calculated non–suicide-related mortality rates for this population using age-specific all-cause mortality rates from the 2017 US life tables²³ and age-specific suicide mortality rates from the WONDER (Wide-ranging Online Data for Epidemiologic Research) database of the Centers for Disease Control and Prevention (CDC).³² We estimated mean health-related quality of life for a US primary care population from a national analysis using the EuroQol-5 Dimension (EQ-5D) questionnaire, a validated rating scale that assesses health status across 5 dimensions; the mean EQ-5D utility was 0.866.²⁴

We used the Bureau of Economic Analysis’s medical care expenditure indexes to inflate costs from earlier years to 2016 US dollars, the most recent year for which these indexes are available.^33,34 We deflated costs from later years to 2016 US dollars using the US Federal Reserve’s personal consumption expenditure indexes.³⁵

We derived health care and lost productivity costs associated with suicide attempts using data for individuals 20 years or older in the CDC’s WISQARS (Web-based Injury Statistics Query and Reporting System) database.²⁵ For nonfatal suicide attempts resulting in emergency department presentation and/or hospitalization, costs did not vary markedly with age; the mean health care cost was $10 830, and the mean lost productivity cost was $17 369. Because the costs of fatal suicide attempts varied more than 30-fold with age, we allowed these costs to change as our simulated population aged; mean health care costs started at $4528 and decreased to $4354, and mean lost productivity costs started at and $1 173 159 and decreased to $61 150 by the end of the lifetime horizon. Annual background health care costs, derived from the 2017 Medical Expenditure Panel Survey, ranged from $4016 to $11 740, depending on age.²⁶

We assumed that any patient with high predicted suicide risk would require in-person evaluation that consisted of an added or extended primary care appointment. We estimated the cost of this evaluation based on Centers for Medicare & Medicaid Services (CMS) reimbursement for a 15-minute primary care appointment (Current Procedural Terminology [CPT] code 99213) and a brief emotional assessment (CPT code 96127), yielding a health care cost of $76.²⁷ We estimated the lost productivity cost of this assessment using the February 2020 mean hourly earnings ($29)²⁸ and employment-to-population ratio (61.1%)²⁹; assuming the assessment would require 1 hour of patient time, lost productivity cost was $16.

To characterize the 2 suicide risk reduction interventions (ACF and CBT), we used efficacy estimates drawn from meta-analyses. For other variables (health care cost, societal cost, and uptake), no meta-analytic estimates were available; we instead derived estimates from single exemplar studies.^5,31 We assumed the reduction in suicide risk with any intervention would last for 12 months, which may underestimate longer-term benefits^5,31 but ensures a conservative assessment of the overall value of risk prediction and intervention.

For ACF interventions, a 2015 meta-analysis⁵ of 9 randomized clinical trials found a relative risk of suicide attempts of 0.83 (95% CI, 0.71-0.97); of the included trials, 6 involved postcard- or telephone-based follow-up, and 3 involved home visits by community nurses. Other variables were derived from a randomized clinical trial of safety planning and telephone follow-up among military personnel with suicidal ideation or attempts.³⁰ Participants completed a structured suicide safety plan on their initial encounter and a mean of 3.7 follow-up calls during 6 months; initial uptake of the intervention was 99.4%. We assumed that initial safety planning would be incorporated into the baseline assessment described above. For follow-up, we used a cost per telephone call of $27 based on CMS reimbursement for an 11- to 20-minute nonphysician telephone call (CPT code 98967).²⁷ These factors resulted in a total health care cost of $96; for comparison, a 2014 analysis³⁶ of follow-up calls after suicide-related hospitalizations estimated a cost of $11 to $140 for 2 to 5 follow-up calls. Finally, we assumed that each call would average 20 minutes of patient time, resulting in a lost productivity cost of $20.^28,29

For CBT, a 2017 meta-analysis⁴ of 10 randomized clinical trials found a relative risk of suicide attempts of 0.47 (95% CI, 0.30-0.73). Other variables were derived from a randomized clinical trial of brief CBT among military personnel with suicidal ideation or attempts.³¹ Of 169 participants who were not excluded, 17 dropped out or refused randomization, yielding an uptake of 89.9%. Participants completed a mean of 11.75 CBT sessions as part of the intervention, one of which was 90 minutes and the remainder 60 minutes; based on CMS reimbursement for individual therapy (CPT 90834 and 90837),²⁷ this equated to a health care cost of $1088. Finally, we assumed each session would require a mean of 2 hours of patient time, resulting in a lost productivity cost of $385.^28,29

To assess overall uncertainty in our results and establish CIs around estimated accuracy requirements, we performed probabilistic sensitivity analyses. In this process, 1000 individual model runs were performed using parameter values drawn at random from distributions reflecting their uncertainty. Aggregating model results across this range of input values provides an estimate of the uncertainty in results attributable to the joint uncertainty in input values.

To assess the robustness of our results to alternative assumptions, we also performed several univariate sensitivity analyses. In this process, we evaluated our model’s outcomes under the following input parameter assumptions. First, mean suicide risk in the simulated population decreased proportionally by 50% or increased by 100%. Second, suicide risk reduction after intervention increased or decreased proportionally by 50%; notably, this range includes the estimated efficacy of CBT obtained via meta-analysis with 1 outlier excluded (0.61).⁴ Third, intervention uptake varied between 50% and 100%; for comparison, among the studies included in our meta-analytic efficacy estimates, the lowest uptake was 51.1%.³⁷ Fourth, intervention cost increased or decreased proportionally by 50%.

Example outcomes based on a risk prediction method with 95% specificity and 25% sensitivity are given in Table 2. With usual care, the rate of suicide death was 15.34 per 100 000 person-years and the rate of suicide attempt was 174.15 per 100 000 person-years. If used to target ACF, this risk prediction method reduced the population’s suicide death rate by 0.52 and suicide attempt rate by 5.90 per 100 000 person-years; if used to target CBT, risk prediction reduced the population's suicide death rate by 1.56 and suicide attempt rate by 17.76 per 100 000 person-years. Mean QALYs increased by 0.0016 with ACF and 0.0046 with CBT. With both interventions, risk prediction decreased suicide-related costs but increased other health care costs. For risk prediction followed by ACF, ICERs were $107 000 per QALY from a health care sector perspective and $57 000 per QALY from a societal perspective; for risk prediction followed by CBT, ICERs were $221 000 per QALY from a health care sector perspective and $224 000 per QALY from a societal perspective.

Figure 1 shows the threshold accuracy values required for suicide risk prediction and intervention to become cost-effective (ie, achieve an ICER ≤$150 000 per QALY) from a health care sector perspective; accuracy estimates from several real-world risk prediction methods are included for comparison.^11-13 At a specificity of 95%, risk prediction followed by ACF needed to attain a sensitivity of 17% (95% CI, 7%-37%) to become cost-effective; risk prediction followed by CBT needed to attain a sensitivity of 36% (95% CI, 23%-60%). To achieve lower ICER values, accuracy requirements were more stringent (eFigure 2 in the Supplement); for example, to achieve an ICER of $50 000 per QALY or less, risk prediction followed by ACF required a specificity of 95% or higher and a sensitivity of 57% or higher.

Threshold values for PPV (predicting suicide attempt) are shown in Figure 2. Notably, the base case PPV threshold was stable across specificity values from 90% to 99%, ranging from 0.8% to 0.9% for ACF and 1.6% to 1.8% for CBT, under a health care sector perspective.

Results from a societal perspective are shown in eFigures 3 through 5 in the Supplement; the required accuracy to achieve cost-effectiveness was consistently lower than under a health care sector perspective.

Table 3 gives the threshold values of sensitivity, PPV predicting suicide attempt, and PPV predicting suicide death under a range of alternative input value assumptions, from a health care sector perspective. Several notable trends are apparent. Threshold PPV values differed substantially, depending on whether suicide attempt or suicide death was being predicted. For ACF, threshold PPV predicting suicide attempt ranged from 0.5% to 1.6% (base case, 0.8%), whereas threshold PPV predicting suicide death ranged from 0.05% to 0.14% (base case 0.07%). For CBT, threshold PPV predicting suicide attempt ranged from 0.9% to 3.4% (base case 1.7%), whereas threshold PPV predicting suicide death ranged from 0.1% to 0.3% (base case, 0.2%). In contrast, threshold PPV was quite stable to changes in population suicide risk; for example, for CBT, threshold PPV predicting suicide death varied from 0.14% to 0.15% when population risk was halved or doubled. Finally, under all cases, the accuracy required to achieve cost-effectiveness was lower under a societal perspective (eTable 2 in the Supplement).

This economic evaluation estimated threshold values of sensitivity, specificity, and PPV that suicide risk prediction methods must attain to enable cost-effective targeting of suicide risk reduction interventions. For a less intensive intervention, such as safety planning with follow-up telephone calls, risk prediction could be cost-effective, with a specificity of 95% and a sensitivity of 20%, equating to a PPV of approximately 1%. For a more intensive intervention, such as CBT, risk prediction would likely require a specificity of 95% and sensitivity greater than 35%, equating to a PPV of approximately 2%.

The identified PPV thresholds (1%-2%) may intuitively seem quite low. However, that these values are so low reflects the immense societal and individual consequences of suicide.^14,25 Notably, these PPV estimates are in line with risk thresholds used for primary prevention of other similarly impactful conditions. For comparison, US guidelines recommend offering anticoagulants to patients with atrial fibrillation whose annual risk of stroke exceeds 0.6% to 2%^38,39; offering bisphosphonates to patients with low bone density whose annual risk of hip fracture or major osteoporotic fracture exceeds 0.3% or 2%, respectively^40,41; and offering endocrine therapy to women whose annual risk of breast cancer exceeds 0.3%.^42,43

When judged against these benchmarks, many existing suicide risk prediction models perform favorably. For comparison, recent risk prediction models have reported the following sensitivities at 95% specificity: 26%, from a model developed in the Veterans Health Administration¹³; 33%, from a model developed in Massachusetts academic medical centers¹¹; and 48%, from a model developed in a multistate collaboration.¹² The corresponding PPVs were 0.1% to 0.3% for predicting suicide death^12,13 and 2% to 6%^11,12 for predicting suicide attempts; of note, even higher PPVs (>10%)¹² could be attained by increasing specificity above 99%, although with a consequent decrease in the proportion of cases identified. Overall, this study’s benchmarks suggest that any of these models could be adequate for targeting an ACF intervention, and some may be accurate enough to use for targeting CBT.

However, the results of this study do not definitively demonstrate that any specific risk prediction model is cost-effective, in light of a significant limitation: although this study evaluates risk prediction in an outpatient population, most data on the efficacy of risk reduction interventions are based on individuals seeking help at an emergency department, often after a suicide attempt.^4,5 Currently, no evidence is available on the efficacy of interventions in a proactively identified outpatient population, although a randomized clinical trial in this population is underway.⁴⁴ Until such data are available, the current analysis must rely on extrapolation. For this reason, using the current findings to guide large-scale implementation of any specific suicide risk prediction model would be premature, but short of this, the findings provide strong support for initiating real-world pilot studies of existing risk prediction models.

This analysis has several additional limitations in addition to the aforementioned shortcomings in the population that input data are drawn from. As a modeling analysis, the work elides some real-world complexities (eg, heterogeneity of suicide case-fatality rates by method and demographic factors²⁰) and requires assumptions about model structure and input data. Where possible, this study made conservative assumptions about the benefits of risk prediction and intervention (eg, assuming a 12-month duration of intervention efficacy), which may result in more stringent estimates of threshold accuracy values.

In addition, there is uncertainty around both the primary and secondary benefits and harms of risk reduction interventions. With respect to primary benefits (preventing suicide attempts and deaths), previous meta-analyses^4-6,45 have estimated varying effect sizes, with not all showing statistically significant benefits; this finding is likely attributable to variation in outcome measures (suicidal ideation vs suicide attempts vs suicide deaths) and the heterogeneity in the interventions themselves. This heterogeneity in interventions also makes it impossible to generate a single cost estimate that can be generalized to all risk reduction interventions; this issue was addressed by instead performing a thorough cost accounting from 2 specific exemplar interventions.^30,31 More broadly, the uncertainty in both cost and efficacy have been addressed by assessing a broad range of possible model inputs in sensitivity analyses.

With respect to secondary effects, there are no data from primary care populations to inform whether risk reduction interventions would produce benefits (eg, improvements in depression) or harms (eg, increased stigma). Previous studies^31,46-48 in emergency department patients have identified primarily benefits, including reductions in health care costs, medical retirement, and depressive symptoms. Because these secondary benefits were observed in selected patient populations (eg, after a suicide attempt or psychiatric hospitalization), the most conservative approach was to not assume equivalent benefits in primary care patients; however, this choice could underestimate or overestimate the value of risk reduction interventions to some extent.

Finally, the analysis did not account for several aspects of real-world implementation of risk reduction interventions, such as staff training, the up-front costs of integrating a risk prediction model into clinical workflows, and the effects of merging statistical prediction with practitioner assessment on the accuracy of risk assessment.^49-51 These factors were not relevant to this analysis given the primary objective of establishing accuracy benchmarks, but they will be important to consider when evaluating specific risk reduction programs.

This economic evaluation estimated threshold accuracy values that suicide risk prediction methods must attain to be cost-effective in the US. The study found that many existing risk prediction models exceed these accuracy requirements. Although the results do not conclusively demonstrate that any specific suicide risk prediction model is cost-effective, they suggest that current risk prediction models have achieved sufficient accuracy for health systems to move forward with pilot implementation projects.

Accepted for Publication: January 17, 2021.

Published Online: March 17, 2021. doi:10.1001/jamapsychiatry.2021.0089

Correction: This article was corrected on April 7, 2021, to change the word “variant” to “variable” in 3 places in the Methods section under Risk Assessment and Intervention.

Corresponding Author: Eric L. Ross, MD, Department of Psychiatry, Massachusetts General Hospital, Wang Ambulatory Care Center 812, 15 Parkman St, Boston, MA 02114 (eross9@mgh.harvard.edu).

Author Contributions: Dr Ross had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Ross, Zuromski, Reis, Smoller.

Acquisition, analysis, or interpretation of data: Ross, Zuromski, Nock, Kessler, Smoller.

Drafting of the manuscript: Ross, Reis.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Ross.

Administrative, technical, or material support: Zuromski.

Supervision: Reis, Smoller.

Conflict of Interest Disclosures: Dr Zuromski reported receiving grants from the National Institute of Mental Health during the conduct of the study. Dr Kessler reported receiving personal fees from DataStat Inc and consultant fees from Sage Pharmaceuticals and Takeda during the conduct of the study. Dr Smoller reported serving as an unpaid member of the Bipolar/Depression Research Community Advisory Panel of 23andMe and a member of the Leon Levy Foundation Neuroscience Advisory Board, being a Tepper Family Massachusetts General Hospital Research Scholar, and receiving an honorarium for an internal seminar at Biogen Inc. No other disclosures were reported.

Funding/Support: This work was supported by grants R25 MH094612 (Dr Ross) and R01MH117599 (Dr Smoller) from the National Institute of Mental Health, a gift from the Tommy Fuss Fund (Dr Smoller), and a grant from the Harvard Medical School (Dr Smoller).

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.