Accuracy Requirements for Cost-effective Suicide Risk Prediction Among Primary Care Patients in the US | Psychiatry and Behavioral Health | JAMA Psychiatry | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Threshold Specificity and Sensitivity Values at Which Suicide Risk Prediction Becomes Cost-effective From a Health Care Sector Perspective
Threshold Specificity and Sensitivity Values at Which Suicide Risk Prediction Becomes Cost-effective From a Health Care Sector Perspective

Solid lines indicate the threshold sensitivity that suicide risk prediction must attain before it becomes cost-effective (ie, achieves an incremental cost-effectiveness ratio ≤$150 000 per quality-adjusted life-year). The orange line shows results with an active contact and follow-up intervention; the blue line shows results with a cognitive behavioral therapy intervention. Orange and blue areas around these lines indicate 95% CIs around the threshold sensitivity estimates. Where the upper edge of the orange or blue area is not bounded by a dotted line, the upper limit of the 95% CI was not estimable; that is, for low enough specificity values, there is no achievable value of sensitivity for which there is 97.5% or greater likelihood that risk prediction would be cost-effective. Black icons indicate sensitivity estimates at a range of specificity values for several published suicide risk prediction models.

Figure 2.  Threshold Specificity and Positive Predictive Value (PPV) at Which Suicide Risk Prediction Becomes Cost-effective From a Health Care Sector Perspective
Threshold Specificity and Positive Predictive Value (PPV) at Which Suicide Risk Prediction Becomes Cost-effective From a Health Care Sector Perspective

Solid lines indicate the threshold PPV that suicide risk prediction must attain before it becomes cost-effective (ie, achieves an incremental cost-effectiveness ratio ≤$150 000 per quality-adjusted life-year). The orange line shows results with an active contact and follow-up intervention; the blue line shows results with a cognitive behavioral therapy intervention. Orange and blue areas around these lines indicate 95% CIs around the threshold PPV estimates. Where the upper edge of the orange or blue area is not bounded by a dotted line, the upper limit of the 95% CI was not estimable; that is, for low enough specificity values, there is no achievable value of PPV for which there is 97.5% or greater likelihood that risk prediction would be cost-effective.

Table 1.  Model Input Data
Model Input Data
Table 2.  Health Economic Outcomes With Suicide Risk Prediction Specificity of 95% and Sensitivity of 25%
Health Economic Outcomes With Suicide Risk Prediction Specificity of 95% and Sensitivity of 25%
Table 3.  Threshold Accuracy Values at Which Suicide Risk Prediction Becomes Cost-effective Given 95% Specificity, Health Care Sector Perspective, and Sensitivity Analyses
Threshold Accuracy Values at Which Suicide Risk Prediction Becomes Cost-effective Given 95% Specificity, Health Care Sector Perspective, and Sensitivity Analyses
1.
Xu  J, Murphy  SL, Kockanek  KD, Arias  E.  Mortality in the United States, 2018.   NCHS Data Brief. 2020;(355):1-8.PubMedGoogle Scholar
2.
Wang  J, Sumner  SA, Simon  TR,  et al.  Trends in the incidence and lethality of suicidal acts in the United States, 2006 to 2015.   JAMA Psychiatry. 2020;77(7):684-693. doi:10.1001/jamapsychiatry.2020.0596 PubMedGoogle ScholarCrossref
3.
D’Anci  KE, Uhl  S, Giradi  G, Martin  C.  Treatments for the prevention and management of suicide.   Ann Intern Med. 2019;171(5):334-342. doi:10.7326/M19-0869 PubMedGoogle ScholarCrossref
4.
Gøtzsche  PC, Gøtzsche  PK.  Cognitive behavioural therapy halves the risk of repeated suicide attempts: systematic review.   J R Soc Med. 2017;110(10):404-410. doi:10.1177/0141076817731904 PubMedGoogle ScholarCrossref
5.
Inagaki  M, Kawashima  Y, Kawanishi  C,  et al.  Interventions to prevent repeat suicidal behavior in patients admitted to an emergency department for a suicide attempt: a meta-analysis.   J Affect Disord. 2015;175:66-78. doi:10.1016/j.jad.2014.12.048 PubMedGoogle ScholarCrossref
6.
Riblet  NBV, Shiner  B, Young-Xu  Y, Watts  BV.  Strategies to prevent death by suicide: meta-analysis of randomised controlled trials.   Br J Psychiatry. 2017;210(6):396-402. doi:10.1192/bjp.bp.116.187799 PubMedGoogle ScholarCrossref
7.
Stone  DM, Simon  TR, Fowler  KA,  et al.  Trends in state suicide rates 1999-2016.   Morbidity and Mortality Weekly Report. 2018;67(22):617-624. doi:10.15585/mmwr.mm6722a1 PubMedGoogle ScholarCrossref
8.
Ahmedani  BK, Westphal  J, Autio  K,  et al.  Variation in patterns of health care before suicide: a population case-control study.   Prev Med. 2019;127:105796. doi:10.1016/j.ypmed.2019.105796 PubMedGoogle Scholar
9.
Luoma  JB, Martin  CE, Pearson  JL.  Contact with mental health and primary care providers before suicide: a review of the evidence.   Am J Psychiatry. 2002;159(6):909-916. doi:10.1176/appi.ajp.159.6.909 PubMedGoogle ScholarCrossref
10.
Belsher  BE, Smolenski  DJ, Pruitt  LD,  et al.  Prediction models for suicide attempts and deaths: a systematic review and simulation.   JAMA Psychiatry. 2019;76(6):642-651. doi:10.1001/jamapsychiatry.2019.0174 PubMedGoogle ScholarCrossref
11.
Barak-Corren  Y, Castro  VM, Javitt  S,  et al.  Predicting suicidal behavior from longitudinal electronic health records.   Am J Psychiatry. 2017;174(2):154-162. doi:10.1176/appi.ajp.2016.16010077 PubMedGoogle ScholarCrossref
12.
Simon  GE, Johnson  E, Lawrence  JM,  et al.  Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records.   Am J Psychiatry. 2018;175(10):951-960. doi:10.1176/appi.ajp.2018.17101167 PubMedGoogle ScholarCrossref
13.
Kessler  RC, Hwang  I, Hoffmire  CA,  et al.  Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans Health Administration.   Int J Methods Psychiatr Res. 2017;26(3):1-7. doi:10.1002/mpr.1575 PubMedGoogle ScholarCrossref
14.
Kessler  RC.  Clinical epidemiological research on suicide-related behaviors—where we are and where we need to go.   JAMA Psychiatry. 2019;76(8):777-778. doi:10.1001/jamapsychiatry.2019.1238 PubMedGoogle ScholarCrossref
15.
Sanders  GD, Neumann  PJ, Basu  A,  et al.  Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.   JAMA. 2016;316(10):1093-1103. doi:10.1001/jama.2016.12195 PubMedGoogle ScholarCrossref
16.
Briggs  AH, Claxton  C, Sculpher  MJ.  Decision Modelling for Health Economic Evaluation. Oxford University Press; 2006.
17.
Neumann  PJ, Cohen  JT, Weinstein  MC.  Updating cost-effectiveness: the curious resilience of the $50,000-per-QALY threshold.   N Engl J Med. 2014;371(9):796-797. doi:10.1056/NEJMp1405158 PubMedGoogle ScholarCrossref
18.
Anderson  JL, Heidenreich  PA, Barnett  PG,  et al.  ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines.   J Am Coll Cardiol. 2014;63(21):2304-2322. doi:10.1016/j.jacc.2014.03.016 PubMedGoogle ScholarCrossref
19.
Husereau  D, Drummond  M, Petrou  S,  et al; ISPOR Health Economic Evaluation Publication Guidelines-CHEERS Good Reporting Practices Task Force.  Consolidated Health Economic Evaluation Reporting Standards (CHEERS): explanation and elaboration: a report of the ISPOR Health Economic Evaluations Publication Good Reporting Practices Task Force.   Value Health. 2013;16(2):231-250. doi:10.1016/j.jval.2013.02.002 PubMedGoogle ScholarCrossref
20.
Conner  A, Azrael  D, Miller  M.  Suicide case-fatality rates in the United States, 2007 to 2014: a nationwide population-based study.   Ann Intern Med. 2019;171(12):885-895. doi:10.7326/M19-1324 PubMedGoogle ScholarCrossref
21.
Olfson  M, Blanco  C, Wall  M,  et al.  National trends in suicide attempts among adults in the United States.   JAMA Psychiatry. 2017;74(11):1095-1103. doi:10.1001/jamapsychiatry.2017.2582 PubMedGoogle ScholarCrossref
22.
Yarborough  BJH, Perrin  NA, Stumbo  SP, Muench  J, Green  CA.  Preventive service use among people with and without serious mental illnesses.   Am J Prev Med. 2018;54(1):1-9. doi:10.1016/j.amepre.2017.08.020 PubMedGoogle ScholarCrossref
23.
Arias  E, Xu  J.  United States life tables, 2017.   Natl Vital Stat Rep. 2019;68(7):1-66.PubMedGoogle Scholar
24.
Sullivan  PW, Ghushchyan  V.  Preference-Based EQ-5D index scores for chronic conditions in the United States.   Med Decis Making. 2006;26(4):410-420. doi:10.1177/0272989X06290495 PubMedGoogle ScholarCrossref
25.
Centers for Disease Control and Prevention. WISQARS cost of injury reports. Accessed March 19, 2020. https://wisqars.cdc.gov:8443/costT/
26.
Agency for Healthcare Research and Quality. Mean expenditure per person with expense (standard errors) by age groups, United States, 2017, Medical Expenditure Panel Survey. Accessed March 19, 2020. https://meps.ahrq.gov/mepstrends/hc_use/
27.
Centers for Medicare and Medicaid Services. Physician Fee Schedule. 2020. Updated February 3, 2021. Accessed March 19, 2019. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/PhysicianFeeSched
28.
Bureau of Labor Statistics. Current and real (constant 1982-1984 dollars) earnings for all employees on private nonfarm payrolls, seasonally adjusted, February 2020. Accessed March 19, 2019. https://www.bls.gov/news.release/realer.t01.htm
29.
Bureau of Labor Statistics. Employment status of the civilian population by sex and age, February 2020. Published 2020. Accessed March 19, 2020. https://www.bls.gov/news.release/empsit.t01.htm
30.
Stanley  B, Brown  GK, Brenner  LA,  et al.  Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department.   JAMA Psychiatry. 2018;75(9):894-900. doi:10.1001/jamapsychiatry.2018.1776 PubMedGoogle ScholarCrossref
31.
Rudd  MD, Bryan  CJ, Wertenberger  EG,  et al.  Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up.   Am J Psychiatry. 2015;172(5):441-449. doi:10.1176/appi.ajp.2014.14070843 PubMedGoogle ScholarCrossref
32.
Centers for Disease Control and Prevention. Underlying cause of death 1999-2018 on CDC WONDER online database, release in 2020. National Center for Health Statistics. Published 2020. Accessed March 16, 2020. https://wonder.cdc.gov/ucd-icd10.html
33.
Bureau of Economic Analysis. Medical Care Expenditure Indices from MEPS. 2019. Accessed September 5, 2019. https://www.bea.gov/national/health_care_satellite_account.htm
34.
Dunn  A, Grosse  SD, Zuvekas  SH.  Adjusting health expenditures for inflation: a review of measures for health services research in the United States.   Health Serv Res. 2018;53(1):175-196. doi:10.1111/1475-6773.12612 PubMedGoogle ScholarCrossref
35.
Federal Reserve Bank of St. Louis. Personal consumption expenditures. 2020. Accessed February 13, 2020. https://fred.stlouisfed.org/series/DPCERG3A086NBEA
36.
Richardson  JS, Mark  TL, McKeon  R.  The return on investment of postdischarge follow-up calls for suicidal ideation or deliberate self-harm.   Psychiatr Serv. 2014;65(8):1012-1019. doi:10.1176/appi.ps.201300196 PubMedGoogle ScholarCrossref
37.
Guthrie  E, Kapur  N, Mackway-Jones  K,  et al.  Randomised controlled trial of brief psychological intervention after deliberate self poisoning.   BMJ. 2001;323(7305):135-138. doi:10.1136/bmj.323.7305.135 PubMedGoogle ScholarCrossref
38.
January  CT, Wann  LS, Calkins  H,  et al.  2019 AHA/ACC/HRS focused update of the 2014 AHA/ACC/HRS guideline for the management of patients with atrial fibrillation: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the Heart R.   J Am Coll Cardiol. 2019;74(1):104-132. doi:10.1016/j.jacc.2019.01.011 PubMedGoogle ScholarCrossref
39.
Friberg  L, Rosenqvist  M, Lip  GYH.  Net clinical benefit of warfarin in patients with atrial fibrillation: a report from the Swedish atrial fibrillation cohort study.   Circulation. 2012;125(19):2298-2307. doi:10.1161/CIRCULATIONAHA.111.055079 PubMedGoogle ScholarCrossref
40.
Cosman  F, de Beur  SJ, LeBoff  MS,  et al; National Osteoporosis Foundation.  Clinician’s guide to prevention and treatment of osteoporosis.   Osteoporos Int. 2014;25(10):2359-2381. doi:10.1007/s00198-014-2794-2 PubMedGoogle ScholarCrossref
41.
Tosteson  ANA, Melton  LJ  III, Dawson-Hughes  B,  et al; National Osteoporosis Foundation Guide Committee.  Cost-effective osteoporosis treatment thresholds: the United States perspective.   Osteoporos Int. 2008;19(4):437-447. doi:10.1007/s00198-007-0550-6 PubMedGoogle ScholarCrossref
42.
Owens  DK, Davidson  KW, Krist  AH,  et al; US Preventive Services Task Force.  Medication use to reduce risk of breast cancer: US Preventive Services Task Force recommendation statement.   JAMA. 2019;322(9):857-867. doi:10.1001/jama.2019.11885 PubMedGoogle ScholarCrossref
43.
Visvanathan  K, Fabian  CJ, Bantug  E,  et al.  Use of endocrine therapy for breast cancer risk reduction: ASCO clinical practice guideline update.   J Clin Oncol. 2019;37(33):3152-3165. doi:10.1200/JCO.19.01472 PubMedGoogle ScholarCrossref
44.
Simon  GE, Beck  A, Rossom  R,  et al.  Population-based outreach versus care as usual to prevent suicide attempt: study protocol for a randomized controlled trial.   Trials. 2016;17(1):452. doi:10.1186/s13063-016-1566-z PubMedGoogle ScholarCrossref
45.
Milner  AJ, Carter  G, Pirkis  J, Robinson  J, Spittal  MJ.  Letters, green cards, telephone calls and postcards: systematic and meta-analytic review of brief contact interventions for reducing self-harm, suicide attempts and suicide.   Br J Psychiatry. 2015;206(3):184-190. doi:10.1192/bjp.bp.114.147819 PubMedGoogle ScholarCrossref
46.
Bernecker  SL, Zuromski  KL, Curry  JC,  et al.  Economic evaluation of brief cognitive behavioral therapy vs treatment as usual for suicidal US Army soldiers.   JAMA Psychiatry. 2020;77(3):256-264. doi:10.1001/jamapsychiatry.2019.3639 PubMedGoogle ScholarCrossref
47.
O’Connor  RC, Ferguson  E, Scott  F,  et al.  A brief psychological intervention to reduce repetition of self-harm in patients admitted to hospital following a suicide attempt: a randomised controlled trial.   Lancet Psychiatry. 2017;4(6):451-460. doi:10.1016/S2215-0366(17)30129-3 PubMedGoogle ScholarCrossref
48.
Carter  GL, Clover  K, Whyte  IM, Dawson  AH, D’Este  C.  Postcards from the EDge: 5-year outcomes of a randomised controlled trial for hospital-treated self-poisoning.   Br J Psychiatry. 2013;202(5):372-380. doi:10.1192/bjp.bp.112.112664 PubMedGoogle ScholarCrossref
49.
Whiting  D, Fazel  S.  How accurate are suicide risk prediction models? asking the right questions for clinical practice.   Evid Based Ment Health. 2019;22(3):125-128. doi:10.1136/ebmental-2019-300102 PubMedGoogle ScholarCrossref
50.
Brown  S, Iqbal  Z, Burbidge  F,  et al.  Embedding an evidence-based model for suicide prevention in the national health service: a service improvement initiative.   Int J Environ Res Public Health. 2020;17(14):1-13. doi:10.3390/ijerph17144920 PubMedGoogle ScholarCrossref
51.
Boudreaux  ED, Haskins  BL, Larkin  C,  et al.  Emergency department safety assessment and follow-up evaluation 2: an implementation trial to improve suicide prevention.   Contemp Clin Trials. 2020;95:106075. doi:10.1016/j.cct.2020.106075 PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    1 Comment for this article
    EXPAND ALL
    Cost effectiveness alone cannot justify suicide risk assessment
    Matthew Large |
    Dear Editor

    In a welcome analysis that shifts arguments about suicide risk assessment from suicide prediction to issues of resource allocation, Ross and associates argue that active care and follow-up and cognitive behavioural therapy (CBT) are both cost effective interventions when delivered to higher-risk primary health care patients. The authors reach this conclusion by balancing the expense of risk assessment and the resulting indicated treatments and the costs suicide behaviours among those judged to be high-risk. In one example they conclude that almost half (as prevented by CBT) of one quarter of all potential suicides (as identified by
    risk assessment) might be prevented by providing CBT to 5% of the population identified as higher-risk with no overall cost to the community. [1]

    It has previously been argued that suicide risk assessment can only be justified if an intervention delivered to higher risk people is both inexpensive and benign enough to be tolerated by the vast majority of false positives and when there are also sound reasons for not providing these same interventions to lower risk people, among whom many if not the majority of suicides occur. [2]

    Noting the generous assumptions made by Ross and associates (including that CBT can reduce suicide attempts by 53%) and that the CBT designed to protect against repeated suicide attempts is equally effective in reducing suicides in primary care, the authors have made an economic case for the treatment of approximately 1000 higher risk people in order to prevent one primary care suicide.

    However, authors do not seem to have realistically examined the broader arguments against offering such suicide reducing treatment to lower risk people who included three quarters of all suicides in their models. Healthcare is not usually allocated on the basis of profitability and any measure that could potentially save about 25,000 American lives a year and halve the national suicide rate should be seriously considered as a universal precaution rather than a risk indicated intervention.

    References

    1. Ross EL, Zuromski KL, Reis BY, Nock MK, Kessler RC, Smoller JW.
    Accuracy Requirements for Cost-effective Suicide Risk Prediction Among Primary Care Patients in the US. JAMA Psychiatry 2021 doi:10.1001/jamapsychiatry.2021.0089

    2. Large MDoes the emphasis on risk in psychiatry serve the interests of patients or the public? BMJ 2013 346:f857. doi: 10.1136/bmj.f857.
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Original Investigation
    March 17, 2021

    Accuracy Requirements for Cost-effective Suicide Risk Prediction Among Primary Care Patients in the US

    Author Affiliations
    • 1Department of Psychiatry, McLean Hospital, Belmont, Massachusetts
    • 2Department of Psychiatry, Massachusetts General Hospital, Boston
    • 3Department of Psychiatry, Harvard Medical School, Boston, Massachusetts
    • 4Department of Psychology, Harvard University, Cambridge, Massachusetts
    • 5Computational Health Informatics Program, Boston Children’s Hospital, Boston, Massachusetts
    • 6Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts
    • 7Psychiatric and Neurodevelopmental Genetics Unit, Massachusetts General Hospital, Boston
    JAMA Psychiatry. 2021;78(6):642-650. doi:10.1001/jamapsychiatry.2021.0089
    Key Points

    Question  How accurate must suicide risk prediction models be to be cost-effective in targeting interventions to high-risk individuals in a US primary care population?

    Findings  This economic evaluation found that suicide risk prediction could be cost-effective for targeting a safety planning and telephone call intervention if its specificity was 95% or higher and its sensitivity was 17% or higher, corresponding to a positive predictive value of 1% or greater. For a more expensive cognitive behavioral therapy intervention, the required positive predictive value was 2% or greater.

    Meaning  Existing suicide risk prediction models may be accurate enough to be cost-effective in US health care settings.

    Abstract

    Importance  Several statistical models for predicting suicide risk have been developed, but how accurate such models must be to warrant implementation in clinical practice is not known.

    Objective  To identify threshold values of sensitivity, specificity, and positive predictive value that a suicide risk prediction method must attain to cost-effectively target a suicide risk reduction intervention to high-risk individuals.

    Design, Setting, and Participants  This economic evaluation incorporated published data on suicide epidemiology, the health care and societal costs of suicide, and the costs and efficacy of suicide risk reduction interventions into a novel decision analytic model. The model projected suicide-related health economic outcomes over a lifetime horizon among a population of US adults with a primary care physician. Data analysis was performed from September 19, 2019, to July 5, 2020.

    Interventions  Two possible interventions were delivered to individuals at high predicted risk: active contact and follow-up (ACF; relative risk of suicide attempt, 0.83; annual health care cost, $96) and cognitive behavioral therapy (CBT; relative risk of suicide attempt, 0.47; annual health care cost, $1088).

    Main Outcomes and Measures  Fatal and nonfatal suicide attempts, quality-adjusted life-years (QALYs), health care sector costs and societal costs (in 2016 US dollars), and incremental cost-effectiveness ratios (ICERs) (with ICERs ≤$150 000 per QALY designated cost-effective).

    Results  With a specificity of 95% and a sensitivity of 25%, primary care–based suicide risk prediction could reduce suicide death rates by 0.5 per 100 000 person-years (if used to target ACF) or 1.6 per 100 000 person-years (if used to target CBT) from a baseline of 15.3 per 100 000 person-years. To be cost-effective from a health care sector perspective at a specificity of 95%, a risk prediction method would need to have a sensitivity of 17.0% or greater (95% CI, 7.4%-37.3%) if used to target ACF and 35.7% or greater (95% CI, 23.1%-60.3%) if used to target CBT. To achieve cost-effectiveness, ACF required positive predictive values of 0.8% for predicting suicide attempt and 0.07% for predicting suicide death; CBT required values of 1.7% for suicide attempt and 0.2% for suicide death.

    Conclusions and Relevance  These findings suggest that with sufficient accuracy, statistical suicide risk prediction models can provide good health economic value in the US. Several existing suicide risk prediction models exceed the accuracy thresholds identified in this analysis and thus may warrant pilot implementation in US health care systems.

    Introduction

    Suicide is the 10th most common cause of death in the US, with 48 344 deaths by suicide in 2018.1 Despite substantial efforts at public education and prevention, suicide rates continue to increase; between 2006 and 2015, the estimated incidence of suicide attempt increased by 10%.2

    Multiple interventions have evidence of efficacy in reducing suicide risk, including cognitive behavioral therapy,3,4 dialectical behavioral therapy,3 and active contact and follow-up interventions.5,6 However, identifying people at risk for suicide who would benefit from these interventions is challenging: fewer than half of people who die by suicide in the US have a previously identified mental health condition,7 and only one-third are seen by a specialized mental health care practitioner in the year preceding their suicide.8,9

    To address this challenge, multiple research groups have developed suicide risk prediction models using electronic health record data.10-13 A recent systematic review10 identified 17 such predictive models that have been described in the literature. In reviewing the accuracy of these models, the authors highlighted their low positive predictive value (PPV) (most models had a PPV <1% for predicting suicide mortality) and suggested that this attribute might preclude any practical application of such models. Subsequent commentators have challenged this conclusion, noting that the health economic value of risk prediction depends not only on its accuracy but also on the consequences of the condition it predicts and the efficacy of interventions for that condition; within this cost-benefit framework, what constitutes an adequate accuracy for suicide risk prediction is unknown.14

    To address this evidence gap, we developed an economic evaluation that incorporated published data on suicide epidemiology, the health care and societal costs of suicide, and the costs and efficacy of suicide risk reduction interventions into a novel decision analytic model. We used this model to estimate threshold values of sensitivity, specificity, and PPV that a suicide risk prediction method must attain to be cost-effective among primary care patients in the US.

    Methods
    Overview of Analysis

    We used a decision analytic model to simulate use of a suicide risk prediction method to identify high-risk individuals who would be offered 1 of 2 interventions to reduce suicide risk: an active contact and follow-up (ACF) intervention or a cognitive behavioral therapy (CBT) intervention. We chose these 2 options to capture the diversity of evidence-based interventions for suicide risk, with one (CBT) being intensive and costly and the other (ACF) being less intensive and less costly.4,5 For each possible intervention, we varied the test characteristics of the risk prediction across a wide range of sensitivity and specificity values. For each set of test characteristics, we then used the model to simulate the clinical and economic effects of suicide risk prediction and intervention over a lifetime horizon.

    For each simulation, we tracked the following outcomes: fatal and nonfatal suicide attempts, quality-adjusted life-years (QALYs), and health care sector and societal costs in 2016 US dollars. To determine their present value, future costs and QALYs were discounted at 3% annually.15 Using these outcomes, we calculated the incremental cost-effectiveness ratio (ICER) of suicide risk prediction and intervention as the ratio of its incremental cost to its incremental QALYs (compared with usual care).16 In the US, previous authors have advocated using ICER thresholds between $50 000 per QALY and $150 000 per QALY to define cost-effective health care interventions17,18; for our primary analysis, we designated predictive methods with an ICER of $150 000 or less as cost-effective and explored a range of alternative cost-effectiveness thresholds. Data analysis was performed from September 19, 2019, to July 5, 2020.

    On the basis of recommendations of the Second Panel on Cost-effectiveness in Health and Medicine, we report cost-effectiveness from both a health care sector perspective (including only health care–related costs) and a societal perspective (including additional costs, such as patient time and lost productivity); an impact inventory detailing each perspective is provided in eTable 1 in the Supplement.15 In describing our methods and results, we adhere to the 2013 Consolidated Health Economic Evaluation Reporting Standards (CHEERS) reporting guideline.19 As secondary research using only published data, this study was not regulated by the Massachusetts General Hospital Institutional Review Board and did not require consent from any participants.

    Model Description

    We developed a state-transition model of suicide using Microsoft Excel; its structure is shown in eFigure 1 in the Supplement. The model divides the population into 1000 strata of true suicide risk, with 1/1000th of the population in each; the probability of suicide within a stratum during each 1-year model cycle is determined based on a logit-normal distribution of suicide risk. Those who make a suicide attempt have a specified probability of dying from the attempt, which does not vary by risk stratum. Finally, patients in all strata have an annual probability of dying of other causes, which increases as the population ages.

    To simulate a risk prediction method, we directly specify its predictive accuracy rather than simulating the various clinical and demographic factors that real-world risk prediction methods incorporate.11,12 Accuracy is specified using 2 variables: a risk threshold (eg, 95th percentile) and an accuracy reduction; the derivation and implementation of these variables are described in the eAppendix in the Supplement. By adjusting these variables, a wide range of sensitivity and specificity values can be simulated. For clarity, in reporting our results, we provide only sensitivity and specificity estimates rather than the underlying model variables used to produce those values.

    Under a risk prediction and intervention strategy, any patient deemed high risk (ie, positive) is offered a risk reduction intervention; a specified fraction of patients will accept this intervention. Those who accept the intervention incur an intervention-specific cost and have their risk of suicide reduced according to a specified relative risk, which is constant across risk strata.

    Input Values

    Model input values, uncertainty intervals, and sources are given in Table 14,15,20-31; their derivation is described below.

    Suicide Risk

    We calibrated the population suicide risk distribution to reproduce the following outcomes: suicide attempt and suicide death rates of 175 per 100 000 person-years and 15 per 100 000 person-years, derived from individuals 15 years or older in national emergency department and inpatient databases,20 and fraction of suicide attempts preceded by a prior attempt of 54%, drawn from the National Epidemiologic Survey on Alcohol and Related Conditions.21 The calibrated risk distribution is given in Table 1.4,15,20-31 We specified an 8.81% probability of death per suicide attempt, derived from the aforementioned national database study.20

    Population Characteristics and Mortality

    We simulated a population with mean (SD) age of 48.8 (17.2) years based on the age distribution of primary care patients in a large US health system.22 We calculated non–suicide-related mortality rates for this population using age-specific all-cause mortality rates from the 2017 US life tables23 and age-specific suicide mortality rates from the WONDER (Wide-ranging Online Data for Epidemiologic Research) database of the Centers for Disease Control and Prevention (CDC).32 We estimated mean health-related quality of life for a US primary care population from a national analysis using the EuroQol-5 Dimension (EQ-5D) questionnaire, a validated rating scale that assesses health status across 5 dimensions; the mean EQ-5D utility was 0.866.24

    Suicide-Related and Background Costs

    We used the Bureau of Economic Analysis’s medical care expenditure indexes to inflate costs from earlier years to 2016 US dollars, the most recent year for which these indexes are available.33,34 We deflated costs from later years to 2016 US dollars using the US Federal Reserve’s personal consumption expenditure indexes.35

    We derived health care and lost productivity costs associated with suicide attempts using data for individuals 20 years or older in the CDC’s WISQARS (Web-based Injury Statistics Query and Reporting System) database.25 For nonfatal suicide attempts resulting in emergency department presentation and/or hospitalization, costs did not vary markedly with age; the mean health care cost was $10 830, and the mean lost productivity cost was $17 369. Because the costs of fatal suicide attempts varied more than 30-fold with age, we allowed these costs to change as our simulated population aged; mean health care costs started at $4528 and decreased to $4354, and mean lost productivity costs started at and $1 173 159 and decreased to $61 150 by the end of the lifetime horizon. Annual background health care costs, derived from the 2017 Medical Expenditure Panel Survey, ranged from $4016 to $11 740, depending on age.26

    Risk Assessment and Intervention

    We assumed that any patient with high predicted suicide risk would require in-person evaluation that consisted of an added or extended primary care appointment. We estimated the cost of this evaluation based on Centers for Medicare & Medicaid Services (CMS) reimbursement for a 15-minute primary care appointment (Current Procedural Terminology [CPT] code 99213) and a brief emotional assessment (CPT code 96127), yielding a health care cost of $76.27 We estimated the lost productivity cost of this assessment using the February 2020 mean hourly earnings ($29)28 and employment-to-population ratio (61.1%)29; assuming the assessment would require 1 hour of patient time, lost productivity cost was $16.

    To characterize the 2 suicide risk reduction interventions (ACF and CBT), we used efficacy estimates drawn from meta-analyses. For other variables (health care cost, societal cost, and uptake), no meta-analytic estimates were available; we instead derived estimates from single exemplar studies.5,31 We assumed the reduction in suicide risk with any intervention would last for 12 months, which may underestimate longer-term benefits5,31 but ensures a conservative assessment of the overall value of risk prediction and intervention.

    For ACF interventions, a 2015 meta-analysis5 of 9 randomized clinical trials found a relative risk of suicide attempts of 0.83 (95% CI, 0.71-0.97); of the included trials, 6 involved postcard- or telephone-based follow-up, and 3 involved home visits by community nurses. Other variables were derived from a randomized clinical trial of safety planning and telephone follow-up among military personnel with suicidal ideation or attempts.30 Participants completed a structured suicide safety plan on their initial encounter and a mean of 3.7 follow-up calls during 6 months; initial uptake of the intervention was 99.4%. We assumed that initial safety planning would be incorporated into the baseline assessment described above. For follow-up, we used a cost per telephone call of $27 based on CMS reimbursement for an 11- to 20-minute nonphysician telephone call (CPT code 98967).27 These factors resulted in a total health care cost of $96; for comparison, a 2014 analysis36 of follow-up calls after suicide-related hospitalizations estimated a cost of $11 to $140 for 2 to 5 follow-up calls. Finally, we assumed that each call would average 20 minutes of patient time, resulting in a lost productivity cost of $20.28,29

    For CBT, a 2017 meta-analysis4 of 10 randomized clinical trials found a relative risk of suicide attempts of 0.47 (95% CI, 0.30-0.73). Other variables were derived from a randomized clinical trial of brief CBT among military personnel with suicidal ideation or attempts.31 Of 169 participants who were not excluded, 17 dropped out or refused randomization, yielding an uptake of 89.9%. Participants completed a mean of 11.75 CBT sessions as part of the intervention, one of which was 90 minutes and the remainder 60 minutes; based on CMS reimbursement for individual therapy (CPT 90834 and 90837),27 this equated to a health care cost of $1088. Finally, we assumed each session would require a mean of 2 hours of patient time, resulting in a lost productivity cost of $385.28,29

    Sensitivity Analyses

    To assess overall uncertainty in our results and establish CIs around estimated accuracy requirements, we performed probabilistic sensitivity analyses. In this process, 1000 individual model runs were performed using parameter values drawn at random from distributions reflecting their uncertainty. Aggregating model results across this range of input values provides an estimate of the uncertainty in results attributable to the joint uncertainty in input values.

    To assess the robustness of our results to alternative assumptions, we also performed several univariate sensitivity analyses. In this process, we evaluated our model’s outcomes under the following input parameter assumptions. First, mean suicide risk in the simulated population decreased proportionally by 50% or increased by 100%. Second, suicide risk reduction after intervention increased or decreased proportionally by 50%; notably, this range includes the estimated efficacy of CBT obtained via meta-analysis with 1 outlier excluded (0.61).4 Third, intervention uptake varied between 50% and 100%; for comparison, among the studies included in our meta-analytic efficacy estimates, the lowest uptake was 51.1%.37 Fourth, intervention cost increased or decreased proportionally by 50%.

    Results
    Base Case

    Example outcomes based on a risk prediction method with 95% specificity and 25% sensitivity are given in Table 2. With usual care, the rate of suicide death was 15.34 per 100 000 person-years and the rate of suicide attempt was 174.15 per 100 000 person-years. If used to target ACF, this risk prediction method reduced the population’s suicide death rate by 0.52 and suicide attempt rate by 5.90 per 100 000 person-years; if used to target CBT, risk prediction reduced the population's suicide death rate by 1.56 and suicide attempt rate by 17.76 per 100 000 person-years. Mean QALYs increased by 0.0016 with ACF and 0.0046 with CBT. With both interventions, risk prediction decreased suicide-related costs but increased other health care costs. For risk prediction followed by ACF, ICERs were $107 000 per QALY from a health care sector perspective and $57 000 per QALY from a societal perspective; for risk prediction followed by CBT, ICERs were $221 000 per QALY from a health care sector perspective and $224 000 per QALY from a societal perspective.

    Required Accuracy Values

    Figure 1 shows the threshold accuracy values required for suicide risk prediction and intervention to become cost-effective (ie, achieve an ICER ≤$150 000 per QALY) from a health care sector perspective; accuracy estimates from several real-world risk prediction methods are included for comparison.11-13 At a specificity of 95%, risk prediction followed by ACF needed to attain a sensitivity of 17% (95% CI, 7%-37%) to become cost-effective; risk prediction followed by CBT needed to attain a sensitivity of 36% (95% CI, 23%-60%). To achieve lower ICER values, accuracy requirements were more stringent (eFigure 2 in the Supplement); for example, to achieve an ICER of $50 000 per QALY or less, risk prediction followed by ACF required a specificity of 95% or higher and a sensitivity of 57% or higher.

    Threshold values for PPV (predicting suicide attempt) are shown in Figure 2. Notably, the base case PPV threshold was stable across specificity values from 90% to 99%, ranging from 0.8% to 0.9% for ACF and 1.6% to 1.8% for CBT, under a health care sector perspective.

    Results from a societal perspective are shown in eFigures 3 through 5 in the Supplement; the required accuracy to achieve cost-effectiveness was consistently lower than under a health care sector perspective.

    Sensitivity Analyses

    Table 3 gives the threshold values of sensitivity, PPV predicting suicide attempt, and PPV predicting suicide death under a range of alternative input value assumptions, from a health care sector perspective. Several notable trends are apparent. Threshold PPV values differed substantially, depending on whether suicide attempt or suicide death was being predicted. For ACF, threshold PPV predicting suicide attempt ranged from 0.5% to 1.6% (base case, 0.8%), whereas threshold PPV predicting suicide death ranged from 0.05% to 0.14% (base case 0.07%). For CBT, threshold PPV predicting suicide attempt ranged from 0.9% to 3.4% (base case 1.7%), whereas threshold PPV predicting suicide death ranged from 0.1% to 0.3% (base case, 0.2%). In contrast, threshold PPV was quite stable to changes in population suicide risk; for example, for CBT, threshold PPV predicting suicide death varied from 0.14% to 0.15% when population risk was halved or doubled. Finally, under all cases, the accuracy required to achieve cost-effectiveness was lower under a societal perspective (eTable 2 in the Supplement).

    Discussion

    This economic evaluation estimated threshold values of sensitivity, specificity, and PPV that suicide risk prediction methods must attain to enable cost-effective targeting of suicide risk reduction interventions. For a less intensive intervention, such as safety planning with follow-up telephone calls, risk prediction could be cost-effective, with a specificity of 95% and a sensitivity of 20%, equating to a PPV of approximately 1%. For a more intensive intervention, such as CBT, risk prediction would likely require a specificity of 95% and sensitivity greater than 35%, equating to a PPV of approximately 2%.

    The identified PPV thresholds (1%-2%) may intuitively seem quite low. However, that these values are so low reflects the immense societal and individual consequences of suicide.14,25 Notably, these PPV estimates are in line with risk thresholds used for primary prevention of other similarly impactful conditions. For comparison, US guidelines recommend offering anticoagulants to patients with atrial fibrillation whose annual risk of stroke exceeds 0.6% to 2%38,39; offering bisphosphonates to patients with low bone density whose annual risk of hip fracture or major osteoporotic fracture exceeds 0.3% or 2%, respectively40,41; and offering endocrine therapy to women whose annual risk of breast cancer exceeds 0.3%.42,43

    When judged against these benchmarks, many existing suicide risk prediction models perform favorably. For comparison, recent risk prediction models have reported the following sensitivities at 95% specificity: 26%, from a model developed in the Veterans Health Administration13; 33%, from a model developed in Massachusetts academic medical centers11; and 48%, from a model developed in a multistate collaboration.12 The corresponding PPVs were 0.1% to 0.3% for predicting suicide death12,13 and 2% to 6%11,12 for predicting suicide attempts; of note, even higher PPVs (>10%)12 could be attained by increasing specificity above 99%, although with a consequent decrease in the proportion of cases identified. Overall, this study’s benchmarks suggest that any of these models could be adequate for targeting an ACF intervention, and some may be accurate enough to use for targeting CBT.

    However, the results of this study do not definitively demonstrate that any specific risk prediction model is cost-effective, in light of a significant limitation: although this study evaluates risk prediction in an outpatient population, most data on the efficacy of risk reduction interventions are based on individuals seeking help at an emergency department, often after a suicide attempt.4,5 Currently, no evidence is available on the efficacy of interventions in a proactively identified outpatient population, although a randomized clinical trial in this population is underway.44 Until such data are available, the current analysis must rely on extrapolation. For this reason, using the current findings to guide large-scale implementation of any specific suicide risk prediction model would be premature, but short of this, the findings provide strong support for initiating real-world pilot studies of existing risk prediction models.

    Limitations

    This analysis has several additional limitations in addition to the aforementioned shortcomings in the population that input data are drawn from. As a modeling analysis, the work elides some real-world complexities (eg, heterogeneity of suicide case-fatality rates by method and demographic factors20) and requires assumptions about model structure and input data. Where possible, this study made conservative assumptions about the benefits of risk prediction and intervention (eg, assuming a 12-month duration of intervention efficacy), which may result in more stringent estimates of threshold accuracy values.

    In addition, there is uncertainty around both the primary and secondary benefits and harms of risk reduction interventions. With respect to primary benefits (preventing suicide attempts and deaths), previous meta-analyses4-6,45 have estimated varying effect sizes, with not all showing statistically significant benefits; this finding is likely attributable to variation in outcome measures (suicidal ideation vs suicide attempts vs suicide deaths) and the heterogeneity in the interventions themselves. This heterogeneity in interventions also makes it impossible to generate a single cost estimate that can be generalized to all risk reduction interventions; this issue was addressed by instead performing a thorough cost accounting from 2 specific exemplar interventions.30,31 More broadly, the uncertainty in both cost and efficacy have been addressed by assessing a broad range of possible model inputs in sensitivity analyses.

    With respect to secondary effects, there are no data from primary care populations to inform whether risk reduction interventions would produce benefits (eg, improvements in depression) or harms (eg, increased stigma). Previous studies31,46-48 in emergency department patients have identified primarily benefits, including reductions in health care costs, medical retirement, and depressive symptoms. Because these secondary benefits were observed in selected patient populations (eg, after a suicide attempt or psychiatric hospitalization), the most conservative approach was to not assume equivalent benefits in primary care patients; however, this choice could underestimate or overestimate the value of risk reduction interventions to some extent.

    Finally, the analysis did not account for several aspects of real-world implementation of risk reduction interventions, such as staff training, the up-front costs of integrating a risk prediction model into clinical workflows, and the effects of merging statistical prediction with practitioner assessment on the accuracy of risk assessment.49-51 These factors were not relevant to this analysis given the primary objective of establishing accuracy benchmarks, but they will be important to consider when evaluating specific risk reduction programs.

    Conclusions

    This economic evaluation estimated threshold accuracy values that suicide risk prediction methods must attain to be cost-effective in the US. The study found that many existing risk prediction models exceed these accuracy requirements. Although the results do not conclusively demonstrate that any specific suicide risk prediction model is cost-effective, they suggest that current risk prediction models have achieved sufficient accuracy for health systems to move forward with pilot implementation projects.

    Back to top
    Article Information

    Accepted for Publication: January 17, 2021.

    Published Online: March 17, 2021. doi:10.1001/jamapsychiatry.2021.0089

    Correction: This article was corrected on April 7, 2021, to change the word “variant” to “variable” in 3 places in the Methods section under Risk Assessment and Intervention.

    Corresponding Author: Eric L. Ross, MD, Department of Psychiatry, Massachusetts General Hospital, Wang Ambulatory Care Center 812, 15 Parkman St, Boston, MA 02114 (eross9@mgh.harvard.edu).

    Author Contributions: Dr Ross had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Ross, Zuromski, Reis, Smoller.

    Acquisition, analysis, or interpretation of data: Ross, Zuromski, Nock, Kessler, Smoller.

    Drafting of the manuscript: Ross, Reis.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Ross.

    Administrative, technical, or material support: Zuromski.

    Supervision: Reis, Smoller.

    Conflict of Interest Disclosures: Dr Zuromski reported receiving grants from the National Institute of Mental Health during the conduct of the study. Dr Kessler reported receiving personal fees from DataStat Inc and consultant fees from Sage Pharmaceuticals and Takeda during the conduct of the study. Dr Smoller reported serving as an unpaid member of the Bipolar/Depression Research Community Advisory Panel of 23andMe and a member of the Leon Levy Foundation Neuroscience Advisory Board, being a Tepper Family Massachusetts General Hospital Research Scholar, and receiving an honorarium for an internal seminar at Biogen Inc. No other disclosures were reported.

    Funding/Support: This work was supported by grants R25 MH094612 (Dr Ross) and R01MH117599 (Dr Smoller) from the National Institute of Mental Health, a gift from the Tommy Fuss Fund (Dr Smoller), and a grant from the Harvard Medical School (Dr Smoller).

    Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    References
    1.
    Xu  J, Murphy  SL, Kockanek  KD, Arias  E.  Mortality in the United States, 2018.   NCHS Data Brief. 2020;(355):1-8.PubMedGoogle Scholar
    2.
    Wang  J, Sumner  SA, Simon  TR,  et al.  Trends in the incidence and lethality of suicidal acts in the United States, 2006 to 2015.   JAMA Psychiatry. 2020;77(7):684-693. doi:10.1001/jamapsychiatry.2020.0596 PubMedGoogle ScholarCrossref
    3.
    D’Anci  KE, Uhl  S, Giradi  G, Martin  C.  Treatments for the prevention and management of suicide.   Ann Intern Med. 2019;171(5):334-342. doi:10.7326/M19-0869 PubMedGoogle ScholarCrossref
    4.
    Gøtzsche  PC, Gøtzsche  PK.  Cognitive behavioural therapy halves the risk of repeated suicide attempts: systematic review.   J R Soc Med. 2017;110(10):404-410. doi:10.1177/0141076817731904 PubMedGoogle ScholarCrossref
    5.
    Inagaki  M, Kawashima  Y, Kawanishi  C,  et al.  Interventions to prevent repeat suicidal behavior in patients admitted to an emergency department for a suicide attempt: a meta-analysis.   J Affect Disord. 2015;175:66-78. doi:10.1016/j.jad.2014.12.048 PubMedGoogle ScholarCrossref
    6.
    Riblet  NBV, Shiner  B, Young-Xu  Y, Watts  BV.  Strategies to prevent death by suicide: meta-analysis of randomised controlled trials.   Br J Psychiatry. 2017;210(6):396-402. doi:10.1192/bjp.bp.116.187799 PubMedGoogle ScholarCrossref
    7.
    Stone  DM, Simon  TR, Fowler  KA,  et al.  Trends in state suicide rates 1999-2016.   Morbidity and Mortality Weekly Report. 2018;67(22):617-624. doi:10.15585/mmwr.mm6722a1 PubMedGoogle ScholarCrossref
    8.
    Ahmedani  BK, Westphal  J, Autio  K,  et al.  Variation in patterns of health care before suicide: a population case-control study.   Prev Med. 2019;127:105796. doi:10.1016/j.ypmed.2019.105796 PubMedGoogle Scholar
    9.
    Luoma  JB, Martin  CE, Pearson  JL.  Contact with mental health and primary care providers before suicide: a review of the evidence.   Am J Psychiatry. 2002;159(6):909-916. doi:10.1176/appi.ajp.159.6.909 PubMedGoogle ScholarCrossref
    10.
    Belsher  BE, Smolenski  DJ, Pruitt  LD,  et al.  Prediction models for suicide attempts and deaths: a systematic review and simulation.   JAMA Psychiatry. 2019;76(6):642-651. doi:10.1001/jamapsychiatry.2019.0174 PubMedGoogle ScholarCrossref
    11.
    Barak-Corren  Y, Castro  VM, Javitt  S,  et al.  Predicting suicidal behavior from longitudinal electronic health records.   Am J Psychiatry. 2017;174(2):154-162. doi:10.1176/appi.ajp.2016.16010077 PubMedGoogle ScholarCrossref
    12.
    Simon  GE, Johnson  E, Lawrence  JM,  et al.  Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records.   Am J Psychiatry. 2018;175(10):951-960. doi:10.1176/appi.ajp.2018.17101167 PubMedGoogle ScholarCrossref
    13.
    Kessler  RC, Hwang  I, Hoffmire  CA,  et al.  Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans Health Administration.   Int J Methods Psychiatr Res. 2017;26(3):1-7. doi:10.1002/mpr.1575 PubMedGoogle ScholarCrossref
    14.
    Kessler  RC.  Clinical epidemiological research on suicide-related behaviors—where we are and where we need to go.   JAMA Psychiatry. 2019;76(8):777-778. doi:10.1001/jamapsychiatry.2019.1238 PubMedGoogle ScholarCrossref
    15.
    Sanders  GD, Neumann  PJ, Basu  A,  et al.  Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.   JAMA. 2016;316(10):1093-1103. doi:10.1001/jama.2016.12195 PubMedGoogle ScholarCrossref
    16.
    Briggs  AH, Claxton  C, Sculpher  MJ.  Decision Modelling for Health Economic Evaluation. Oxford University Press; 2006.
    17.
    Neumann  PJ, Cohen  JT, Weinstein  MC.  Updating cost-effectiveness: the curious resilience of the $50,000-per-QALY threshold.   N Engl J Med. 2014;371(9):796-797. doi:10.1056/NEJMp1405158 PubMedGoogle ScholarCrossref
    18.
    Anderson  JL, Heidenreich  PA, Barnett  PG,  et al.  ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines.   J Am Coll Cardiol. 2014;63(21):2304-2322. doi:10.1016/j.jacc.2014.03.016 PubMedGoogle ScholarCrossref
    19.
    Husereau  D, Drummond  M, Petrou  S,  et al; ISPOR Health Economic Evaluation Publication Guidelines-CHEERS Good Reporting Practices Task Force.  Consolidated Health Economic Evaluation Reporting Standards (CHEERS): explanation and elaboration: a report of the ISPOR Health Economic Evaluations Publication Good Reporting Practices Task Force.   Value Health. 2013;16(2):231-250. doi:10.1016/j.jval.2013.02.002 PubMedGoogle ScholarCrossref
    20.
    Conner  A, Azrael  D, Miller  M.  Suicide case-fatality rates in the United States, 2007 to 2014: a nationwide population-based study.   Ann Intern Med. 2019;171(12):885-895. doi:10.7326/M19-1324 PubMedGoogle ScholarCrossref
    21.
    Olfson  M, Blanco  C, Wall  M,  et al.  National trends in suicide attempts among adults in the United States.   JAMA Psychiatry. 2017;74(11):1095-1103. doi:10.1001/jamapsychiatry.2017.2582 PubMedGoogle ScholarCrossref
    22.
    Yarborough  BJH, Perrin  NA, Stumbo  SP, Muench  J, Green  CA.  Preventive service use among people with and without serious mental illnesses.   Am J Prev Med. 2018;54(1):1-9. doi:10.1016/j.amepre.2017.08.020 PubMedGoogle ScholarCrossref
    23.
    Arias  E, Xu  J.  United States life tables, 2017.   Natl Vital Stat Rep. 2019;68(7):1-66.PubMedGoogle Scholar
    24.
    Sullivan  PW, Ghushchyan  V.  Preference-Based EQ-5D index scores for chronic conditions in the United States.   Med Decis Making. 2006;26(4):410-420. doi:10.1177/0272989X06290495 PubMedGoogle ScholarCrossref
    25.
    Centers for Disease Control and Prevention. WISQARS cost of injury reports. Accessed March 19, 2020. https://wisqars.cdc.gov:8443/costT/
    26.
    Agency for Healthcare Research and Quality. Mean expenditure per person with expense (standard errors) by age groups, United States, 2017, Medical Expenditure Panel Survey. Accessed March 19, 2020. https://meps.ahrq.gov/mepstrends/hc_use/
    27.
    Centers for Medicare and Medicaid Services. Physician Fee Schedule. 2020. Updated February 3, 2021. Accessed March 19, 2019. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/PhysicianFeeSched
    28.
    Bureau of Labor Statistics. Current and real (constant 1982-1984 dollars) earnings for all employees on private nonfarm payrolls, seasonally adjusted, February 2020. Accessed March 19, 2019. https://www.bls.gov/news.release/realer.t01.htm
    29.
    Bureau of Labor Statistics. Employment status of the civilian population by sex and age, February 2020. Published 2020. Accessed March 19, 2020. https://www.bls.gov/news.release/empsit.t01.htm
    30.
    Stanley  B, Brown  GK, Brenner  LA,  et al.  Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department.   JAMA Psychiatry. 2018;75(9):894-900. doi:10.1001/jamapsychiatry.2018.1776 PubMedGoogle ScholarCrossref
    31.
    Rudd  MD, Bryan  CJ, Wertenberger  EG,  et al.  Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up.   Am J Psychiatry. 2015;172(5):441-449. doi:10.1176/appi.ajp.2014.14070843 PubMedGoogle ScholarCrossref
    32.
    Centers for Disease Control and Prevention. Underlying cause of death 1999-2018 on CDC WONDER online database, release in 2020. National Center for Health Statistics. Published 2020. Accessed March 16, 2020. https://wonder.cdc.gov/ucd-icd10.html
    33.
    Bureau of Economic Analysis. Medical Care Expenditure Indices from MEPS. 2019. Accessed September 5, 2019. https://www.bea.gov/national/health_care_satellite_account.htm
    34.
    Dunn  A, Grosse  SD, Zuvekas  SH.  Adjusting health expenditures for inflation: a review of measures for health services research in the United States.   Health Serv Res. 2018;53(1):175-196. doi:10.1111/1475-6773.12612 PubMedGoogle ScholarCrossref
    35.
    Federal Reserve Bank of St. Louis. Personal consumption expenditures. 2020. Accessed February 13, 2020. https://fred.stlouisfed.org/series/DPCERG3A086NBEA
    36.
    Richardson  JS, Mark  TL, McKeon  R.  The return on investment of postdischarge follow-up calls for suicidal ideation or deliberate self-harm.   Psychiatr Serv. 2014;65(8):1012-1019. doi:10.1176/appi.ps.201300196 PubMedGoogle ScholarCrossref
    37.
    Guthrie  E, Kapur  N, Mackway-Jones  K,  et al.  Randomised controlled trial of brief psychological intervention after deliberate self poisoning.   BMJ. 2001;323(7305):135-138. doi:10.1136/bmj.323.7305.135 PubMedGoogle ScholarCrossref
    38.
    January  CT, Wann  LS, Calkins  H,  et al.  2019 AHA/ACC/HRS focused update of the 2014 AHA/ACC/HRS guideline for the management of patients with atrial fibrillation: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the Heart R.   J Am Coll Cardiol. 2019;74(1):104-132. doi:10.1016/j.jacc.2019.01.011 PubMedGoogle ScholarCrossref
    39.
    Friberg  L, Rosenqvist  M, Lip  GYH.  Net clinical benefit of warfarin in patients with atrial fibrillation: a report from the Swedish atrial fibrillation cohort study.   Circulation. 2012;125(19):2298-2307. doi:10.1161/CIRCULATIONAHA.111.055079 PubMedGoogle ScholarCrossref
    40.
    Cosman  F, de Beur  SJ, LeBoff  MS,  et al; National Osteoporosis Foundation.  Clinician’s guide to prevention and treatment of osteoporosis.   Osteoporos Int. 2014;25(10):2359-2381. doi:10.1007/s00198-014-2794-2 PubMedGoogle ScholarCrossref
    41.
    Tosteson  ANA, Melton  LJ  III, Dawson-Hughes  B,  et al; National Osteoporosis Foundation Guide Committee.  Cost-effective osteoporosis treatment thresholds: the United States perspective.   Osteoporos Int. 2008;19(4):437-447. doi:10.1007/s00198-007-0550-6 PubMedGoogle ScholarCrossref
    42.
    Owens  DK, Davidson  KW, Krist  AH,  et al; US Preventive Services Task Force.  Medication use to reduce risk of breast cancer: US Preventive Services Task Force recommendation statement.   JAMA. 2019;322(9):857-867. doi:10.1001/jama.2019.11885 PubMedGoogle ScholarCrossref
    43.
    Visvanathan  K, Fabian  CJ, Bantug  E,  et al.  Use of endocrine therapy for breast cancer risk reduction: ASCO clinical practice guideline update.   J Clin Oncol. 2019;37(33):3152-3165. doi:10.1200/JCO.19.01472 PubMedGoogle ScholarCrossref
    44.
    Simon  GE, Beck  A, Rossom  R,  et al.  Population-based outreach versus care as usual to prevent suicide attempt: study protocol for a randomized controlled trial.   Trials. 2016;17(1):452. doi:10.1186/s13063-016-1566-z PubMedGoogle ScholarCrossref
    45.
    Milner  AJ, Carter  G, Pirkis  J, Robinson  J, Spittal  MJ.  Letters, green cards, telephone calls and postcards: systematic and meta-analytic review of brief contact interventions for reducing self-harm, suicide attempts and suicide.   Br J Psychiatry. 2015;206(3):184-190. doi:10.1192/bjp.bp.114.147819 PubMedGoogle ScholarCrossref
    46.
    Bernecker  SL, Zuromski  KL, Curry  JC,  et al.  Economic evaluation of brief cognitive behavioral therapy vs treatment as usual for suicidal US Army soldiers.   JAMA Psychiatry. 2020;77(3):256-264. doi:10.1001/jamapsychiatry.2019.3639 PubMedGoogle ScholarCrossref
    47.
    O’Connor  RC, Ferguson  E, Scott  F,  et al.  A brief psychological intervention to reduce repetition of self-harm in patients admitted to hospital following a suicide attempt: a randomised controlled trial.   Lancet Psychiatry. 2017;4(6):451-460. doi:10.1016/S2215-0366(17)30129-3 PubMedGoogle ScholarCrossref
    48.
    Carter  GL, Clover  K, Whyte  IM, Dawson  AH, D’Este  C.  Postcards from the EDge: 5-year outcomes of a randomised controlled trial for hospital-treated self-poisoning.   Br J Psychiatry. 2013;202(5):372-380. doi:10.1192/bjp.bp.112.112664 PubMedGoogle ScholarCrossref
    49.
    Whiting  D, Fazel  S.  How accurate are suicide risk prediction models? asking the right questions for clinical practice.   Evid Based Ment Health. 2019;22(3):125-128. doi:10.1136/ebmental-2019-300102 PubMedGoogle ScholarCrossref
    50.
    Brown  S, Iqbal  Z, Burbidge  F,  et al.  Embedding an evidence-based model for suicide prevention in the national health service: a service improvement initiative.   Int J Environ Res Public Health. 2020;17(14):1-13. doi:10.3390/ijerph17144920 PubMedGoogle ScholarCrossref
    51.
    Boudreaux  ED, Haskins  BL, Larkin  C,  et al.  Emergency department safety assessment and follow-up evaluation 2: an implementation trial to improve suicide prevention.   Contemp Clin Trials. 2020;95:106075. doi:10.1016/j.cct.2020.106075 PubMedGoogle Scholar
    ×