Evaluation of the Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment Suicide Risk Modeling Clinical Program in the Veterans Health Administration

Key Points

Question  Is the Veterans Health Administration Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (REACH VET) program, which facilitates care enhancements for individuals in the top 0.1% suicide risk tier using a validated algorithm, associated with health care utilization, treatment engagement, suicide attempts, suicide safety plan documentation, and suicide mortality?

Findings  In this cohort study including 173 313 individuals before and after implementation of the REACH VET program using triple differences, inclusion in the REACH VET program was associated with having more outpatient encounters, increased documentation of new suicide prevention safety plans, and fewer inpatient mental health admissions, emergency department visits, and documented suicide attempts.

Meaning  These findings suggest that clinical programs using predictive modeling can support care enhancements and risk reduction.

Importance  The Veterans Health Administration (VHA) implemented a national clinical program using a suicide risk prediction algorithm, Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (REACH VET), in which clinicians facilitate care enhancements for individuals identified in local top 0.1% suicide risk tiers. Evaluation studies are needed.

Objective  To determine associations with treatment engagement, health care utilization, suicide attempts, safety plan documentation, and 6-month mortality.

Design, Setting, and Participants  This cohort study used triple differences analyses comparing 6-month changes in outcomes after vs before program entry for individuals entering the REACH VET program (March 2017-December 2018) vs a similarly identified top 0.1% suicide risk tier cohort from prior to program initiation (March 2014-December 2015), adjusting for trends across subthreshold cohorts. Subcohort analyses (including individuals from March 2017-June 2018) evaluated difference-in-differences for cause-specific mortality using death certificate data. The subthreshold cohorts included individuals in the top 0.3% to 0.1% suicide risk tier, below the threshold for REACH VET eligibility, from the concurrent REACH VET period and from the pre–REACH VET period. Data were analyzed from December 2019 through September 2021.

Exposures  REACH VET–designated clinicians treatment reevaluation and outreach for care enhancements, including safety planning, increased monitoring, and interventions to enhance coping.

Main Outcomes and Measures  Process outcomes included VHA scheduled, completed, and missed appointments; mental health visits; and safety plan documentation and documentation within 6 months for individuals without plans within the prior 2 years. Clinical outcomes included mental health admissions, emergency department visits, nonfatal suicide attempts, and all-cause, suicide, and nonsuicide external-cause mortality.

Results  A total of 173 313 individuals (mean [SD] age, 51.0 [14.7] years; 161 264 [93.1%] men and 12 049 [7.0%] women) were included in analyses, including 40 816 individuals eligible for REACH VET care and 36 604 individuals from the pre–REACH VET period in the top 0.1% of suicide risk. The REACH VET intervention was associated with significant increases in completed outpatient appointments (adjusted triple difference [ATD], 0.31; 95% CI, 0.06 to 0.55) and proportion of individuals with new safety plans (ATD, 0.08; 95% CI, 0.06 to 0.10) and reductions in mental health admissions (ATD, −0.08; 95% CI, −0.10 to −0.05), emergency department visits (ADT, −0.03; 95% CI, −0.06 to −0.01), and suicide attempts (ADT, −0.05; 95% CI, −0.06 to −0.03). Subcohort analyses did not identify differences in suicide or all-cause mortality (eg, age-and-sex-adjusted difference-in-difference for suicide mortality, 0.0007; 95% CI, −0.0006 to 0.0019).

Conclusions and Relevance  These findings suggest that REACH VET implementation was associated with greater treatment engagement and new safety plan documentation and fewer mental health admissions, emergency department visits, and suicide attempts. Clinical programs using risk modeling may be effective tools to support care enhancements and risk reduction.

Veteran suicide rates exceed those of other US adults.^1,2 Suicide prevention is the top clinical priority of the Department of Veterans Affairs (VA).^3-5 Informed by prior work,^6,7 VA and National Institute of Mental Health scientists developed an expansive suicide mortality risk prediction algorithm using Veterans Health Administration (VHA) electronic health records.⁸ They identified individuals at risk rather than individual risk factors. Patients in the top 0.1% tier of risk died from suicide at a rate 30-fold higher than the overall VHA patient population.⁸ Machine learning analyses⁹ generated prediction similar to that of the proof-of-concept model⁸ using fewer variables, facilitating further targeted efforts¹⁰ to complement clinical assessments.

In 2017, the VHA began national implementation of the Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (REACH VET) program, applying the algorithm⁹ to identify individuals in facilities’ top 0.1% suicide risk tiers.¹¹ Using an interactive dashboard, program coordinators receive information and communicate with clinicians who re-evaluate treatment strategies and conduct outreach to collaboratively initiate care enhancements. Reevaluation includes assessment of service needs, and clinicians ensure that patients can access services. Treatment enhancements include suicide prevention safety planning, enhanced communication, increased monitoring of stressful events, and interventions to support coping. To facilitate fidelity, actions are documented using templated notes.

Focused outreach and engagement through REACH VET may affect health care processes,¹² access,¹³ quality, health behavior,¹⁴ and suicide-related behavior. The program may enhance engagement in evidence-based treatment and treatment adherence; affect health behaviors by increasing the perceived benefits of treatment seeking and adherence, trouble-shooting barriers, and providing cues to action; and address access barriers and enhance quality via improved resources and processes (eg, treatment review, follow-up). These may result in better outcomes, including reduced suicide attempts and deaths.

Little is known regarding program effectiveness. It is important to evaluate outcomes associated with the REACH VET program.¹⁵ We assess associations with treatment engagement, care processes, health care utilization, and mortality outcomes. We hypothesized that inclusion in REACH VET is associated with enhanced engagement (ie, more scheduled and completed appointments, fewer missed appointments, and more outpatient mental health encounters) and quality of care (ie, suicide safety plans), reduced acute care needs (ie, fewer inpatient mental health admissions and emergency department [ED] visits), and reduced suicide-related behavior (ie, suicide attempts and suicide deaths) and nonsuicide external cause and all-cause mortality.

This study was conducted as part of ongoing VHA operations and program evaluation and approved by the Office of Mental Health and Suicide Prevention; it was not classified as research and was exempt from VHA institutional review board review and informed consent. This study is reported following the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

The predictive model includes sociodemographic, diagnosis, health care utilization, medication, and prior suicide attempt indicators. Coded into VHA’s Business Intelligence platform, the risk prediction algorithm draws from the VHA Corporate Data Warehouse and is processed monthly for all individuals with VHA inpatient or outpatient care in the prior 2 years, identifying those entering top 0.1% suicide risk tiers at each of 141 VHA administrative parent facilities (service points with shared administrative responsibility).

Measures were assessed from Corporate Data Warehouse except where specified. These examined treatment engagement (ie, non–same-day outpatient appointments, non–same-day completed outpatient appointments, and percentage of scheduled appointments resulting in missed encounters); processes (ie, documentation of suicide prevention safety plan, documentation within 6 months among individuals without plans in the prior 2 years); health care utilization (ie, outpatient mental health visits, inpatient mental health admissions, and ED visit days); documentation of nonfatal suicide attempts (per site reports in the Suicide Prevention Applications Network and the Suicide Behavior and Overdose Report¹⁶); 6-month mortality per the Vital Status File and all-cause, suicide, and nonsuicide external cause mortality per the VA/Department of Defense Mortality Data Repository of National Death Index search results through 2018; and implementation indicators from operations systems. To enhance sensitivity, health care utilization measures assessed counts rather than some vs none.

Preliminary to statistical analyses, for outcomes measured in the prior and subsequent 6 months, we tested the parallel trend assumption for difference-in-differences analyses (eAppendix in the Supplement). Outcomes were collected across four 6-month periods prior to cohort entry. Parallel trends between the REACH VET era’s top 0.1% risk tier and subthreshold cohorts were assessed using an interaction between time period and group. For top 0.1% risk tier cohorts, parallel trends were assessed using interaction between time period and era. The parallel trend assumption failed for outcomes other than completed appointments, scheduled appointments, and proportion of appointments missed, validating use of the triple differences (TD) design. We similarly evaluated residual trends after adoption of the TD approach. Significant preidentification trends were observed for all except for appointment-related outcomes. Trends were close to zero and must be interpreted in the context of main analysis results (eAppendix in the Supplement).

With an intention-to-treat framework, we examined correlates of REACH VET entry. Pre-post comparisons were necessary because health care utilization measures were elements of the algorithm, and individuals entering the top risk tier could return to lower-risk states as regression to the mean. We evaluated differential change, comparing differences in changes for the REACH VET cohort vs a comparable cohort identified prior to program initiation. We used a TD design to isolate potential program influences from period and cohort trends (Figure).¹⁷ We compared changes 6 months after vs 6 months before cohort entry for REACH VET and pre–REACH VET top 0.1% suicide risk cohorts, adjusting for differences in pre-post differences for subthreshold risk cohorts (top 0.3%-0.1% tiers) in the REACH VET and pre–REACH VET eras. Inclusion of subthreshold cohorts enabled adjustment for period differences. Tiers were evaluated for patients’ local VHA facilities.

The TD design examined health care utilization, engagement, suicide safety plan documentation, and nonfatal suicide attempts measures assessed in the prior and subsequent 6 months. For outcomes assessed only after cohort entry (ie, new safety plan documentation, mortality), difference-in-difference analyses compared 6-month outcomes for the top 0.1% risk tier and subthreshold cohorts across the REACH VET implementation and preimplementation periods.

Program implementation involved training and development. After several months of piloting, in February 2017, the VHA began monthly dissemination of information regarding individuals in top 0.1% risk tiers, with enhanced program dashboards.¹⁸ We assessed measures for individuals who entered a REACH VET top 0.1% risk tier cohort in March 2017 to December 2018, excluding those identified during initial development (November 2016-February 2017).

Pre–REACH VET comparison cohorts were generated by applying the algorithm monthly for March 2014 to December 2015. Paralleling the implementation period, these cohorts excluded individuals in a top 0.1% risk tier within 4 months prior to March 2014.

For both periods, inclusion required VHA encounters in the prior 2 years. Six-month postidentification periods (180 days) began when individuals were identified as being in a top 0.1% risk tier. Subthreshold cohorts were generated for top 0.3% to 0.1% risk tiers, excluding individuals who were identified in a top 0.1% risk tier prior to or within 6 months after their subthreshold cohort identification date. Cohorts excluded 43 records owing to missing age or sex data and 456 records owing to death prior to their cohort entry date.

For measures assessed in both the previous and subsequent 6 months, TDs were calculated using generalized estimating equations at the individual level, with repeated measures and a robust sandwich estimator covariance structure. The TD approach was needed to account for period and cohort trends that violated the difference-in-differences parallel trend assumption. The key variable was a 3-way interaction among subsequent 6-month period (vs prior 6 months), REACH VET era cohort (vs pre–REACH VET cohort), and top 0.1% risk tier (vs top 0.3%-0.1% risk tier). The 3-way interaction was calculated as: Outcome = (β₀ + β₁ × Post6Months + β₂ × REACH_VET_Era + β₃ × Top0.1% + β₄ × Post6Months × REACH_VET_Era + β₅ × Post6Months × Top0.1% + β₆ × REACH_VET_Era × Top0.1% + β₇ × Post6Months × REACH_VET_Era × Top0.1% + ε_i). This interaction measures differential changes in the REACH VET era between individuals identified for the program (top 0.1% risk tier) vs those who were not, isolating the program’s associations while controlling for period and cohort confounders. Models included interactions between the grouping variables (ie, prior vs subsequent 6 months, REACH VET vs pre–REACH VET periods, and top 0.1% vs top 0.3%-0.1% risk tiers). Adjusted models controlled for age and sex on the individual’s cohort entry date.

Count outcome models used the Poisson distribution; binary outcome models used the binomial distribution. When binomial models failed to converge, a modified Poisson was used.¹⁹ For percentage of missed appointments, cohorts were specific to individuals with appointments in both prior and subsequent periods, and models used the normal distribution and identity-link function. In difference-in-differences models, the key variable was the interaction between period and risk tier.

When overdispersion or zero inflation was suspected for count outcomes, negative binomial and zero inflation models were calculated to control for these issues, respectively; these did not change conclusions. We calculated absolute differences using the SAS NLEstimate macro (SAS Institute). Significance was assessed with 2-tailed χ² tests, with α = .05. Analyses used SAS Enterprise Guide, versions 7.1 and 8.2.

For exploratory analyses using cause of death data through 2018, we identified a REACH VET subcohort who qualified between March 2017 to June 2018 and a concurrent subthreshold subcohort. We assessed 6-month all-cause, suicide, and nonsuicide external cause mortality. Pre–REACH VET top risk tier and subthreshold cohorts were included in these analyses. Data were analyzed from December 2019 through September 2021.

A total of 173 313 individuals (mean [SD] age, 51.0 [14.7] years; 161 264 [93.1%] men and 12 049 [7.0%] women) were included in analyses, including 40 816 individuals in the top 0.1% risk tier from the REACH VET period, and 36 604 individuals in the top 0.1% risk tier from the pre–REACH VET period. Individuals in top 0.1% risk tiers were younger than sub-threshold cohorts in both periods (Table 1). Individuals in the REACH VET era were older than those in the earlier period. Top 0.1% risk tiers had higher proportions of men than subthreshold cohorts. Pre–REACH VET cohorts had higher proportions of men than REACH VET–era cohorts (Table 1).

REACH VET was implemented at all VHA facilities. From March 2017 to December 2018, program performance increased: coordinator acknowledgment that individuals were identified for REACH VET increased from 5484 of 6334 individuals (86.6%) to 6485 of 6624 individuals (97.9%); clinician acknowledgment increased from 3200 of 6334 individuals (50.5%) to 6008 of 6624 individuals (90.7%); and care evaluations increased from 2923 of 6334 individuals (46.1%) to 5796 of 6624 individuals (87.5%). Outreach attempt prevalence increased from 2736 of 6334 individuals (43.2%) to 5665 of 6624 individuals (85.5%), but successful outreach attempts decreased from 2024 of 2736 individuals for whom outreach was attempted (74.0%) to 4020 of 5665 individuals for whom outreach was attempted (71.0%).

eTable 1 and eTable 2 in the Supplement provide detail regarding 6-month outcomes for the pre–REACH VET period compared with during the REACH VET program. Table 2 presents TD results. Accounting for potential period and cohort differences and adjusting for age and sex, REACH VET identification was associated with having more completed outpatient appointments (adjusted TD [ATD], 0.31; 95% CI, 0.06 to 0.55) and lower proportions of scheduled outpatient appointments that were missed (1% reduced; ATD, −0.01; 95% CI, −0.02 to −0.01). We did not observe significant program associations with scheduled outpatient appointments or outpatient mental health visit volume. REACH VET was associated with reduced inpatient mental health admissions (ATD, −0.08; 95% CI, −0.10 to −0.05) and ED visit days (ATD, −0.03; 95% CI, −0.06 to −0.01).

REACH VET identification was associated with reduced likelihood of suicide attempt documentation in the subsequent 6 months. Accounting for potential period and cohort differences, REACH VET was associated with a 5% reduction in the probability of documentation of a suicide attempt, adjusting for age and sex (ATD, −0.05; 95% CI, −0.06 to −0.03). In unadjusted TD analyses, individuals identified by REACH VET were less likely to have documented suicide safety plans in the subsequent 6 months compared with the prior 6 months (TD, −0.02; 95% CI, −0.03 to −0.01), but the difference was no longer significant when adjusted for age and sex.

Individuals in REACH VET were more likely to have new suicide safety plans within 6 months (Table 3). Accounting for period and cohort differences and age and sex, REACH VET was associated with an increase in the probability of receiving a new suicide safety plan (ATD, 0.08; 95% CI, 0.06 to 0.10). Sensitivity analyses using 3-month periods generated consistent findings, except that analyses for proportion with safety plan documentation was statistically significant.

REACH VET was not associated with all-cause mortality within 6 months, per the VHA Vital Status File (age- and sex-adjusted difference-in-difference[ADiD], <0.0001; 95% CI, −0.0001 to 0.0001). Subcohort analyses using death certificate data through 2018 did not identify significant program associations with suicide (ADiD, 0.0007; 95% CI, −0.0006 to 0.0019), nonsuicide external cause (ADiD, 0.0001; 95% CI, −0.0015 to 0.0018), or all-cause mortality (ADiD, 0.0001; 95% CI, −0.0001 to 0.0002). Table 4 provides information regarding 6-month mortality for the REACH VET and concurrent subthreshold subcohorts and the pre–REACH VET top 0.1% and subthreshold cohorts.

This cohort study found that REACH VET was associated with increased completed outpatient appointments; reduced missed appointments, inpatient mental health admissions, and ED visit days; greater initiation of suicide safety plans; and reductions in documented suicide attempts. This last is noteworthy, as focused clinical attention may increase VHA ascertainment of suicide attempts, in which case study findings could underestimate this association. REACH VET was not associated with changes in overall outpatient mental health encounter volume. Future studies should assess whether this lack of change results from altered patterns of health care utilization, with increased regular and coordinated treatment maintenance contacts and decreased episodic encounters.

The feasibility and utility of suicide predictive modeling has long been anticipated and debated.^20-22 The VHA has completed national implementation of a clinical program using a risk prediction algorithm to identify individuals with high suicide risk for care enhancements.

This study has some limitations. First, cause of death data were not available for the entire cohort. Subcohort analyses assessed available data (146 223 observations). These did not identify associations, yet may be underpowered. Power analyses using Lehr equation adapted for binomial outcomes²³ suggest that at least 430 598 observations would be required to detect a 10% reduction in 6-month suicide mortality. Future studies are warranted when more recent mortality data becomes available. More work is needed to understand site ascertainment of nonfatal suicide attempts. It was not possible to test whether individual program components were associated with study outcomes. Additionally, while outcomes were assessed for 6-month periods, considering alternative periods may be helpful.

This cohort study evaluated multiple aspects of program implementation and effectiveness of REACH VET, the nation’s first clinical implementation of a validated algorithm to support suicide risk identification. Results document the dynamic nature of health care utilization in the nation’s largest integrated health system, requiring TD analyses to adjust for nonparallel trends. Understanding impacts of the REACH VET program has clinical, scientific, and public policy significance, and REACH VET has received substantial public and Congressional attention.^24,25

Since the development of the original VA suicide prediction model in 2015,⁸ there has been greater acceptance that electronic health records may contribute important tools for improved suicide risk identification to support care enhancement.²⁶ Implementation of the REACH VET program has been substantial and program inclusion was associated with greater treatment engagement and safety plan documentation, fewer inpatient mental health admissions and emergency department visit days, and reduced prevalence of nonfatal suicide attempts. REACH VET represents a promising intervention that enhances care. The program is the direct result of long-term VA investment^{1-3,5,8-10,16} in suicide surveillance, data analytics, and clinical operations.

Accepted for Publication: August 15, 2021.

Published: October 18, 2021. doi:10.1001/jamanetworkopen.2021.29900

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 McCarthy JF et al. JAMA Network Open.

Corresponding Author: John F. McCarthy, PhD, MPH, 2800 Plymouth Road, Building 16, Floor 2, Ann Arbor, MI 48109 (john.mccarthy2@va.gov).

Author Contributions: Ms Cooper and Ms Dent had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: McCarthy, Dent, Eagan, Landes, Trafton, Schoenbaum, Katz.

Acquisition, analysis, or interpretation of data: McCarthy, Cooper, Dent, Eagan, Matarazzo, Hannemann, Reger, Trafton, Schoenbaum, Katz.

Drafting of the manuscript: McCarthy, Dent.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Cooper, Dent, Hannemann, Schoenbaum.

Obtained funding: Eagan, Landes.

Administrative, technical, or material support: McCarthy, Eagan, Trafton, Katz.

Supervision: McCarthy, Eagan, Trafton.

Conflict of Interest Disclosures: The authors indicate no conflicts of interest. Dr Trafton reported being employed by and serving on the board of directors of the Institute for Brain Potential. No other disclosures were reported.

Funding/Support: This study was conducted as part of the quality improvement operations analyses in the Veterans Affairs Office of Mental Health and Suicide Prevention.

Role of the Funder/Sponsor: The Office of Mental Health and Suicide Prevention at the US Department of Veterans Affairs reviewed the manuscript but had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, or approval of the manuscript; or decision to submit the manuscript for publication.