Proportion of patients with suicidal behavior in the 6 months following emergency department discharge in SPI+ compared with usual care patients. Error bars denote the standard error of the proportion.
Proportion of patients with at least 1 outpatient behavioral health appointment in the 6 months following emergency department discharge in SPI+ compared with usual care patients. Error bars denote the standard error of the proportion.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Stanley B, Brown GK, Brenner LA, et al. Comparison of the Safety Planning Intervention With Follow-up vs Usual Care of Suicidal Patients Treated in the Emergency Department. JAMA Psychiatry. 2018;75(9):894–900. doi:10.1001/jamapsychiatry.2018.1776
Can a brief suicide prevention intervention reduce suicidal behaviors and improve treatment engagement among patients who present to the emergency department for suicide-related concerns?
In this cohort comparison study, patients who visited the emergency department for suicide-related concerns and received the Safety Planning Intervention with structured follow-up telephone contact were half as likely to exhibit suicidal behavior and more than twice as likely to attend mental health treatment during the 6-month follow-up period compared with their counterparts who received usual care following their ED visit.
The Safety Planning Intervention with structured follow-up telephone contact may be an effective brief suicide prevention intervention that can be implemented in emergency departments.
Suicidal behavior is a major public health problem in the United States. The suicide rate has steadily increased over the past 2 decades; middle-aged men and military veterans are at particularly high risk. There is a dearth of empirically supported brief intervention strategies to address this problem in health care settings generally and particularly in emergency departments (EDs), where many suicidal patients present for care.
To determine whether the Safety Planning Intervention (SPI), administered in EDs with follow-up contact for suicidal patients, was associated with reduced suicidal behavior and improved outpatient treatment engagement in the 6 months following discharge, an established high-risk period.
Design, Setting, and Participants
Cohort comparison design with 6-month follow-up at 9 EDs (5 intervention sites and 4 control sites) in Veterans Health Administration hospital EDs. Patients were eligible for the study if they were 18 years or older, had an ED visit for a suicide-related concern, had inpatient hospitalization not clinically indicated, and were able to read English. Data were collected between 2010 and 2015; data were analyzed between 2016 and 2018.
The intervention combines SPI and telephone follow-up. The SPI was defined as a brief clinical intervention that combined evidence-based strategies to reduce suicidal behavior through a prioritized list of coping skills and strategies. In telephone follow-up, patients were contacted at least 2 times to monitor suicide risk, review and revise the SPI, and support treatment engagement.
Main Outcomes and Measures
Suicidal behavior and behavioral health outpatient services extracted from medical records for 6 months following ED discharge.
Of the 1640 total patients, 1186 were in the intervention group and 454 were in the comparison group. Patients in the intervention group had a mean (SD) age of 47.15 (14.89) years and 88.5% were men (n = 1050); patients in the comparison group had a mean (SD) age of 49.38 (14.47) years and 88.1% were men (n = 400). Patients in the SPI+ condition were less likely to engage in suicidal behavior (n = 36 of 1186; 3.03%) than those receiving usual care (n = 24 of 454; 5.29%) during the 6-month follow-up period. The SPI+ was associated with 45% fewer suicidal behaviors, approximately halving the odds of suicidal behavior over 6 months (odds ratio, 0.56; 95% CI, 0.33-0.95, P = .03). Intervention patients had more than double the odds of attending at least 1 outpatient mental health visit (odds ratio, 2.06; 95% CI, 1.57-2.71; P < .001).
Conclusions and Relevance
This large-scale cohort comparison study found that SPI+ was associated with a reduction in suicidal behavior and increased treatment engagement among suicidal patients following ED discharge and may be a valuable clinical tool in health care settings.
Quiz Ref IDThe suicide rate in the United States increased dramatically over the past 2 decades, with more than 44 000 suicides in 2016.1 The rate among veterans is 21% higher than nonveterans.2 Development of effective treatments has not kept pace. Often the only treatment patients receive during a suicidal crisis is in hospital emergency departments (EDs),3-5 in which care usually consists of risk assessment and triage to inpatient or outpatient treatment. More than 4% of ED visits are attributable to psychiatric conditions,6 with approximately 420 000 visits annually for intentional self-harm.7 Patients are at increased risk for suicide attempts and suicide following an ED visit.8-11 Unfortunately, nearly half of suicidal ED patients do not attend treatment12,13 or discontinue quickly.14 Poor engagement is troubling because risk for subsequent suicidal behavior is greatest during the 6 months following a suicidal crisis.15 Given this picture, the ED visit is an important opportunity for brief, targeted interventions to prevent further suicidal crises.
Up to 10 years ago, no-suicide contracts, in which patients promised to not engage in suicidal behavior,16-18 were frequently used. However, these contracts are ineffective.16,19-21 The Joint Commission22 recommended development of a collaborative safety plan as an alternative to no-suicide contracts.23,24 Safety planning has been found to be acceptable to and feasible by both suicidal patients and staff. Patients report it helps maintain their safety.25,26 In a randomized clinical trial, crisis response planning, a form of safety planning, resulted in fewer suicide attempts, lower suicidal ideation, and greater treatment engagement than no-suicide contracts during a 6-month follow-up with active duty military members.27
Postdischarge follow-up interventions, particularly those that involve in-person and telephone contact, are effective.28-32 One ED study found a 30% lower suicide attempt rate with an intervention consisting of screening, providing basic written material on safety planning, and several calls to suicidal patients and their significant others.30 Some33-35 but not all36 studies have reported similar findings with mail interventions.
Given the potential of safety planning and follow-up contact in reducing suicidal behavior, we evaluated a combination intervention, SPI+, which consisted of (1) the Safety Planning Intervention (SPI)24 administered in the ED and (2) structured follow-up37 following ED discharge to prevent suicidal behavior and enhance treatment engagement. The SPI is a brief clinical intervention24,38 widely used in health systems including the Veterans Health Administration.38 The SPI combines evidence-based strategies to reduce suicidal behavior by providing prioritized coping strategies including lethal means counseling to reduce access to potential suicide methods. Coping strategies were prioritized, ranging from strategies that can be done alone to those involving social contacts (ie, family and friends), followed by outreach to professionals and the ED. The follow-up component (SPI+) consisted of at least 2 brief telephone calls following ED discharge to assess risk, review and revise the safety plan, and support treatment engagement. Calls continued on a weekly basis until the patient began treatment or withdrew. In a pilot study of repeated ED patients, SPI+ was associated with improved treatment attendance at 3-month follow-up compared with a prior ED visit for suicidal behavior.39 We also found that, among suicidal patients provided with a safety plan, nearly two-thirds reported using the plan to mitigate risk.25 The purpose of this cohort comparison study was to evaluate the association of SPI+ compared with usual care with suicidal behavior and treatment engagement in the 6 months following discharge from the ED. We hypothesized that SPI+ would be associated with fewer suicidal behavior reports and that SPI+ would be associated with enhanced treatment engagement in the 6 months following ED discharge.
Quiz Ref IDA total of 1640 patients (1186 in the intervention group and 454 in the comparison group) meeting eligibility criteria from 5 Veterans Affairs (VA) ED intervention sites where SPI+ was implemented as standard care and 4 VA ED comparison usual care sites were included in the analyses. Patients were not prospectively assigned to intervention or comparison groups. Instead, in the intervention site EDs, all eligible patients were provided the SPI+ as standard care. Matched sites that did not have the intervention were identified later to obtain electronic health record data during the same time as a comparison. Sites were matched on geographic location, approximate number of psychiatric ED evaluations per year, and presence of a psychiatric inpatient unit. Each ED treated a mean of approximately 10 000 patients per year; about 10% of these patients were admitted for mental health/substance abuse treatment.
Patients who came to the ED for a suicidal crisis and were determined to not require inpatient hospitalization were included. Usual care for suicidal patients not requiring inpatient hospitalization typically consisted of evaluation and outpatient referral. Emergency department patients who met eligibility criteria were included in the project. Inclusion criteria were (1) 18 years or older; (2) ED visit for a suicide-related concern; (3) inpatient hospitalization not clinically indicated; and (4) able to read and understand English. This project was reviewed and approved by each of the VA institutional review boards at the VA Medical Center of Denver, Colorado, the Manhattan VA Medical Center, the Portland Corporal Michael J. Crescenz VA Medical Center, and the VA Medical Center of Philadelphia, Pennsylvania. Because this was a clinical demonstration project, the institutional review boards determined that informed consent was not required. Patients were able to opt out of the SPI+ during the ED visit or during follow-up calls.
Quiz Ref IDIn addition to usual care, SPI+ included the original SPI, a brief, structured intervention24,38 and a best practice on the Suicide Prevention Resource Center/American Foundation for Suicide Prevention Best Practices Registry (http://www.sprc.org), designed to mitigate future risk by providing suicidal individuals with a written, personalized safety plan to be used in the event of a suicidal crisis. The SPI has 6 key steps: (1) identify personalized warning signs for an impending suicide crisis; (2) determine internal coping strategies that distract from suicidal thoughts and urges; (3) identify family and friends who are able to distract from suicidal thoughts and urges and social places that provide the opportunity for interaction; (4) identify individuals who can help provide support during a suicidal crisis; (5) list mental health professionals and urgent care services to contact during a suicidal crisis; and (6) lethal means counseling for making the environment safer (http://www.suicidesafetyplan.com).
The SPI+ adds a component that consists of telephone contact after discharge from the ED, usually done by project staff who were social workers or psychologists and trained and supervised by senior project staff. Contacts were attempted within 72 hours of discharge and included 3 components: (1) brief risk assessment and mood check; (2) review and revision of the SPI, if needed; and (3) facilitation of treatment engagement. Follow-up outreach continued weekly and generally discontinued after at least 2 calls if the patient had at least 1 outpatient behavioral health appointment or no longer wished to be contacted.
Usual care varied somewhat between sites because it was not protocol driven; it generally consisted of an initial assessment by a nurse or social worker followed by a secondary evaluation by an ED physician. Medical care was provided if indicated and the patient was medically stabilized. Medications were initiated or adjusted as indicated. Patients discharged were typically provided with either a specific outpatient appointment or information about how to seek psychiatric care if they declined a referral at discharge. Usual care patients did not receive a safety plan during the ED visit, although some may have received one in the past if they were at risk for suicide. Patients in the usual care condition were identified retrospectively by clinical medical record review for the same time that SPI+ was implemented in the intervention EDs.
Medical records were reviewed to obtain demographic information, diagnoses, health service use, and physician-rated Global Assessment of Functioning scores at the index ED visit. Suicide Behavior Reports for suicide behaviors in the postdischarge 6 months were also retrieved from the medical records at each hospital. These reports are mandated in the VA and include descriptions of all suicide attempts, suicide deaths, and other suicidal behaviors including interrupted attempts.2,40
All statistical analyses were performed in the statistical language R, version 3.1.2 (2014-10-31, The R Foundation for Statistical Computing). All tests performed were 2-sided and used α = .05 for significance level cutoff. All eligible patients were included in the analyses. Comparisons of demographic and clinical characteristics between patients receiving SPI+ and usual care were conducted using t tests for quantitative measures and Pearson χ2 tests for categorical measures. To account for intrasite correlations in the outcome measures, mixed-effect logistic regression models were run using the “glmer” function41 in R, with intervention group as a fixed effect and with site-specific random intervention effects to account for differences in treatment outcome by site to test for differences in suicidal behavior reports between the intervention and comparison groups. This generalized linear mixed model uses the logit link and was estimated using maximum likelihood methods through the Laplace approximation. We performed 2 analyses using this model to examine whether treatment condition predicted suicidal behavior during the 6-month follow-up, 1 with only treatment condition as the independent variable and 1 where we controlled for patient suicidal behavior in the 6 months preintervention (not including suicidal behavior that was the reason for the ED visit). Because patients were not randomized, we next calculated propensity scores based on variables that differed by condition. Significant predictors of group membership (Table) with less than 5% missing data, namely age, homelessness status, service period Operation Enduring Freedom or Operation Iraqi Freedom vs other service periods (World War II, Vietnam-era, post-Vietnam, and Gulf War), indicator variables for a history of more than 5 mental health visits, bipolar, depression, posttraumatic stress disorder (PTSD), and substance abuse diagnoses and suicidal behavior 6 months preintervention, were entered into a binary logistic regression model with group as outcome, and the propensity score was calculated as the patient-level predicted values of likelihood of belonging to the intervention vs control group. We used propensity scores to perform covariate adjustment: first with a categorical propensity variable by breaking the propensity score into quartiles (ie, 4 strata) and then with a continuous covariate. We adjusted the mixed-effect logistic model with postintervention suicidal behavior as the outcome variable by the strata and strata by condition interaction, removing the interaction when found to be not significant. We repeated the analysis with the quantitative propensity score to test for linear effect. Owing to the relatively low event rate, we did not perform separate analyses by propensity score strata. In exploratory analyses, the effect of baseline diagnosis on treatment outcome was tested using separate mixed-effect logistic regression models as described here, with randomization group, baseline diagnosis, and their interaction as fixed predictors.
A total of 1640 patients with a mean (SD) age of 48 (14) years, 88% men, were included in the analyses. Of the eligible patients at the intervention sites, 99.4% (n = 1179) agreed to receive the SPI. Fewer patients (89.6%; n = 1063) were able to be contacted for at least 1 follow-up call. Patients completed a mean (SD) of 3.7 (3.3) calls (median, 4; range, 0-26).
Patients in the SPI+ and usual care conditions did not differ on race/ethnicity or sex. However, there were some baseline differences between conditions. Specifically, those in SPI+ were significantly younger (t1625 = 2.716; P = .007), less educated (χ21 = 4.627; P = .03), and less likely to be homeless (χ21 = 8.453; P = .004) than those in usual care (Table). Those in the SPI+ group were more likely to have more than 5 mental health visits in the prior 3 years than those in the usual care (Table). Also, patients in SPI+ were also more likely to have served in the 2 most recent military operations (Operation Enduring Freedom or Operation Iraqi Freedom) vs other service periods (Table).
Patients in SPI+ did not differ significantly from those in usual care on Global Assessment of Functioning at the index ED visit. Patients in the SPI+ condition were more likely to have had suicidal behavior in the 6 months previous (χ21 = 8.31; P = .004). Patients in the SPI+ condition were significantly less likely to have had the following lifetime diagnoses: major depression, bipolar disorder, PTSD, and substance abuse (Table).
Quiz Ref IDThe SPI+ was associated with 45% fewer suicidal behaviors in the 6-month period following the ED visit compared with usual care (Figure 1). Patients in the SPI+ condition were less likely to engage in suicidal behavior (n = 36 of 1186; 3.03%) than those receiving usual care (n = 24 of 454; 5.29%) during the 6-month follow-up period, yielding a number needed to treat of 44.43. Mixed-effect logistic regression analysis examining whether treatment condition was related to suicidal behavior found that when we included treatment condition as the independent variable and adjusted for random intervention effects by site, SPI+ had approximately half the odds of suicidal behavior reports during the study period (OR, 0.50; 95% CI, 0.25-0.99; P = .05). In a secondary analysis using the mixed-effect logistic model, when we controlled for whether the patient had a history of suicidal behavior in the 6 months preintervention, we found that treatment condition remained significantly associated with suicidal behavior 6 months following intervention (odds ratio [OR], 0.48; 95% CI, 0.24-0.93; P = .03). Preintervention suicidal behavior was found to be a significant factor in this model (OR, 2.93; 95% CI, 1.16-7.37; P = .02). In another secondary analysis, we used the propensity score to adjust for the significant differences between the groups. The propensity score for each individual in SPI+ vs comparison groups was calculated based on significant factors from the Table with less than 5% missing values, as described previously. Patients were stratified into 4 groups based on its quartiles. In a mixed-effect logistic regression with condition, propensity-based strata and their interaction as predictors, we found no evidence for differential treatment effect by strata (χ23 = 1.38; P = .71), and after removing the interaction, we found that treatment condition remained significant (OR, 0.47; 95% CI, 0.24-0.92; P = .03), while the propensity strata was also significant (χ23 = 17.88; P < .001). Adjusting for the propensity score as a linear covariate yielded a significant treatment effect (OR, 0.46; 95% CI, 0.23-0.91; P = .03) and a nonsignificant propensity score effect (OR, 1.08; 95% CI, 0.81-1.43; P = .60).
In exploratory analyses, we tested the association of baseline diagnosis with suicidal behavior outcomes using separate mixed-effect logistic regression models. Depression, PTSD, and substance use disorder had no moderating effect on the treatment (interaction with treatment: depression OR, 0.62; 95% CI, 0.21-1.85; P = .39; PTSD OR, 0.37; 95% CI, 0.11-1.28; P = .12; and substance abuse OR, 2.72; 95% CI, 0.87-8.51; P = .09) nor were they significantly associated with suicidal behavior during follow-up overall (depression: z = −0.71; P = .48; PTSD: z = −1.05; P = .29; substance abuse: z = 0.22; P = .83). However, bipolar disorder was a significant moderator of treatment effect (interaction OR, 7.05; 95% CI, 1.30-38.18; P = .02), whereby patients with bipolar disorder who received SPI+ did not differ in the risk of suicidal behavior from those in the usual care condition (SPI: OR, 2.1; 95% CI, 0.28-15.73; P = .47). For patients with all other diagnoses, those in the SPI+ intervention had fewer suicidal behaviors in the 6-month postintervention period (OR, 0.40; 95% CI, 0.22-0.75; P = .05).
Using mixed-effect logistic models, we tested whether the intervention was associated with the likelihood of attending mental health and/or substance abuse treatment (at least 1 visit in the 6 months following ED discharge). Patients in SPI+ had more than double the odds of attending mental health treatment (OR, 2.20; 95% CI, 1.35-2.20; P = .002) (Figure 2), and the difference stayed significant after adjusting for the propensity score strata (OR, 1.74; 95% CI, 1.08-2.81; P = .02). The likelihood of attending substance abuse treatment did not differ between the groups (OR, 1.23; 95% CI, 0.88-1.71; P = .22), even after adjusting for the propensity score strata (OR, 1.44; 95% CI, 0.90-2.28; P = .13).
When entering both treatment group and mental health treatment attendance as predictors of posttreatment suicidal behavior, the treatment group effect stayed significant, and the effect size did not decrease (OR, 0.46; 95% CI, 0.23-0.91; P = .03), indicating that the protective effect of the intervention was not mediated by the increased likelihood of attending mental health treatment. We also examined the association between the number of follow-up calls and suicidal behavior within the SPI+ group and found that the number of calls was not associated with whether the patient had at least 1 suicide event (Wilcoxon W = 23 711.5; P = .13), possibly pointing to a key role of the safety plan.
To our knowledge, this is the first large-scale study to demonstrate the association of SPI+ with decreased suicidal behavior and increased behavioral health treatment engagement following ED discharge for suicide-related concerns. The SPI+ was associated with about 50% fewer suicidal behaviors over a 6-month follow-up and more than double the odds of engaging in outpatient behavioral health care. The significant decrease in suicidal behavior underscores the utility of SPI+ as an effective prevention strategy in EDs. Interestingly, our mediation analysis demonstrated that the association with suicidal behavior was not attributable to the increased treatment engagement in the intervention group nor was it accounted for by the number of follow-up calls. This finding suggests that SPI+ has a positive association with suicidal behavior apart from attendance in outpatient behavioral health care. We chose attendance at more than 1 outpatient appointment as our outcome because the principal difficulty with suicidal patients is that they tend to not attend a first appointment. Because patients admitted to inpatient units were not included, the range of suicidality in our sample is restricted to a lower-risk population. The reach of this intervention may be greater if the range of suicide risk is not restricted and the intervention is provided to all suicidal patients including those admitted to inpatient units.
Our results are consistent with findings that crisis response planning reduced suicide attempts during a 6-month follow-up period in high-risk active duty soldiers.42 Crisis response planning is an abbreviated form of safety planning that uses 4 of 6 elements of the SPI (without social interaction as a means of distracting from suicidal thoughts and lethal means counseling). Our results are also consistent with past research30 that found that a combination of in-person intervention using screening and a safety plan coupled with telephone phone follow-up reduced suicide attempts.
Quiz Ref IDThere are several limitations to consider. Although intervention sites were matched with control sites and the analyses used propensity scoring, this was not a randomized trial, so there is a potential for confounding. Given that this was a clinical demonstration project, limited information is available about individual patients. We were reliant on medical records for suicidal behavior reports and diagnostic and demographic information. Some suicidal behaviors may have occurred that were not entered into the medical record because staff were unaware of them. However, we have no reason to believe that this occurred more often in either condition. Also, some patients in usual care may have had a safety plan in prior treatment. This could have diminished the difference between the SPI+ and usual care groups. Furthermore, this project was conducted in Veterans Health Administration hospital EDs with predominantly men; therefore, we do not know how well our findings generalize to civilian settings and women.
Additional limitations are the low observed suicide event rate in both groups and the lower proportion of eligible patients in the comparison group. This low rate probably resulted from excluding patients admitted to inpatient units from the ED. Thus, the study likely included patients at lower risk for suicide. Interestingly, this lower-than-expected rate may indicate that ED clinicians are correctly hospitalizing higher-risk patients. Another possibility is that some suicide events were not captured by suicide behavior reports either because reports were not written or patients did not reveal all events to clinicians. Given that our sample was limited to patients who presented to an ED for suicide-related concerns but were not hospitalized, the efficacy of SPI+ is unknown for patients who required immediate psychiatric hospitalization. Further randomized trials with the full range of suicidal patients are required to assess the effect of SPI+ on suicidal behavior.
Our findings are promising and indicate that safety planning and active outreach, a set of low-burden strategies, are useful components of effective suicide prevention. Importantly, using the low-burden intervention in this project and others30 was associated with about the same reduction in suicidal behavior as more intensive and costly psychosocial interventions.43,44 If implemented broadly, SPI+ has the potential to reduce suicidal behavior and enhance behavioral health treatment engagement, particularly during high-risk periods following ED discharge. This strategy may help decrease suicide risk in the long term.
Corresponding Author: Barbara Stanley, PhD, Columbia University, New York State Psychiatric Institute, 1051 Riverside Dr, Unit 42, New York, NY 10032 (firstname.lastname@example.org).
Accepted for Publication: August 18, 2018.
Published Online: July 11, 2018. doi:10.1001/jamapsychiatry.2018.1776
Author Contributions: Drs Stanley and Brown had full access to all data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Stanley, Brown, Brenner, Currier, Knox, Green.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Stanley, Brown, Brenner, Galfalvy, Currier, Knox, Chaudhury, Bush.
Critical revision of the manuscript for important intellectual content: Stanley, Brown, Brenner, Galfalvy, Green.
Statistical analysis: Stanley, Galfalvy, Currier.
Obtained funding: Stanley, Brown, Currier.
Administrative, technical, or material support: Stanley, Brown, Brenner, Currier, Bush, Green.
Supervision: Brown, Knox.
Conflict of Interest Disclosures: None reported.
Funding/Support: This project was supported by the Office of Mental Health Services, Department of Veterans Affairs.
Role of the Funder/Sponsor: The Department of Veterans Affairs assisted in the design and conduct of the study but did not have a role in the data analysis; the interpretation of the data; preparation, review or approval of the manuscript; and the decision to submit the manuscript for publication.
Additional Contributions: Site leaders were Lisa Brenner, PhD, Denver VA Medical Center; Joan Chipps, LCSW, Buffalo VA Medical Center; Joshua Hooberman, PhD, Manhattan VA Medical Center; Christine Jackson, PhD, Manhattan VA Medical Center; Mitchell Kling, MD, Corporal Michael J. Crescenz VA Medical Center; and Keith Rogers, MD, Portland VA Medical Center. Acute services coordinators were Patricia Alexander, PhD, Denver VA Medical Center; Laura Blandy, PsyD, Corporal Michael J. Crescenz VA Medical Center; Aimee Coughlin, MSW, Portland VA Medical Center; John Dennis, PhD, Corporal Michael J. Crescenz VA Medical Center; Michael Miello, PhD, Buffalo VA Medical Center; Katherine Mostkoff, LCSW, Manhattan VA Medical Center; and Jarrod Reisweber, PsyD, Corporal Michael J. Crescenz VA Medical Center. Site lead psychiatrists were Steven Dobscha, MD, Portland VA Medical Center; Keith Rogers, MD, Portland VA Medical Center; Donald Tavakoli, MD, Corporal Michael J. Crescenz VA Medical Center; and Adam Wolkin, MD, Manhattan VA Medical Center. Site leads and site psychiatrists were not compensated, and acute service coordinators were compensated through their work as project staff employees. We thank Ira Katz, MD, and Janet Kemp, PhD, who were instrumental in facilitating the implementation of this project, and Marjan Ghahramanlou Holloway, PhD, who provided guidance. We would also like to acknowledge Walter Matweychuk, PhD and Lauren Denneson, PhD, for their contributions in on-site implementation and staff supervision.