[Skip to Content]
[Skip to Content Landing]
Figure.  Participant Flowchart
Participant Flowchart
Table 1.  Demographic and Clinical Characteristics
Demographic and Clinical Characteristics
Table 2.  Suicide-Specific Outcomes at Baseline and 12 Months
Suicide-Specific Outcomes at Baseline and 12 Months
Table 3.  Hurdle Model and Logistic Regression Analysesa
Hurdle Model and Logistic Regression Analysesa
1.
Kessler  RC, Warner  CH, Ivany  C,  et al; Army STARRS Collaborators.  Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study To Assess Risk and Resilience in Servicemembers (Army STARRS).  JAMA Psychiatry. 2015;72(1):49-57. doi:10.1001/jamapsychiatry.2014.1754PubMedGoogle ScholarCrossref
2.
Hoge  CW, Castro  CA, Messer  SC, McGurk  D, Cotting  DI, Koffman  RL.  Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care.  N Engl J Med. 2004;351(1):13-22. doi:10.1056/NEJMoa040603PubMedGoogle ScholarCrossref
3.
Hoge  CW, Auchterlonie  JL, Milliken  CS.  Mental health problems, use of mental health services, and attrition from military service after returning from deployment to Iraq or Afghanistan.  JAMA. 2006;295(9):1023-1032. doi:10.1001/jama.295.9.1023PubMedGoogle ScholarCrossref
4.
Kim  PY, Thomas  JL, Wilk  JE, Castro  CA, Hoge  CW.  Stigma, barriers to care, and use of mental health services among active duty and National Guard soldiers after combat.  Psychiatr Serv. 2010;61(6):582-588. doi:10.1176/ps.2010.61.6.582PubMedGoogle ScholarCrossref
5.
VanSickle  M, Werbel  A, Perera  K, Pak  K, DeYoung  K, Ghahramanlou-Holloway  M.  Perceived barriers to seeking mental health care among United States Marine Corps noncommissioned officers serving as gatekeepers for suicide prevention.  Psychol Assess. 2016;28(8):1020-1025. doi:10.1037/pas0000212PubMedGoogle ScholarCrossref
6.
Hoge  CW, Lesikar  SE, Guevara  R,  et al.  Mental disorders among U.S. military personnel in the 1990s: association with high levels of health care utilization and early military attrition.  Am J Psychiatry. 2002;159(9):1576-1583. doi:10.1176/appi.ajp.159.9.1576PubMedGoogle ScholarCrossref
7.
Luxton  DD, June  JD, Comtois  KA.  Can postdischarge follow-up contacts prevent suicide and suicidal behavior? a review of the evidence.  Crisis. 2013;34(1):32-41. doi:10.1027/0227-5910/a000158PubMedGoogle ScholarCrossref
8.
Motto  JA.  Suicide prevention for high-risk persons who refuse treatment.  Suicide Life Threat Behav. 1976;6(4):223-230.PubMedGoogle Scholar
9.
Motto  JA, Bostrom  AG.  A randomized controlled trial of postcrisis suicide prevention.  Psychiatr Serv. 2001;52(6):828-833. doi:10.1176/appi.ps.52.6.828PubMedGoogle ScholarCrossref
10.
Fleischmann  A, Bertolote  JM, Wasserman  D,  et al.  Effectiveness of brief intervention and contact for suicide attempters: a randomized controlled trial in five countries.  Bull World Health Organ. 2008;86(9):703-709. doi:10.2471/BLT.07.046995PubMedGoogle ScholarCrossref
11.
Hassanian-Moghaddam  H, Sarjami  S, Kolahi  A-A, Carter  GL.  Postcards in Persia: randomised controlled trial to reduce suicidal behaviours 12 months after hospital-treated self-poisoning.  Br J Psychiatry. 2011;198(4):309-316. doi:10.1192/bjp.bp.109.067199PubMedGoogle ScholarCrossref
12.
Hassanian-Moghaddam  H, Sarjami  S, Kolahi  A-A, Lewin  T, Carter  G.  Postcards in Persia: a twelve to twenty-four month follow-up of a randomized controlled trial for hospital-treated deliberate self-poisoning.  Arch Suicide Res. 2017;21(1):138-154. doi:10.1080/13811118.2015.1004473PubMedGoogle ScholarCrossref
13.
Carter  GL, Clover  K, Whyte  IM, Dawson  AH, D’Este  C.  Postcards from the edge project: randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated deliberate self poisoning.  BMJ. 2005;331(7520):805-807. doi:10.1136/bmj.38579.455266.E0PubMedGoogle ScholarCrossref
14.
Carter  GL, Clover  K, Whyte  IM, Dawson  AH, D’Este  C.  Postcards from the edge: 24-month outcomes of a randomised controlled trial for hospital-treated self-poisoning.  Br J Psychiatry. 2007;191:548-553. doi:10.1192/bjp.bp.107.038406PubMedGoogle ScholarCrossref
15.
Carter  GL, Clover  K, Whyte  IM, Dawson  AH, D’Este  C.  Postcards from the edge: 5-year outcomes of a randomised controlled trial for hospital-treated self-poisoning.  Br J Psychiatry. 2013;202(5):372-380. doi:10.1192/bjp.bp.112.112664PubMedGoogle ScholarCrossref
16.
Robinson  J, Yuen  HP, Gook  S,  et al.  Can receipt of a regular postcard reduce suicide-related behaviour in young help seekers? a randomized controlled trial.  Early Interv Psychiatry. 2012;6(2):145-152. doi:10.1111/j.1751-7893.2011.00334.xPubMedGoogle ScholarCrossref
17.
Kapur  N, Gunnell  D, Hawton  K,  et al.  Messages from Manchester: pilot randomised controlled trial following self-harm.  Br J Psychiatry. 2013;203(1):73-74. doi:10.1192/bjp.bp.113.126425PubMedGoogle ScholarCrossref
18.
Beautrais  AL, Gibb  SJ, Faulkner  A, Fergusson  DM, Mulder  RT.  Postcard intervention for repeat self-harm: randomised controlled trial.  Br J Psychiatry. 2010;197(1):55-60. doi:10.1192/bjp.bp.109.075754PubMedGoogle ScholarCrossref
19.
Berrouiguet  S, Gravey  M, Le Galudec  M, Alavi  Z, Walter  M.  Post-acute crisis text messaging outreach for suicide prevention: a pilot study.  Psychiatry Res. 2014;217(3):154-157. doi:10.1016/j.psychres.2014.02.034PubMedGoogle ScholarCrossref
20.
Chen  H, Mishara  BL, Liu  XX.  A pilot study of mobile telephone message interventions with suicide attempters in China.  Crisis. 2010;31(2):109-112. doi:10.1027/0227-5910/a000017PubMedGoogle ScholarCrossref
21.
Beck  AT, Kovacs  M, Weissman  A.  Assessment of suicidal intention: the Scale for Suicide Ideation.  J Consult Clin Psychol. 1979;47(2):343-352. doi:10.1037/0022-006X.47.2.343PubMedGoogle ScholarCrossref
22.
Beck  AT, Brown  GK, Steer  RA.  Psychometric characteristics of the Scale for Suicide Ideation with psychiatric outpatients.  Behav Res Ther. 1997;35(11):1039-1046. doi:10.1016/S0005-7967(97)00073-9PubMedGoogle ScholarCrossref
23.
Beck  AT, Brown  GK, Steer  RA, Dahlsgaard  KK, Grisham  JR.  Suicide ideation at its worst point: a predictor of eventual suicide in psychiatric outpatients.  Suicide Life Threat Behav. 1999;29(1):1-9. doi:10.1111/j.1943-278X.1999.tb00758.xPubMedGoogle Scholar
24.
Rudd  MD, Bryan  CJ, Wertenberger  EG,  et al.  Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up.  Am J Psychiatry. 2015;172(5):441-449. doi:10.1176/appi.ajp.2014.14070843PubMedGoogle ScholarCrossref
25.
LaCroix  JM, Perera  KU, Neely  LL, Grammer  G, Weaver  J, Ghahramanlou-Holloway  M.  Pilot trial of post-admission cognitive therapy: inpatient program for suicide prevention.  Psychol Serv. 2018;15(3):279-288. doi:10.1037/ser0000224PubMedGoogle ScholarCrossref
26.
Linehan  MM, Heard  HL. University of Washington Behavioral Research & Therapy Clinics Treatment History Interview (THI-4). http://depts.washington.edu/uwbrtc/wp-content/uploads/THI-4.pdf. Published 1996. Accessed January 10, 2019.
27.
Linehan  M, Comtois  KA. Suicide Attempt Self-Injury Count (SASI-Count). http://depts.washington.edu/uwbrtc/wp-content/uploads/L-SASI-Count.pdf. Published 1996. Accessed November 21, 2018.
28.
Linehan  MM, Comtois  KA, Brown  MZ, Heard  HL, Wagner  A.  Suicide Attempt Self-Injury Interview (SASII): development, reliability, and validity of a scale to assess suicide attempts and intentional self-injury.  Psychol Assess. 2006;18(3):303-312. doi:10.1037/1040-3590.18.3.303PubMedGoogle ScholarCrossref
29.
Posner  K, Brown  GK, Stanley  B,  et al.  The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults.  Am J Psychiatry. 2011;168(12):1266-1277. doi:10.1176/appi.ajp.2011.10111704PubMedGoogle ScholarCrossref
30.
Crosby  AE, Ortega  L, Melanson  C. Centers for Disease Control and Prevention. Self-directed violence surveillance: uniform definitions and recommended data elements. https://www.cdc.gov/violenceprevention/pdf/Self-Directed-Violence-a.pdf. Published 2011. Accessed January 7, 2019.
31.
Landau  S, Stahl  D.  Sample size and power calculations for medical studies by simulation when closed form expressions are not available.  Stat Methods Med Res. 2013;22(3):324-345. doi:10.1177/0962280212439578PubMedGoogle ScholarCrossref
32.
Atkins  DC, Baldwin  SA, Zheng  C, Gallop  RJ, Neighbors  C.  A tutorial on count regression and zero-altered count models for longitudinal substance use data.  Psychol Addict Behav. 2013;27(1):166-177. doi:10.1037/a0029508PubMedGoogle ScholarCrossref
33.
Graham  JW.  Missing data analysis: making it work in the real world.  Annu Rev Psychol. 2009;60:549-576. doi:10.1146/annurev.psych.58.110405.085530PubMedGoogle ScholarCrossref
34.
R Core Team; R Foundation for Statistical Computing. R: a language and environment for statistical computing. https://www.R-project.org/. Published 2017. Accessed January 7, 2019.
35.
Internal Revenue Service. Combat zones approved for tax benefits. https://www.irs.gov/newsroom/combat-zones. Published October 29, 2018. Accessed January 7, 2019.
Original Investigation
February 13, 2019

Effect of Augmenting Standard Care for Military Personnel With Brief Caring Text Messages for Suicide Prevention: A Randomized Clinical Trial

Author Affiliations
  • 1Department of Psychiatry & Behavioral Sciences, School of Medicine, University of Washington, Seattle
  • 2Womack Army Medical Center, Fort Bragg, North Carolina
  • 3Naval Medical Center Camp Lejeune, Camp Lejeune, North Carolina
JAMA Psychiatry. 2019;76(5):474-483. doi:10.1001/jamapsychiatry.2018.4530
Key Points

Question  Does a text message-based intervention (Caring Contacts) reduce suicidal thoughts and behaviors among active-duty military personnel?

Findings  In this randomized clinical trial of 658 Soldiers and Marines, augmenting standard care with Caring Contacts did not reduce current suicidal ideation or suicide risk events at 12-month follow-up. However, Caring Contacts reduced the odds of having any suicidal ideation (80% vs 88%) and making a suicide attempt (9% vs 15%).

Meaning  Although the primary hypotheses were not supported, Caring Contacts was found to be a simple, scalable intervention that may be effective in reducing the occurrence of suicide ideation and attempts.

Abstract

Importance  Accessible and cost-effective interventions for suicidality are needed to address high rates of suicidal behavior among military service members. Caring Contacts are brief periodic messages that express unconditional care and concern and have been previously shown to prevent suicide deaths, attempts, ideation, and hospitalizations.

Objective  To test the effectiveness of augmenting standard military health care with Caring Contacts delivered via text message to reduce suicidal thoughts and behaviors over 12 months.

Design, Setting, and Participants  This randomized clinical trial was conducted at 3 military installations in the southern and western United States. Soldiers and Marines identified as being at risk of suicide were recruited between April 2013 and September 2016. The final follow-up was in September 2017.

Interventions  Both groups received standard care, and the Caring Contacts group also received consisted of 11 text messages delivered on day 1, at week 1, at months 1, 2, 3, 4, 6, 8, 10, and 12, and on participants’ birthdays.

Main Outcomes and Measures  Primary outcomes were current suicidal ideation and suicide risk incidents (hospitalization or medical evacuation). Secondary outcomes were worst-point suicidal ideation, emergency department visits, and suicide attempts. Suicidal ideation was measured by the Scale for Suicide Ideation, suicide risk incidents, and emergency department visits by the Treatment History Interview; attempted suicide was measured by the Suicide Attempt Self-Injury Count.

Results  Among 658 randomized participants (329 randomizely assigned to each group), data were analyzed for 657 individuals (mean [SD] age, 25.2 [6.1] years; 539 men [82.0%]). All participants reported suicidal ideation at baseline, and 291 (44.3%) had previously attempted suicide. Of the 657 participants, 461 (70.2%) were assessed at 12 months. Primary outcomes were nonsignificant. There was no significant effect on likelihood or severity of current suicidal ideation or likelihood of a suicide risk incident; there was also no effect on emergency department visits. However, participants who received Caring Contacts (172 of 216 participants [79.6%]) had lower odds than those receiving standard care alone (179 of 204 participants [87.7%]) of experiencing any suicidal ideation between baseline and follow-up (odds ratio, 0.56 [95% CI, 0.33-0.95]; P = .03) and fewer had attempted suicide since baseline (21 of 233 [9.0%] in the group receiving Caring Contacts vs 34 of 228 [14.9%] in the standard-care group; odds ratio, 0.52 [95% CI, 0.29-0.92]; P = .03).

Conclusions and Relevance  This trial provides inconsistent results on the effectiveness of caring text messages between primary and secondary outcomes, but this inexpensive and scalable intervention offers promise for preventing suicide attempts and ideation in military personnel. Additional research is needed.

Trial Registration  ClinicalTrials.gov identifier: NCT01829620

Introduction

Suicide rates in the military remain elevated despite increased awareness and implementation of targeted interventions.1 Service members screening positive for mental health concerns, including suicide, frequently do not attend behavioral health outpatient care.2,3 Barriers to care, such as stigma and organizational obstacles (eg, difficulty getting time off work to attend treatment),2,4,5 are persistent despite considerable efforts within the Department of Defense (DoD) to reduce them. Mental health problems also contribute to attrition from military service.3,6 This highlights the need for suicide prevention interventions that do not require ongoing visits to a behavioral health clinic and can follow service members into civilian life.

Caring Contacts is an intervention that periodically reaches out to suicidal individuals by letter or other brief contact with nondemanding expressions of care, concern, and interest over 1 or more years.7 It is the only intervention shown to prevent death by suicide in randomized clinical trials, as was shown in the original study8,9 and an international trial.10 Use of Caring Contacts has also reduced hospitalizations, suicide attempts, and suicidal ideation in 2 large randomized clinical trials11-15 but has shown null effects in 3 smaller studies.16-18

Despite generally positive results across multiple clinical trials, Caring Contacts research has several limitations. First, previous studies have found improved suicide attempt outcomes for women but not men in samples in which women were most of the participants.10,11,13 Second, most studies have included individuals only after a suicide attempt or an emergency department or hospital admission. Its effectiveness for individuals with suicidal ideation of lower acuity is unknown. Third, none of these studies were conducted with military personnel or veterans. Finally, it is unclear whether Caring Contacts will work with newer communication tools. Pilot research on Caring Contacts via text message showed high feasibility and acceptability,19,20 but to our knowledge, this had not been rigorously evaluated in a clinical trial.

The present study is a randomized clinical trial of a Caring Contacts via text message intervention that extended continuity of care for suicidal active-duty Soldiers and Marines by sending 11 nondemanding caring text messages at regular intervals over 1 year. Participants were randomly assigned to receive Caring Contacts via text message in addition to standard care (Caring Contacts) or standard care alone. Hypotheses were that at 12-month follow-up, participants assigned to Caring Contacts would have (1) reduced current suicidal ideation, (2) lower odds of a suicide risk incident (ie, an incident requiring medical evacuation or inpatient admission), and (3) fewer suicide risk incidents compared with those receiving standard care alone. The study also made 3 other a priori exploratory hypotheses that, during the 12-month follow-up, participants in the Caring Contacts group would have (4) reduced worst-point ideation, (5) reduced emergency department visits, and (6) reduced suicide attempts.

Methods
Design

This was a randomized clinical trial with 2 arms. Informed written consent was obtained from all participants. Institutional review boards at the University of Washington, Womack Army Medical Center, and the US Army Medical Research and Materiel Command Human Research Protection Office, as well as a data and safety monitoring board and a research monitor, approved and monitored the research protocol and adverse events throughout the trial. The research protocol and statistical analysis plan are included in Supplement 1.

Setting

Recruitment took place between April 18, 2013, and September 19, 2016, at 3 military installations: an Army base in the southern United States, a Marine Corps base and nearby air stations in the southern United States, and a Marine Corps base in the western United States. All outcome assessments were completed by September 19, 2017.

Participants

Inclusion criteria were being (1) on active duty, in the Reserve, or in the National Guard; (2) 18 years or older; (3) English speaking; (4) identified to a behavioral health or medical service with suicidal ideation or a suicide attempt; (5) identified as having current suicidal ideation, defined by a score greater than 0 on the Scale for Suicide Ideation (SSI) administered at screening; and (6) in possession of a mobile telephone or pager that could affordably receive 11 text messages in a year. Exclusion criteria were (1) being too cognitively impaired to consent; (2) having a treating clinician determine that participation would have been contraindicated (eg, by paranoia potentially exacerbated by repeated contacts); or (3) being a prisoner or otherwise under judicial order, such that study participation could not be considered to be truly voluntary. Participants could enter the study through clinician referral or self-referral. All self-referrals were confirmed as appropriate to participate by an installation clinician prior to screening.

Procedures

Licensed master’s-level mental health clinicians, who were called continuity clinicians (CCs) to distinguish them from installation clinicians, were hired by the study and credentialed as behavioral health clinicians so they could provide backfill services in the clinic to compensate the installation for the time and effort hosting the study required. Continuity clinicians conducted informed consent, screening, and baseline assessments in a style that promoted care and support, so that those randomized to receive contacts would experience text messages from the CC as meaningful. When concluding the baseline assessment, the CCs used information collected during the interview to offer summary comments and resources that may be beneficial to the participant and carry out suicide risk management in coordination with installation clinicians if needed. Continuity clinicians did not provide further therapy or coordination to participants after randomization aside from Caring Contacts.

After the baseline assessment, eligible participants were randomized in equal proportions to Caring Contacts or standard care via a randomization generator built into the study database. Randomization was stratified on military installation and history of 0, 1, or 2 or more lifetime suicide attempts. Randomization assignment was revealed to the CC in a popup box in the study database that contained only the current participant’s assigned condition. Concealment was aided by the use of stratified blocks of 4 or 6 for randomization. Evaluators, blinded to treatment assignment, conducted a check of participants’ contact information at 6 months and a follow-up assessment by telephone at 12 months.

Treatment
Standard Care

As active-duty military service members, participants received behavioral health care as determined by their treating clinicians (ie, independently of the study). The modal standard care services were individual therapy and psychiatric medications, followed by group therapy and substance abuse treatment (eTable 1 in Supplement 2).

Caring Contacts via Text Message

In addition to standard care, participants in the Caring Contacts condition received caring texts at 1 day; 1 week; and 1, 2, 3, 4, 6, 8, 10, and 12 months and on their birthdays. Text messages were nondemanding, asking nothing of the recipient and only expressing care and concern. A link to a study website was included in the first text with resources including behavioral health and crisis services (Box includes messages). Texts were sent earlier in the day to allow maximum time to receive responses while the CC was on site and outpatient clinics were open. The study contracted with an online messaging platform that provided Health Insurance Portability and Accountability Act–compliant, secure text messages between health care clinician and patients’ cellular telephones. The online messaging platform allowed preprogramming of text messages on the planned schedule and routed participant replies to the CC’s study cellular telephone as well as others on clinical coverage. Text message replies from participants were monitored 24 hours a day. If participants were deployed without personal cellular telephone service or there was a problem texting their cellular telephones, the same message content was sent to their preferred email address instead.

Box Section Ref ID
Box.

Text Messages

Day 1
  • [NAME]- Good to meet you yesterday. Check out our website: http://www.MCproj.org.a Hope the resources are helpful. -[CLINICIAN NAME]

Week 1
  • Hey [NAME]- This is [CLINICIAN NAME] from the MC Project. Hope this week is going well for you.

Months 1, 2, 3, and 4
  • Hi there [NAME], hope you’re having a good day today. -[CLINICIAN NAME], MCProj.org

  • Hey [NAME], hope things are going well and you’re having a good week. -[CLINICIAN NAME], MCProj.org

  • Hi [NAME], hope life is treating you well. -[CLINICIAN NAME], MCProj.org

  • Hi [NAME], hope all’s well and you’re taking good care of yourself. -[CLINICIAN NAME], MCProj.org

Months 6, 8, 10, and 12
  • Hey [NAME]- Just wanted to say hello. Hope things are going OK. -[CLINICIAN NAME], MCProj.org

  • Hello again, [NAME]! Hope things are good. -[CLINICIAN NAME], MCProj.org

  • Hi [NAME], hope you’re having a good day today. -[CLINICIAN NAME], MCProj.org

  • [NAME]- Hope everything's going well. -[CLINICIAN NAME], MCProj.org

Birthday
  • Hey [NAME]! It's your birthday! Hope you have a great one and the next year brings you good things! -[CLINICIAN NAME], MCProj.org

a A link to a study website was included in the first text. The site offered resources, including behavioral health and crisis services; it was only active during the trial and was taken down after the intervention period concluded for all participants.

If a participant replied to a scheduled text with more specific content than positive or neutral comments, such as, “I’m fine, thanks,” or “Hope things are good for you too,” a response was incorporated into the next scheduled outgoing text message. If a response was requested (eg, resources, referrals), the CCs responded. If replies indicated distress, CCs’ general strategy was to express understanding and caring and confirm the participant’s access to behavioral health resources, personalizing these as appropriate on the basis of information gathered at baseline (eg, specifying the participant’s therapist’s name or the name of the clinic the participant had been attending). Replies indicating suicide risk were responded to immediately by telephone. Throughout the trial, less than 5% of responses indicated distress, and only 5 indicated suicidality. Almost all distressed responses were handled by a few text messages exchanged or a single telephone call, because such calls became warm handoffs to the Military/Veterans Crisis Line when immediate action was needed.

As the study proceeded, some participants reported in follow-up assessments that Caring Contacts felt mechanical, because their replies did not always receive a timely response. We concluded based on this feedback that a primarily unidirectional intervention (protocol A) by text message was problematic. In consultation with the study’s methodology team and with approval from the data and safety monitoring board and funder, it was decided halfway through data collection to revise the Caring Contacts protocol to improve responsiveness (protocol B). In protocol B, if a participant replied to a scheduled text with general comments such as, “I’m fine, thanks,” or “Hope things are good for you too,” the CCs responded using minimal text message etiquette, such as an emoticon smiley face or an appropriate response, such as, “Thanks!” or “You’re welcome.” All other responses from CCs (ie, to specific questions or distress) were made as described in protocol A. Thus, the only change in protocol B was the addition of contemporaneous responses by minimal, polite text message to participants’ general positive or neutral replies.

Treatment Fidelity

The online messaging platform allowed daily oversight of text messages sent and received and download of text message content with time stamps for analysis. An author, a licensed clinical social worker, provided training and ongoing supervision to the CCs by monitoring the scheduled text messages and responses through the online messaging platform to assure consistency with the Caring Contacts protocol.

Measures
Suicidal Ideation

The SSI-Current21,22 is an interviewer-rated scale administered in a semistructured interview format that measures suicidal ideation at its worst point in the past 2 weeks. Nineteen items are rated on a 3-point scale from 0 to 2. If both items 4 and 5 (active and passive suicidal ideation) are scored 0, the remaining items are not administered, and the total score is the sum of the first 3 items. Total scores range from 0 (no suicidality) to 38 (highest suicidality), and there is no established cut point. The SSI-Worst23 focused on worst-point ideation lifetime when administered at enrollment. At 12 months, the SSI-Worst captured worst-point ideation, if any, between baseline and follow-up. The SSI-Worst was added as an outcome after the first 35 cases completed follow-up, when the researchers learned it was becoming a standard in clinical trials using the SSI.24,25

Suicide Risk Incidents and Emergency Department Visits

Suicide risk incidents (ie, medical evacuation or inpatient admission) and emergency department visits to prevent suicide were determined using the Treatment History Interview. This assessment captured suicidal participants’ past-year treatment history in detail,26 and it was given at baseline and 12 months.

Suicide Attempt

Suicide attempts were measured by the Suicide Attempt Self-Injury Count (SASI-Count),27 a short version of the Suicide Attempt Self-Injury Interview.28 It assesses the method used, intent to die, highest level of medical treatment received, and greatest potential lethality of all self-directed violence in a specified time period. A lifetime version was administered at baseline. A recent version assessing only the past year was administered at both baseline and follow-up. The SASI-Count allows multiple methods of coding self-directed violence. In this study, suicide attempts were coded to match both the Columbia-Suicide Severity Rating Scale29 and the Centers for Disease Control and Prevention Self-Directed Violence Classification System as self-directed violence with any intent to die.30

Statistical Analysis

Power and sample-size estimates focused on the number of suicide risk incidents requiring medical evacuation or hospital admission (hypothesis 3), using effect sizes from Carter et al,13 on whose version of Caring Contacts this study was modeled. A simulation-based approach to power and sample size estimation was used (discussion in Landau et al31). Specifically, we used the counts of self-poisoning readmission data to simulate outcomes from a negative binomial distribution separately by treatment and sex as presented in Carter et al.13 One thousand new data sets were simulated for each combination of sample size (from 600 to 1200) and treatment rate ratios (0.50, as reported by Carter et al,13 and 0.60 for a more conservative estimate), with regressions fit to each simulated data set to estimate power. These analyses indicated that if the treatment effect size replicated that found by Carter et al13 (ie, a relative risk of 0.50), the study would be powered at 0.80, at a total sample size of 600 participants (300 per arm).

Study hypotheses were evaluated using intent-to-treat analyses, including all participants enrolled and randomized to each condition. Previous studies typically have covaried baseline values of outcomes11,18; thus, all analyses in this study included adjustments for baseline values. Although randomization was stratified by number of lifetime suicide attempts (0, 1, or 2 or more), these were found to be imbalanced between treatment conditions for participants with 2 or more previous attempts, such that Caring Contacts participants in this subgroup had nearly twice as many previous attempts (mean [SD], 9.10 [26.89] attempts) as counterparts receiving standard care alone (mean [SD], 5.14 [10.37] attempts). Therefore, lifetime rather than past-year suicide attempts (Table 1) were used as the baseline value for the suicide attempt analysis.

After data collection was complete, the statistical team reviewed the planned analyses compared with the final data distributions. Owing to low frequency of suicide risk incidents, emergency department visits, and suicide attempts between baseline and follow-up (Table 2), there were insufficient counts for the analyses as planned. Instead, logistic regression was conducted on the presence or absence of each event during the follow-up period. At follow-up, the measure of suicidal ideation (SSI-Current and SSI-Worst) had a positively skewed distribution and many zeroes, precluding the use of analyses based on a normal distribution. For these outcomes, we used a hurdle model,32 which effectively divides the suicidal ideation outcome into 2 outcomes, each modeled in its own regression equation. One outcome is a dichotomous variable representing any vs no suicidal ideation and includes the entire sample. The second outcome represents the degree of suicidal ideation when any is present. Thus, a hurdle model contains 2 submodels: (1) a logistic regression for zero vs nonzero values; and (2) a zero-truncated overdispersed negative binomial regression for the distribution of nonzero values. The hurdle model of the suicidal ideation outcome provided 2 sets of results corresponding to the effect of treatment on (1) likelihood of any suicidal ideation (ie, logit model) and (2) average suicidal ideation given any (ie, zero-truncated count model).

Missing data were assumed to be missing at random and addressed via multiple imputation.33 In addition to all predictor and outcome variables included in the quantitative models for this article, the number of inpatient psychiatry days to prevent suicide at baseline, number of deployments, and length of military service in months were included as auxiliary variables in the estimation of imputed data sets (k = 1000). All analyses used the P < .05 threshold for statistical significance, imputed data sets were calculated using SPSS version 19 (IBM), and all analyses were completed in the R statistical package, version 3.4.4 (R Foundation for Statistical Computing).34

Results
Participant Characteristics

The Figure displays the Consolidated Standards of Reporting Trials diagram. Table 1 summarizes demographic and clinical characteristics. Table 2 provides descriptive statistics on suicide outcomes at baseline and 12 months. Participants in this study were largely young (mean [SD] age, 25.2 [6.1] years), white (399 of 657 [60.7%]), and male (539 of 657 [82.0%]) and fairly evenly divided between US Army Soldiers (355 of 657 [54.0%]) and US Marines (302 of 657 [46.0%]). Fewer than half (277 of 650 [42.6%]) had ever deployed to a combat zone (with the status of 7 participants unknown).35 Two-thirds (438 of 657 [66.7%]) had had a suicide risk incident in the year before enrollment, with approximately half (348 of 657 [53.0%]) referred from an inpatient unit. As shown in Table 1, almost half (291 of 657 [44.3%]) had made a lifetime suicide attempt, with firearms (18 [6.2%]) or hanging (64 [22.0%]) the attempts methods with greatest potential lethality.

Primary Outcomes

Outcomes are presented in Table 3. A hurdle model demonstrated no significant effect of Caring Contacts on the presence or severity of current suicidal ideation (ie, during the 2 weeks prior to follow-up assessment). Logistic regression results indicated that Caring Contacts did not significantly decrease the odds of a suicide risk incident during the follow-up period.

Secondary Outcomes

A hurdle model of the SSI-Worst during the follow-up period demonstrated that Caring Contacts led to a 44% decrease in the odds of reporting any suicidal ideation during the follow-up period (172 of 216 participants [79.6%] in the group receiving Caring Contacts vs 179 of 204 participants [87.7%] in the standard-care group; odds ratio, 0.56 [95% CI, 0.33-0.95]; P = .03), but there was no significant effect of Caring Contacts on the severity of worst-point suicidal ideation among those with any suicidal ideation. Caring Contacts led to a 48% decrease in the odds of reporting 1 or more suicide attempts since baseline (21 of 233 [9.0%] in the group receiving Caring Contacts vs 34 of 228 [14.9%] in the standard-care group; odds ratio, 0.52 [95% CI, 0.29-0.92]; P = .03). eTable 2 in Supplement 2 presents details on suicide attempts in the year before and after enrollment. Caring Contacts did not significantly decrease the odds of an emergency department visit for suicidality during the follow-up period. All findings were consistent across calculations using both multiple imputation and complete case analysis (eTable 3 in Supplement 2).

Discussion

We proposed and powered the study for 2 outcomes: current suicidal ideation (at 12 months) and suicide risk incidents for Marines and Soldiers who were thinking of suicide. Contrary to our hypotheses, neither of these outcomes were improved by Caring Contacts. Caring Contacts also did not decrease the odds of emergency department presentation for suicidality. However, results indicated that Caring Contacts was effective in preventing suicidal ideation and suicide attempts throughout follow-up. Treatment effects were modest in regard to the absolute rates of suicidal ideation (80% vs 88%) and attempts (9% vs 15%), yet clinically meaningful, given the brevity and potential reach of this intervention. These results using text messaging replicate the effectiveness of Caring Contacts in previous studies using postal mail and telephone calls to reduce suicidal ideation and attempts.8-10,12,13 They extend these results in demonstrating effectiveness of Caring Contacts when provided by text message, with active-duty military, with a lower level of baseline suicidality, and with men, although the study was not powered to examine these subgroups directly.

Recent effectiveness studies of Caring Contacts have added appointment reminders or self-help strategies but have shown null or negative effects.16,17 By contrast, Caring Contacts in this study were extremely brief and focused solely on expressing care, interest, and support. It may be that the crucial ingredient to Caring Contacts is regular and long-term contact with another person who expresses caring and concern without demands or expectations, as was originally proposed in 1976.8

Limitations

This study had several limitations. First, based on feedback on surveys of initial participants in the Caring Contacts condition, we added polite responses to the Caring Contacts protocol halfway through the trial. Outcomes appear stronger for protocol B, although this was not a significant difference. Second, only 70% of participants completed follow-up assessments. While this was likely because of the minimal contact the research team had with participants until the study intervention was complete (to minimize any influence of assessment contact on study outcomes), it is nonetheless a concern. Further research will reexamine study outcomes through a combination of DoD and Veterans Health Administration electronic health records. Third, even with a final sample of 657 individuals, the low rates of suicidal behavior limited the ability to evaluate counts of these variables. Fourth, because of the pragmatic nature of the trial, diagnostic interviews were not conducted. Fifth, the participants in this study had disclosed suicidality to a military clinician. Thus, the effects of Caring Contacts may not generalize to service members who do not disclose suicidality. Finally, given the modest absolute differences in the outcomes observed and the relatively low base rates of suicidal behavior, it is possible that small differences in the observed values could change the significance of our findings, suggesting the need for replication of these results.

Conclusions

A crucial next step for research is to empirically evaluate and understand the mechanism by which Caring Contacts have their effect. An examination of social connectedness as a possible mechanism is underway for this study and a new trial evaluating a culturally tailored version of Caring Contacts with American Indian and Alaska Native communities. Another possible mechanism is a greater engagement in behavioral health care because of a positive response to Caring Contacts, which does not appear to be the case in this study, in which the standard care services accessed were almost identical between conditions (eTable 1 in the Supplement). Dissemination and implementation research is also needed to study the logistics of implementing Caring Contacts in military and nonmilitary settings. Important questions remain in regard to who can or should be the authors of the messages in different settings, as well as how the intervention could be funded. Also relevant is addressing liability issues in having minimal contact with suicidal individuals, particularly for civilian community clinicians who are not part of an overarching system such as the DoD, Veterans Health Administration, or large managed-care organizations.

This trial provides inconsistent results on the effectiveness of caring text messages between primary and secondary outcomes. Among active-duty suicidal Soldiers and Marines, Caring Contacts (provided by 11 text messages sent over 1 year) reduced the odds of any suicidal ideation and suicide attempt compared with standard care alone. These outcomes support the effectiveness of Caring Contacts and provide the military with a practical and scalable tool to reduce suicide risk.

Back to top
Article Information

Accepted for Publication: November 26, 2018.

Corresponding Author: Amanda H. Kerbrat, MSW, University of Washington, 325 Ninth Ave, PO Box 359911, Seattle, WA 98104 (ahm2@uw.edu)

Published Online: February 13, 2019. doi:10.1001/jamapsychiatry.2018.4530

Author Contributions: Dr Comtois and Ms Kerbrat had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Comtois, Kerbrat, Ries.

Acquisition, analysis, or interpretation of data: Comtois, Kerbrat, DeCou, Atkins, Majeres, Baker.

Drafting of the manuscript: Comtois, Kerbrat, DeCou.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Kerbrat, DeCou, Atkins.

Obtained funding: Comtois, Ries.

Administrative, technical, or material support: Comtois, Kerbrat, Majeres.

Supervision: Comtois, Kerbrat.

Conflict of Interest Disclosures: Drs Comtois, Atkins, and Ries and Ms. Kerbrat reported receiving research funding from the Military Suicide Research Consortium during the conduct of the study. Dr DeCou received funding from the National Institute of Child Health and Development of the National Institutes of Health (grant T32HD057822) during the conduct of the study. Dr Atkins is also a cofounder with equity stake in a technology company, Lyssn.io, focused on tools to support training, supervision, and quality assurance of psychotherapy and counseling. No other disclosures were reported.

Funding/Support: This study was supported by the Military Suicide Research Consortium, an effort supported by the Office of the Assistant Secretary of Defense for Health Affairs (award W81XWH-10-2-0181). The preparation of the manuscript was also supported in part by the National Institute of Child Health and Development of the National Institutes of Health (grant T32HD057822).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Disclaimer: The views expressed herein are those of the authors and do not reflect the official policy of the Department of the Army, Department of Defense, the US government, or the Military Suicide Research Consortium.

Data Sharing Statement: See Supplement 3.

Additional Contributions: We thank the study clinicians for their steadfast recruitment efforts and providing the intervention to service members with such care: Jessica L. Soule, LCSW, Crystal J. Holman, LCSW, Megan N. Muzychka, LCSW, and Alvin D. Wilson, LCSW, the Geneva Foundation. We are grateful to the outcome assessors on this study, particularly Amber S. Hassan, BA, and Bryan J. Stiles, BA, Department of Psychiatry and Behavioral Sciences, School of Medicine, University of Washington, Seattle. We also thank Michael McDonell, PhD, Department of Psychiatry and Behavioral Sciences, School of Medicine, University of Washington, Seattle, for his role in securing funding and as coinvestigator in the early stages of the project. We thank Colonel Jay E. Earles, PsyD, Womack Army Medical Center, now with Midland Valley Behavioral Health and Consulting, LLC, and Major Justine J. Majeres, PsyD, Womack Army Medical Center, for serving as site principal investigators and Major Stacey Krauss, PsyD, Womack Army Medical Center, for serving as site associate investigator at our US Army site. We appreciate the contribution of our study liaisons in the Marine and Family Programs Division, Headquarters Marine Corps. Finally, we thank Lieutenant Justin C. Baker, PhD, Naval Medical Center Camp Lejeune, as site principal investigator on behalf of the US Navy at our primary Marine Corps site. Mss Soule, Holman, Muzychka, and Hassan; Mssrs Wilson and Stiles; and Dr McDonell were compensated with study funding. Drs Earles, Majeres, Krauss, and Baker and the Marine & Family Programs Division were not compensated for their contributions.

References
1.
Kessler  RC, Warner  CH, Ivany  C,  et al; Army STARRS Collaborators.  Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study To Assess Risk and Resilience in Servicemembers (Army STARRS).  JAMA Psychiatry. 2015;72(1):49-57. doi:10.1001/jamapsychiatry.2014.1754PubMedGoogle ScholarCrossref
2.
Hoge  CW, Castro  CA, Messer  SC, McGurk  D, Cotting  DI, Koffman  RL.  Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care.  N Engl J Med. 2004;351(1):13-22. doi:10.1056/NEJMoa040603PubMedGoogle ScholarCrossref
3.
Hoge  CW, Auchterlonie  JL, Milliken  CS.  Mental health problems, use of mental health services, and attrition from military service after returning from deployment to Iraq or Afghanistan.  JAMA. 2006;295(9):1023-1032. doi:10.1001/jama.295.9.1023PubMedGoogle ScholarCrossref
4.
Kim  PY, Thomas  JL, Wilk  JE, Castro  CA, Hoge  CW.  Stigma, barriers to care, and use of mental health services among active duty and National Guard soldiers after combat.  Psychiatr Serv. 2010;61(6):582-588. doi:10.1176/ps.2010.61.6.582PubMedGoogle ScholarCrossref
5.
VanSickle  M, Werbel  A, Perera  K, Pak  K, DeYoung  K, Ghahramanlou-Holloway  M.  Perceived barriers to seeking mental health care among United States Marine Corps noncommissioned officers serving as gatekeepers for suicide prevention.  Psychol Assess. 2016;28(8):1020-1025. doi:10.1037/pas0000212PubMedGoogle ScholarCrossref
6.
Hoge  CW, Lesikar  SE, Guevara  R,  et al.  Mental disorders among U.S. military personnel in the 1990s: association with high levels of health care utilization and early military attrition.  Am J Psychiatry. 2002;159(9):1576-1583. doi:10.1176/appi.ajp.159.9.1576PubMedGoogle ScholarCrossref
7.
Luxton  DD, June  JD, Comtois  KA.  Can postdischarge follow-up contacts prevent suicide and suicidal behavior? a review of the evidence.  Crisis. 2013;34(1):32-41. doi:10.1027/0227-5910/a000158PubMedGoogle ScholarCrossref
8.
Motto  JA.  Suicide prevention for high-risk persons who refuse treatment.  Suicide Life Threat Behav. 1976;6(4):223-230.PubMedGoogle Scholar
9.
Motto  JA, Bostrom  AG.  A randomized controlled trial of postcrisis suicide prevention.  Psychiatr Serv. 2001;52(6):828-833. doi:10.1176/appi.ps.52.6.828PubMedGoogle ScholarCrossref
10.
Fleischmann  A, Bertolote  JM, Wasserman  D,  et al.  Effectiveness of brief intervention and contact for suicide attempters: a randomized controlled trial in five countries.  Bull World Health Organ. 2008;86(9):703-709. doi:10.2471/BLT.07.046995PubMedGoogle ScholarCrossref
11.
Hassanian-Moghaddam  H, Sarjami  S, Kolahi  A-A, Carter  GL.  Postcards in Persia: randomised controlled trial to reduce suicidal behaviours 12 months after hospital-treated self-poisoning.  Br J Psychiatry. 2011;198(4):309-316. doi:10.1192/bjp.bp.109.067199PubMedGoogle ScholarCrossref
12.
Hassanian-Moghaddam  H, Sarjami  S, Kolahi  A-A, Lewin  T, Carter  G.  Postcards in Persia: a twelve to twenty-four month follow-up of a randomized controlled trial for hospital-treated deliberate self-poisoning.  Arch Suicide Res. 2017;21(1):138-154. doi:10.1080/13811118.2015.1004473PubMedGoogle ScholarCrossref
13.
Carter  GL, Clover  K, Whyte  IM, Dawson  AH, D’Este  C.  Postcards from the edge project: randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated deliberate self poisoning.  BMJ. 2005;331(7520):805-807. doi:10.1136/bmj.38579.455266.E0PubMedGoogle ScholarCrossref
14.
Carter  GL, Clover  K, Whyte  IM, Dawson  AH, D’Este  C.  Postcards from the edge: 24-month outcomes of a randomised controlled trial for hospital-treated self-poisoning.  Br J Psychiatry. 2007;191:548-553. doi:10.1192/bjp.bp.107.038406PubMedGoogle ScholarCrossref
15.
Carter  GL, Clover  K, Whyte  IM, Dawson  AH, D’Este  C.  Postcards from the edge: 5-year outcomes of a randomised controlled trial for hospital-treated self-poisoning.  Br J Psychiatry. 2013;202(5):372-380. doi:10.1192/bjp.bp.112.112664PubMedGoogle ScholarCrossref
16.
Robinson  J, Yuen  HP, Gook  S,  et al.  Can receipt of a regular postcard reduce suicide-related behaviour in young help seekers? a randomized controlled trial.  Early Interv Psychiatry. 2012;6(2):145-152. doi:10.1111/j.1751-7893.2011.00334.xPubMedGoogle ScholarCrossref
17.
Kapur  N, Gunnell  D, Hawton  K,  et al.  Messages from Manchester: pilot randomised controlled trial following self-harm.  Br J Psychiatry. 2013;203(1):73-74. doi:10.1192/bjp.bp.113.126425PubMedGoogle ScholarCrossref
18.
Beautrais  AL, Gibb  SJ, Faulkner  A, Fergusson  DM, Mulder  RT.  Postcard intervention for repeat self-harm: randomised controlled trial.  Br J Psychiatry. 2010;197(1):55-60. doi:10.1192/bjp.bp.109.075754PubMedGoogle ScholarCrossref
19.
Berrouiguet  S, Gravey  M, Le Galudec  M, Alavi  Z, Walter  M.  Post-acute crisis text messaging outreach for suicide prevention: a pilot study.  Psychiatry Res. 2014;217(3):154-157. doi:10.1016/j.psychres.2014.02.034PubMedGoogle ScholarCrossref
20.
Chen  H, Mishara  BL, Liu  XX.  A pilot study of mobile telephone message interventions with suicide attempters in China.  Crisis. 2010;31(2):109-112. doi:10.1027/0227-5910/a000017PubMedGoogle ScholarCrossref
21.
Beck  AT, Kovacs  M, Weissman  A.  Assessment of suicidal intention: the Scale for Suicide Ideation.  J Consult Clin Psychol. 1979;47(2):343-352. doi:10.1037/0022-006X.47.2.343PubMedGoogle ScholarCrossref
22.
Beck  AT, Brown  GK, Steer  RA.  Psychometric characteristics of the Scale for Suicide Ideation with psychiatric outpatients.  Behav Res Ther. 1997;35(11):1039-1046. doi:10.1016/S0005-7967(97)00073-9PubMedGoogle ScholarCrossref
23.
Beck  AT, Brown  GK, Steer  RA, Dahlsgaard  KK, Grisham  JR.  Suicide ideation at its worst point: a predictor of eventual suicide in psychiatric outpatients.  Suicide Life Threat Behav. 1999;29(1):1-9. doi:10.1111/j.1943-278X.1999.tb00758.xPubMedGoogle Scholar
24.
Rudd  MD, Bryan  CJ, Wertenberger  EG,  et al.  Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up.  Am J Psychiatry. 2015;172(5):441-449. doi:10.1176/appi.ajp.2014.14070843PubMedGoogle ScholarCrossref
25.
LaCroix  JM, Perera  KU, Neely  LL, Grammer  G, Weaver  J, Ghahramanlou-Holloway  M.  Pilot trial of post-admission cognitive therapy: inpatient program for suicide prevention.  Psychol Serv. 2018;15(3):279-288. doi:10.1037/ser0000224PubMedGoogle ScholarCrossref
26.
Linehan  MM, Heard  HL. University of Washington Behavioral Research & Therapy Clinics Treatment History Interview (THI-4). http://depts.washington.edu/uwbrtc/wp-content/uploads/THI-4.pdf. Published 1996. Accessed January 10, 2019.
27.
Linehan  M, Comtois  KA. Suicide Attempt Self-Injury Count (SASI-Count). http://depts.washington.edu/uwbrtc/wp-content/uploads/L-SASI-Count.pdf. Published 1996. Accessed November 21, 2018.
28.
Linehan  MM, Comtois  KA, Brown  MZ, Heard  HL, Wagner  A.  Suicide Attempt Self-Injury Interview (SASII): development, reliability, and validity of a scale to assess suicide attempts and intentional self-injury.  Psychol Assess. 2006;18(3):303-312. doi:10.1037/1040-3590.18.3.303PubMedGoogle ScholarCrossref
29.
Posner  K, Brown  GK, Stanley  B,  et al.  The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults.  Am J Psychiatry. 2011;168(12):1266-1277. doi:10.1176/appi.ajp.2011.10111704PubMedGoogle ScholarCrossref
30.
Crosby  AE, Ortega  L, Melanson  C. Centers for Disease Control and Prevention. Self-directed violence surveillance: uniform definitions and recommended data elements. https://www.cdc.gov/violenceprevention/pdf/Self-Directed-Violence-a.pdf. Published 2011. Accessed January 7, 2019.
31.
Landau  S, Stahl  D.  Sample size and power calculations for medical studies by simulation when closed form expressions are not available.  Stat Methods Med Res. 2013;22(3):324-345. doi:10.1177/0962280212439578PubMedGoogle ScholarCrossref
32.
Atkins  DC, Baldwin  SA, Zheng  C, Gallop  RJ, Neighbors  C.  A tutorial on count regression and zero-altered count models for longitudinal substance use data.  Psychol Addict Behav. 2013;27(1):166-177. doi:10.1037/a0029508PubMedGoogle ScholarCrossref
33.
Graham  JW.  Missing data analysis: making it work in the real world.  Annu Rev Psychol. 2009;60:549-576. doi:10.1146/annurev.psych.58.110405.085530PubMedGoogle ScholarCrossref
34.
R Core Team; R Foundation for Statistical Computing. R: a language and environment for statistical computing. https://www.R-project.org/. Published 2017. Accessed January 7, 2019.
35.
Internal Revenue Service. Combat zones approved for tax benefits. https://www.irs.gov/newsroom/combat-zones. Published October 29, 2018. Accessed January 7, 2019.
×