Prospective Development and Validation of the Computerized Adaptive Screen for Suicidal Youth

Key Points

Question  Can use of a computerized adaptive screen reliably predict a youth suicide attempt within 3 months?

Findings  This prognostic study with a derivation cohort of 2075 adolescents and an independent prospective validation cohort of 2754 adolescents developed and validated the Computerized Adaptive Screen for Suicidal Youth (CASSY) to predict a suicide attempt within 3 months. In the independent validation cohort, the area under the curve for the CASSY was 0.87 (95% CI, 0.85-0.89) for prediction of a suicide attempt within 3 months.

Meaning  In this study, the CASSY had a strong, validated classification accuracy, and it may improve recognition of youth suicide risk.

Importance  The rate of suicide among adolescents is rising in the US, yet many adolescents at risk are unidentified and receive no mental health services.

Objective  To develop and independently validate a novel computerized adaptive screen for suicidal youth (CASSY) for use as a universal screen for suicide risk in medical emergency departments (EDs).

Design, Setting, and Participants  Study 1 of this prognostic study prospectively enrolled adolescent patients at 13 geographically diverse US EDs in the Pediatric Emergency Care Applied Research Network. They completed a baseline suicide risk survey and participated in 3-month telephone follow-ups. Using 3 fixed Ask Suicide-Screening Questions items as anchors and additional items that varied in number and content across individuals, we derived algorithms for the CASSY. In study 2, data were collected from patients at 14 Pediatric Emergency Care Applied Research Network EDs and 1 Indian Health Service hospital. Algorithms were independently validated in a prospective cohort of adolescent patients who also participated in 3-month telephone follow-ups. Adolescents aged 12 to 17 years were consecutively approached during randomly assigned shifts.

Exposures  Presentation at an ED.

Main Outcome and Measure  A suicide attempt between ED visit and 3-month follow-up, measured via patient and/or parent report.

Results  The study 1 CASSY derivation sample included 2075 adolescents (1307 female adolescents [63.0%]; mean [SD] age, 15.1 [1.61] years) with 3-month follow-ups (72.9% retention [2075 adolescents]). The study 2 validation sample included 2754 adolescents (1711 female adolescents [62.1%]; mean [SD] age, 15.0 [1.65] years), with 3-month follow-ups (69.5% retention [2754 adolescents]). The CASSY algorithms had excellent predictive accuracy for suicide attempt (area under the curve, 0.89 [95% CI, 0.85-0.91]) in study 1. The mean number of adaptively administered items was 11 (range, 5-21). At a specificity of 80%, the CASSY had a sensitivity of 83%. It also demonstrated excellent accuracy in the study 2 validation sample (area under the curve, 0.87 [95% CI, 0.85-0.89]). In this study, the CASSY had a sensitivity of 82.4% for prediction of a suicide attempt at the 80% specificity cutoff established in study 1.

Conclusions and Relevance  In this study, the adaptive and personalized CASSY demonstrated excellent suicide attempt risk recognition, which has the potential to facilitate linkage to services.

In 2018, the US reported the highest annual number of adolescent suicide deaths to date, with 1750 suicides among adolescents 12 to 17 years of age.¹ In fact, the suicide rate among adolescents in the US in this age range has increased by 62% since 2000.¹ Recent National Violent Death Report System data indicate that only 41.2% of adolescents who die by suicide have ever been treated for a mental health problem.²

Improved suicide risk detection through effective screening has the potential to facilitate treatment linkage, reduce morbidity, and prevent mortality. Emergency departments (EDs) are particularly well suited for suicide risk screening, since approximately 19% of US adolescents visit the ED in a 1-year period.³ Further, ED visits for youth suicide risk have recently doubled,⁴ and EDs are a common point of access for mental health services.^5,6 Many chief complaints place adolescents at increased risk for suicidal behavior (eg, trauma, alcohol abuse)⁷; boys, who account for most suicides,¹ are well represented⁸; and parents and adolescents have favorable views of suicide risk screening in the ED.⁹

A substantial challenge to universal suicide risk screening in EDs is the accurate identification of youth at risk in a setting where efficiency is highly valued and mental health concerns represent only a small minority of chief complaints.¹⁰ Existing tools such as the Ask Suicide-Screening Questions (ASQ)^11,12 and the Multifactor Youth Suicide Risk Screen¹³ have shown effectiveness in identifying previously unrecognized suicide risk. Moreover, the predictive validity of the ASQ as a universal screen was demonstrated in a large retrospective medical record review study¹⁴; it demonstrated sensitivity and specificity of 60.0% and 92.7%, respectively, for prediction of an ED visit for a suicide-associated concern. Despite these overall favorable results, the moderate sensitivity indicates that many individuals at risk were not identified. Given the morbidity and mortality associated with suicide risk, research aimed at improving tools for universal screening in the ED is warranted.¹⁵

A computerized adaptive testing (CAT) strategy offers the possibility of developing an improved youth suicide risk screening tool. The CAT is grounded in a multidimensional extension of item response theory, in which an individual’s initial item responses are used to determine a provisional estimate of their standing on the measured trait (eg, risk of suicide attempt [SA]).¹⁶ The number and content of items presented to individuals varies to achieve the same precision of measurement.¹⁷ This strategy was used by Gibbons et al¹⁸ to develop a dimensional measure of depression severity that outperformed standard screening instruments, and the validity and usefulness of a CAT strategy has been demonstrated for multiple mental health disorders among adults.^19,20 Gibbons et al²¹ recently developed a Kiddie-CAT that demonstrates excellent accuracy for assessing presence and severity of child and adolescent depression and anxiety, among other conditions. The eAppendix in Supplement 1 provides information about CAT-based measurement accomplishments and their dissemination.

To develop and validate a computerized adaptive screen for suicidal youth (CASSY), we conducted 2 independent studies in collaboration with the US Pediatric Emergency Care Applied Research Network (PECARN). In study 1, we used CAT to develop a universal screen that targets items to adolescents’ personal risk profiles and provides a continuous risk score (rather than binary classification) for the likelihood of an SA within 3 months. In study 2, we prospectively validated the CASSY in an independent sample. We chose SA as the outcome because it was the outcome of interest specified by the National Institute of Mental Health, and a 3-month time frame because this is a period of high risk.²²

This study uses data from PECARN EDs that participated in the ED Screen for Teens at Risk for Suicide (ED-STARS) study 1²³ and study 2 cohorts of 6536 and 4031 adolescents, respectively. In study 1, we recruited from 13 PECARN EDs between June 2015 and July 2016; in study 2, we recruited from 14 PECARN EDs and 1 Indian Health Services ED between July 2017 and September 2018. The EDs were located in the US, in the West (study 1, n = 2 [15%]; study 2, n = 4 [27%]), Southwest/Central (study 1, n = 2 [15%]; study 2, n = 2 [13%]), Midwest (study 1, n = 4 [31%]; study 2, n = 4 [27%]), and the Mid-Atlantic/New England (study 1, n = 5 [38%]; study 2, n = 5 [33%]) regions.

Adolescents completed baseline assessments in the ED on a computer tablet. Interviewers, blinded to baseline data, conducted 3-month computer-assisted telephone follow-up interviews. Adolescents were remunerated with online certificates. We obtained institutional review board approval from all sites, parent/guardian written informed consent, and adolescent assent. We followed the Standards for Reporting of Diagnostic Accuracy Studies 2015 guidelines (reported in eTable 1 in Supplement 1).²⁴

Adolescents, ages 12 to 17 years, were recruited during screening shifts randomly selected for each site from times when research staff were available (primarily afternoons and evenings). In study 2 only, patients were also recruited from an Indian Health Services ED. Exclusion criteria were being a ward of the state, non–English speaking, and medically unstable or severely cognitively impaired (as determined by the clinician). Patients were not eligible to participate in both studies.

A subset of 2845 adolescents was randomly assigned to follow-ups. To establish this subset, we first stratified adolescents into low, moderate, and high risk groups, based on their baseline endorsement of suicide risk factors.⁷ We followed more adolescents at moderate or high risk (Figure 1), randomizing risk groups to follow-up in proportions needed to obtain preestablished numbers, to balance obtaining a sample with a full range of suicide risk with the need for a statistically sufficient number of attempt outcomes.

The enrolled sample was enriched for suicide risk by requiring each site to maintain at least a 2:3 enrollment ratio between adolescents with psychiatric vs other chief complaints. All patients were assigned to follow-up.

Adolescents completed a self-report survey (92 primary questions and up to 27 additional questions) at baseline, which included the ASQ^11,12 and Columbia–Suicide Severity Rating Scale (C-SSRS),^25-27 which assess suicidal ideation; suicidal rumination; SA history; nonsuicidal self-injury; depression; hopelessness; homicidal ideation; anxiety; agitation; sleep disturbance; functional impairment; alcohol and drug misuse; impulsive aggression; physical fighting; family, school, and social connectedness; being bullied by peers; physical and sexual abuse; negative life events; and sexual or gender minority identity. We used brief or adapted scales to minimize respondent burden (eMethods in Supplement 1).

Adolescents completed the CASSY at baseline. This was followed by a subset of 37 to 59 items from the study 1 baseline survey to ensure measurement of key demographic and clinical variables to describe the sample and differences in retention by these characteristics (eMethods in Supplement 1).

The main outcome was an SA between baseline and 3-month follow-up. This was defined by (1) an adolescent or parent report of adolescent ED visit or hospitalization with an SA, and/or (2) adolescent endorsement of the statement “In the past 3 months, have you made a suicide attempt?” or “In the past 3 months, have you tried to harm yourself because you were at least partly trying to end your life?” from an adapted C-SSRS.²⁶

To develop the CASSY, we applied a multidimensional item response theory model^28,29 to the 92-item bank using the subdomains of suicidal ideation and behavior, psychopathology, posttraumatic stress disorder, social adjustment, sleep, anger/aggression, and substance use, in addition to the overall primary dimension. The bifactor model allows each item to load on the primary dimension, which includes items from the suicidal ideation and behavior subdomain and highly correlated subdomains, and the single subdomain from which it was drawn. Items with loadings less than 0.30 on the primary dimension were deleted, leaving 72 items. In developing algorithms, we used 3 fixed ASQ items (morbid ideation, suicidal ideation, and history of suicide attempt) as anchors to ensure measurement of commonly assessed constructs, and we allowed the content and number of other items to vary across individuals. The CASSY score is expressed as a continuous score (0%-100%). Additional details are in the eMethods in Supplement 1.

We then used a logistic regression model (primary CASSY dimension score and suicide subdomain score as independent variables) to predict suicide attempt during the next 3 months. We report sensitivity for 2 fixed points on the receiver operating characteristic curve (specificity of 0.80% and 0.90%). Area under the curve (AUC), an overall index of predictive accuracy, was estimated based on 3-fold cross-validation (estimation in two-thirds of the sample, with validation and classification in remainder). This process was repeated 3 times; sensitivity, specificity, and AUC are reported based on participants whose data were not used to estimate the logistic regression. These analyses were performed using the POLYBIF program version 1.0 (University of Illinois at Chicago Center for Health Statistics; https://www.healthstats.org/) from December 2016 to July 2017.

We calculated AUC for prediction of an SA in addition to sensitivity and specificity at the fixed values of the CASSY corresponding to specificities of 0.80% and 0.90% in study 1. We evaluated study 2 data at thresholds derived in study 1 to assess reproducibility of classification accuracy. We conducted sensitivity analyses to assess changes attributable to missing data (eMethods in Supplement 1). These analyses were performed using Stata version 16 (StataCorp) from May 2020 to September 2020. The significance threshold was P < .05, 2-tailed.

Among the 10 544 eligible patients approached for participation, 6536 were enrolled (62.0%), 6344 (97.1% of those enrolled) completed the baseline survey (≥80% of questions), and 2845 were randomized to follow-up. Adolescents who did not complete the survey or whose risk group was unknown because of missing data were ineligible for follow-up. Follow-up data were obtained for 2075 adolescents (72.9% retention) from adolescents and parents (n = 1774 [85.5%]), adolescents only (n = 183 [8.8%]), or parents only (n = 118 [5.7%]). Figure 1 reports the percentages of adolescents in each baseline suicide risk cell who were randomized to follow-up and retained. Female adolescents (1307 of 1841 [71.0%] vs male adolescents: 768 of 1003 [76.6%]; P = .001), Latinx adolescents (455 of 665 [68.4%]; vs adolescents who were not Latinx: 1381 of 1814 [76.1%] and those of unknown ethnicity: 126 of 191 [66.0%]; P < .001), and those with lower parental education levels (eg, at least 1 parent having a college degree or professional training: 993 of 1274 adolescents [77.9%] vs both parents having less education: 834 of 1200 adolescents [69.5%]; P < .001) were less likely to be retained (eTable 2 in Supplement 1). Of these, only female sex was associated with SA (male adolescents: 19 of 768 [2.5%]; female adolescents: 83 of 1307 [6.4%]; P < .001). Associations between demographics and attempt outcomes were reported previously.³⁰

The 2075 participants in this study (1307 female adolescents [63%]) had a mean (SD) age of 15.1 (1.6) years. Table 1 presents demographic and suicide risk characteristics of participants in baseline and follow-up samples. Retention data are in the eMethods in Supplement 1.

At baseline, 832 adolescents (40.1%) had a lifetime C-SSRS suicidal ideation severity score of 3 or more, indicating suicidal thoughts with a method, plan, and/or intent; 1071 (51.7%) reported suicidal ideation (score ≥2). The number of lifetime SAs was distributed as follows: 0 (1451 [69.9%]), 1 (163 [7.9%]), multiple (430 [20.7%]), and unknown (31 [1.5%]).

A total of 102 adolescents (4.9%) made at least 1 SA between baseline and follow-up. Associations between baseline suicide risk factors and SA outcomes were reported previously.³⁰

Simulated adaptive testing from the complete response patterns revealed that a mean of 11 items (range, 5-21), including 3 ASQ items and a varying number of adaptively administered items, provided a correlation (r) of 0.94 with the 72-item total bank score. The median time required for administration of 11 items was 1 minute 24 seconds (interquartile range, 59 seconds to 2 minutes 4 seconds). With the exception of the 3 ASQ items included for all youth, the frequency with which other items were included varied, in keeping with their contribution to measuring each youth’s risk for SA (Table 2).

The predictor model with the highest balance between sensitivity and specificity yielded a 3-fold cross-validated sensitivity of 83% at the fixed specificity of 80%. At a specificity of 90%, sensitivity was 61% (cross-validated). The AUC for the entire receiver operating characteristic curve (Figure 2) was 0.89 (95% CI, 0.85-0.91). The addition of demographic variables (age, sex, and race) did not improve predictive accuracy. In selecting an optimal set of items for each adolescent, the CASSY used a mean (SD) of 15.3% (5.6%) of the 72 items available in the bank for any given individual. The mean (SD) of the CASSY score was 0.19 (0.08) and 0.04 (0.14), respectively, for those who did vs did not make an SA.

Among the 6513 eligible adolescents approached for participation, 4050 were enrolled (62.2%). Adolescents with complete baseline evaluations, (n = 3965 [97.9%]) were eligible for follow-up. Three-month follow-ups were obtained for 2754 adolescents (69.5% retention) from adolescents and parents (n = 2443 [88.7%]), adolescents only (n = 120 [4.4%]), or parents only (n = 191 [6.9%]). Black adolescents (471 of 769 [61.2%] vs, eg, White adolescents 1621 of 2261 [71.7%] P < .001) and adolescents with unknown ethnicity (220 of 364 [60.4%]; Latinx youths: 685 of 993 [69.0%]; non-Latinx youths: 1849 of 2608 [70.9%]; P < .001), lower parental education level (eg, at least 1 parent having a college degree or professional training: 1465 of 1969 adolescents [74.4%] vs both parents having less education: 985 of 1515 adolescents [65.0%]; P < .001), psychiatric complaints (1112 of 1657 adolescents [67.1%] vs those with nonpsychiatric complaints: 1642 of 2304 adolescents [71.3%]; P = .005), and families who received public assistance (1106 of 1639 [67.5%]; vs those who did not, 1595 of 2250 [70.9%]; P = .02) were less likely to be retained. Retention data are in eTable 3 in Supplement 1.

The 2754 study participants (62% female) had a mean (SD) age of 15.0 (1.65) years. Table 3 presents demographic and suicide risk characteristics of participants in baseline and follow-up samples.

The number of lifetime SAs at baseline was distributed as follows: 0 (1970 [71.5%]), 1 (249 [9.0%]), multiple (501 [18.2%]), and unknown (34 [1.2%]). A total of 165 adolescents (6.0%) made at least 1 SA between baseline and follow-up.

The AUC for application of CASSY algorithms, developed in study 1 and applied to the study 2 sample, was 0.87 (95% CI, 0.85-0.89). At the 80% specificity cutoff established in study 1, CASSY’s sensitivity for the prediction of an SA within 3 months in study 2 was 82.4%, and specificity was 72.5%. Figure 2 presents the receiver operating characteristic curves for the CASSY in study 2. The mean (SD) of the CASSY score was 0.18 (0.11) and 0.05 (0.07), respectively, for those who did vs did not make an SA.

The ED-STARS study is the first initiative to develop and independently validate a computerized adaptive youth suicide risk screen for risk of suicide attempt. The CASSY, which includes 3 ASQ items and a wide range of additional, adaptively administered items that vary in number and content across adolescents, is a dimensional suicide risk severity indicator. In this collaborative initiative with the PECARN, CASSY’s cross-validated AUC was 0.89 and 0.87 in study 1 and study 2, respectively, for the prediction of 1 or more SAs within 3 months of the ED visit, which is considered excellent classification accuracy.^31,32

The CASSY is a combination of a multidimensional item response theory model and a prediction model designed to measure interrelated domains of suicide risk and use estimates of these to predict future suicidal behavior. In the derivation study, the CASSY had a cross-validated sensitivity of 83% at specificity of 80% for the prediction of an SA within 3 months, with an overall AUC of 0.89. Perhaps of equal or greater importance, the CASSY also performed strongly in the independent validation sample with an AUC of 0.87, which is considered excellent classification accuracy.^31,32 Requiring a mean of 11 self-report items per adolescent, the CASSY requires 1 to 2 minutes for administration.

It is difficult to directly compare the performance of the CASSY and ASQ, because the CASSY is a screen for risk of SA and provides a dimensional risk estimation, whereas the ASQ was designed to screen for suicide risk more broadly and provides a dichotomous positive or negative result.^11,12 Nevertheless, recent data regarding the ASQ warrant consideration. In a large, universal sample of youth who presented to the ED with psychiatric or nonpsychiatric complaints, the ASQ demonstrated a sensitivity of 60% and a specificity of 92.7% for the prediction of subsequent ED visits for suicide risk, measured by retrospective medical record review.¹⁴ Although the CASSY showed an equivalent level of sensitivity at specificity of 90% for the outcome of suicide attempt, the CASSY, as a dimensional screen, offers the option of choosing a lower specificity (eg, 80%) to achieve higher sensitivity. From a public health perspective, it can be argued that sensitivity—capturing all youth at risk—should be the priority as we strive to improve care models to adequately meet the needs of patients with behavioral health complaints.³³ On the other hand, the ASQ demonstrated high specificity in the DeVylder et al study,¹⁰ and minimizing false positive results and the incremental resources needed for the mental health evaluations in the ED is also important.

The CASSY has several advantages over classical screens such as the ASQ, wherein all adolescents respond to a fixed set of questions: (1) questions are tailored to the severity or risk level of the adolescent, (2) different items are likely to be presented on repeated administration, eliminating response bias, and (3) it provides dimensional severity. Despite these advantages, a standard screen such as the ASQ, which consists of fewer items, may be preferred in some settings, particularly those in which the cost and technical setup of a computerized adaptive screen poses too high a barrier. The CASSY is a proprietary tool; its screening algorithms are available to the public for development of screening software. Input from ED stakeholders about screening priorities will be important to facilitate implementation.³⁴

Several study limitations may affect the generalizability of these findings. We conducted the study in the pediatric EDs of academic health systems, which are not representative of a full range of EDs. We do not have data on study participation refusals and, although we enrolled parents who did not speak English, we did not enroll adolescents who did not speak English. We did not enroll adolescents if they needed surveys read to them, because of privacy concerns. Our loss to follow-up rate was considerable, and these differences may have introduced bias, although with the exception of sex in study 1 and psychiatric chief complaints in study 2, these variables were not associated with SA at the 3-month follow-up. In addition, a significant number of adolescents were approached for study participation but did not enroll, which also creates the potential for selection bias and may limit CASSY accuracy. Under plausible scenarios for the association between the CASSY and risk of suicide attempt, our sensitivity analyses show that performance would remain relatively good, even in the worst cases. We used brief, adapted scales, which may have reduced measurement reliability and validity; however, each clinical risk factor measured was a significant predictor of SAs.³⁰ Mental health evaluations were conducted for youth at high risk, and some youth were referred for treatment, which may have prevented some SAs and reduced the ability to predict them. Finally, we do not know if those lost to follow-up had more self-injury morbidity or mortality.

In summary, we developed and independently validated a computerized adaptive screen (CASSY) for adolescent suicide risk and more specifically the prediction of SAs. We developed and validated this screen with prospective data from geographically diverse EDs. The CASSY demonstrated a high AUC for the prediction of an SA, with an excellent balance of sensitivity and specificity, and is suitable for administration in busy EDs. Important next steps will be to measure the CASSY’s test-retest reliability and develop triage recommendations and conduct implementation studies.

Accepted for Publication: December 3, 2020.

Published Online: February 3, 2021. doi:10.1001/jamapsychiatry.2020.4576

Corresponding Author: Cheryl A. King, PhD, Department of Psychiatry, University of Michigan, 4250 Plymouth Rd, Rachel Upjohn Building, Ann Arbor, MI 48109-2700 (kingca@umich.edu).

Author Contributions: Drs Casper and Gibbons had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: King, Brent, Grupp-Phelan, Casper, Dean, Fein, Gibbons.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: King, Grupp-Phelan, Casper, Mistry, Page, McGuire, Wang, Gibbons.

Critical revision of the manuscript for important intellectual content: King, Brent, Grupp-Phelan, Casper, Dean, Chernick, Fein, Mahabee-Gittens, Patel, Mistry, Duffy, Melzer-Lange, Rogers, Cohen, Keller, Shenoi, Hickey, Rea, Cwik, Page, McGuire, Gibbons.

Statistical analysis: Casper, Page, Wang, Gibbons.

Obtained funding: King, Brent, Grupp-Phelan, Dean, Gibbons.

Administrative, technical, or material support: King, Brent, Grupp-Phelan, Dean, Chernick, Mahabee-Gittens, Patel, Mistry, Duffy, Keller, Hickey, McGuire.

Supervision: King, Brent, Grupp-Phelan, Dean, Fein, Mahabee-Gittens, Mistry, Melzer-Lange, Rogers, Shenoi, Rea, Cwik, Gibbons.

Conflict of Interest Disclosures: Dr Gibbons is a founder of Adaptive Testing Technologies, which distributes the Computerized Adaptive Test—Mental Health suite of computerized adaptive tests (CAT-MH™ is a trademark of Adaptive Testing Technologies); the terms of this arrangement have been reviewed and approved by the University of Chicago in accordance with its conflict of interest policies. Dr King reported grants from the National Institute of Mental Health and is a member of the National Advisory Mental Health Council during the conduct of the study. Dr King also reported being a member of the scientific council of the American Foundation for Suicide Prevention outside the submitted work; in addition, Dr King has copyright work and associated materials are being directly licensed to end users via University of Michigan Office of Technology Transfer and received royalties from Guilford Press. Dr Brent reported royalties from eRT, UpToDate, and Guilford Press; personal fees from Healthwise for consultation and Klingenstein Third Generation Foundation; and grants from the National Institute of Mental Health, American Foundation for Suicide Prevention, Once Upon A Time Foundation, and Beckwith Institute outside the submitted work. Dr Casper reported grants from the National Institute of Mental Health and Health Resources and Services Administration during the conduct of the study. Drs Dean, Patel, Cohen, and Page reported grants from the National Institutes of Health during the conduct of the study. Dr Fein reported grants from the National Institute of Mental Health during the conduct of the study. Dr Mahabee-Gittens reported grants from National Institutes of Health, Maternal and Child Health Bureau, Emergency Medical Services for Children Network Development Demonstration Program, and Health Resources and Services Administration during the conduct of the study, as well as grants from the National Institutes of Health and American Academy of Pediatrics, Ohio Chapter, outside the submitted work. Dr Duffy reported grants from the National Institute of Mental Health during the conduct of the study. Dr Rogers reported grants from the National Institute of Mental Health and Health Resources and Services Administration during the conduct of the study. Dr Shenoi reported grants from the National Institute of Mental Health during the conduct of the study. No other disclosures were reported.

Funding/Support: This study was supported by a grant from the National Institute of Mental Health for the Emergency Department Screen for Teens at Risk for Suicide (ED-STARS) study (grant U01 MH104311 [Drs King, Brent, and Grupp-Phelan]). It was also supported in part by the Health Resources and Services Administration, Maternal and Child Health Bureau, and Emergency Medical Services for Children Network Development Demonstration Program under cooperative agreements U03MC00008, U03MC00001, U03MC00003, U03MC00006, U03MC00007, U03MC22684, U03MC28845, H3MC26201, and U03MC22685.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Group Information: The members of the Pediatric Emergency Care Applied Research Network are listed in Supplement 2.

Disclaimer: The information or content and conclusions in this article are those of the authors and should not be construed as the official position or policy of nor should any endorsements be inferred by the National Institute of Mental Health, Health Resources and Services Administration, Department of Health and Human Services, or the US government.

Meeting Presentation: A subset of preliminary data from Emergency Department Screen for Teens at Risk for Suicide study 1 was presented at the International Summit on Suicide Research; November 7, 2017; Las Vegas, Nevada; the American Psychiatric Association Annual Meeting; May 6, 2018; New York, New York; and the Annual Conference of the American Academy of Child and Adolescent Psychiatry; October 24, 2018; Atlanta, Georgia.

Additional Contributions: We acknowledge the contributions of Dale Woolridge, MD, University of Arizona; Marie Kay, BA, Michelle Robinson, BS, Heather Gramse, BS, and Casey Evans, BS, Pediatric Emergency Care Applied Research Network (PECARN) Data Coordinating Center at the University of Utah; and Esther Ullman, MSW, Lisa Carn, BA, and Maureen O’Brien, MSW, Survey Research Operations at the University of Michigan. we also acknowledge the helpful input of Lisa Colpe, PhD, formerly of the National Institute of Mental Health; Jeffrey Bridge, PhD, Nationwide Children’s Hospital; and Anthony Spirito, PhD, Erin Ryan, MPH, and Thomas Chun, MD, Warren Alpert Medical School at Brown University. Finally, the authors thank the PECARN research coordinators at study sites, the telephone follow-up interview team at the University of Michigan, and study research assistants. The named individuals were compensated for their contributions.