Feasibility and Effects of a Web-Based Adolescent Psychiatric Assessment Administered by Clinical Staff in the Pediatric Emergency Department

Objectives  To determine the adoption rate of the Web-based Behavioral Health Screening–Emergency Department (BHS-ED) system during routine clinical practice in a pediatric ED, and to assess this system's effect on identification and assessment of psychiatric problems.

Design  Descriptive design to evaluate the feasibility of a clinical innovation.

Setting  The ED of an urban tertiary care children's hospital.

Participants  Adolescents from 14 to 18 years of age, without acute or critical injuries or illness, presenting with nonpsychiatric symptoms.

Intervention  The ED clinical staff initiated the use of the BHS-ED system, which identifies and assesses adolescents for depression, suicidal ideation, posttraumatic stress, substance use, and exposure to violence. Treating clinicians reviewed results and followed routine care practices thereafter.

Main Outcome Measures  Adoption rate of the BHS-ED system by nursing staff, identification rates of occult psychiatric problems, and social worker or psychiatrist assessment. Data were collected for 19 months before implementation of the BHS-ED system and for 9 months during implementation.

Results  Of 3979 eligible patients, 1327 (33.4%) were asked by clinical staff to get screened using the BHS-ED; of these 1327 patients, 857 (64.6%) completed the screening and 470 (35.4%) refused. During implementation, identification of adolescents with psychiatric problems increased significantly (4.2% vs 2.5%; odds ratio [OR],1.70; 95% confidence interval [CI], 1.38-2.10), as did ED assessments by a social worker or psychiatrist (2.5% vs 1.7%; OR, 1.47; 95% CI, 1.13-1.90). Of the 857 patients who were screened with the BHS-ED, 90 (10.5%) were identified as having psychiatric problems (OR, 4.58; 95% CI, 3.53-5.94), and 71 (8.3%) were assessed (OR, 5.12; 95% CI, 3.80-6.88).

Conclusions  In a busy pediatric ED, computerized, self-administered adolescent behavioral health screening can be incorporated into routine clinical practice. This can lead to small but significant increases in the identification of unrecognized psychiatric problems.

An estimated 1.5 million adolescents in the United States rely on the emergency department (ED) as their usual source of medical care, a high proportion of whom are from underserved and vulnerable populations.^1,2 A substantial number of these children have a diagnosable behavioral health problem (eg, mental health or addiction).³ Almost 40% of children in need of mental health services for emotional development or behavioral problems do not receive them. This rate increases to more than 50% for uninsured children.⁴ However, untreated behavioral health problems in youth can lead to more severe psychopathology in late adolescence and adulthood.^5-9 In addition to the emotional costs to the child and family, significant economic costs are at stake as well.^10,11

Prior studies have demonstrated that screening in the ED can increase identification rates of psychiatric illness; however, most studies have used research staff to implement and manage the screening procedures.^12-14 This type of screening process can lead to brief interventions in the ED or can inspire the initiation of more long-term therapeutic options. Few studies have attempted to translate research screening into routine clinical practice operated by medical staff, and no studies have investigated a real-time, clinician-initiated, Web-based screening tool for psychiatric illness in a pediatric ED.^15-18 To explore this translational challenge, we integrated a Web-based adolescent behavioral health screening system into the routine clinical care of a busy, urban pediatric ED center. We studied the rates of adoption of this system and the rates of identification of behavioral health problems and subsequent assessment by ED mental health staff.

We used a descriptive design to evaluate the feasibility of a clinical innovation. We also measured the impact of this process using a before and after design.

We enrolled adolescents from 14 to 18 years of age who presented with nonpsychiatric symptoms to an urban, tertiary care pediatric ED with more than 75 000 annual visits. State statute allows patients who are at least 14 years of age to seek mental health care without parental consent, and 18 years is the upper age limit of patients seen at our hospital. We excluded those in the acute and critical triage categories and those who were seen in the urgent care (fast track) area of the ED, which did not offer the patients computers in the treatment rooms. Patients who arrived at the ED with symptoms of or a concern for psychiatric illness were considered to have “self-identified” and were excluded. Other exclusion criteria included patients who were developmentally delayed, patients who did not speak English, patients with significant hearing or vision impairment, patients who were determined by the medical team to be too ill or unable to sit at the computer to complete the screening process, or patients who had been screened in the prior 2 weeks.

To design the Behavioral Health Screening–Emergency Department (BHS-ED) system, we conducted qualitative interviews with 60 adolescent-parent dyads and 45 health care professionals to ascertain their perceptions about, and receptivity to, ED-based screening.^19,20 The resulting screening procedures were then pilot tested by research staff with 40 patients during a 2-month period, revised, and tested again with 6 patients during a 2-week period. Simultaneously, research staff gave several in-service talks to ED physicians, nurses, nurse practitioners, and technicians about the purpose and logistics of the BHS-ED system's content and procedures.

From March 19 through December 19, 2007, ED nurses or ED technicians asked adolescent patients to get screened using the BHS-ED, at any time after the adolescent patient's medical assessment. Using a laminated trifold pamphlet that describes the purpose of the BHS-ED, nurses or medical technicians explained the screening process to the patient and family. The institutional review board waived formal consent because, after we pilot tested the BHS-ED, the screening process was part of the routine care of these adolescent patients in the ED. Family members were encouraged, but not forced, to leave the room so that the patient could complete the screening process in private. The nurses or medical technicians logged the patient onto the Web site and registered them with a password and medical record number. The BHS-ED began with a slide and audio show that explained the rationale for the screening and the standard limits of confidentiality. If desired, the patient could listen to the introduction and subsequent questions using headphones.

After the adolescent completed the screening process by use of the BHS-ED, the ED nurse or ED technician printed out the screening results and placed them on the patient's ED medical record clipboard for review by the physicians and/or nurse practitioners. The first page of the report summarized the results, and subsequent pages provided the patient's answers to all the screening questions. The clinical staff followed routine care in response to the BHS-ED report. This routine care could include further questioning of the adolescent, requesting consultation from a social worker or a psychiatrist, and/or providing referral options. The social worker or psychiatrist did not perform the initial review of the BHS-ED report but could review the data in the report when called for a consultation. For this study, research staff reviewed all BHS-ED reports within 24 hours and confirmed by chart review that any self-reported suicidal ideation had been addressed. To comply with the notion of confidentiality offered to the adolescent patient, a printout of the BHS-ED report was not part of the official medical record; after the patient was discharged from the ED, the report was shredded to prevent unintentional disclosure of identifiable information. However, electronic data were preserved in our data center in accordance with approved institutional review board procedures.

In response to staff feedback during the feasibility phase, we designed and implemented a Web-based, searchable resource database of regional mental health providers. Research staff verified all of the mental health providers' data by telephone. To access the database, a staff member would enter data on a patient's mental health needs, type of insurance, and residency (ie, county) into the computer, and the program generated a list of several local mental health providers with contact information.

The BHS-ED system is a modification of a more comprehensive BHS system, a Web-based, psychosocial assessment tool designed for adolescents in nonpsychiatric medical settings.²¹ The items in the BHS system were based on common risk behavior screens (eg, the Youth Risk Behavior Surveillance Survey) and psychiatric items (eg, diagnostic criteria from the fourth edition of the Diagnostic and Statistical Manual of Mental Disorders).²² The items were reviewed by a panel of adolescent medicine physicians, pediatricians, child psychiatrists, and child psychologists. The items concerning suicide were developed before inclusion in the BHS system and led to an almost 4-fold increase in detection rates in a pediatric primary care setting.²³ The scales are unidimensional, internally consistent (Cronbach α = .75-.87), and discriminate between adolescents with a range of diagnostic syndromes. Confirmatory factor analysis (CFA) verified the existence of the 5 unique but interrelated behavioral health indicators as evidenced by 2 CFA model fit indices: the root mean error of approximation (RMSEA) and the comparative fit index (CFI). The RMSEA provides information about a fit that is adjusted for model parsimony, and the CFI provides information relevant to a null model.²⁴ Guided by suggestions provided by Hu and Bentler,²⁵ we found that the data adequately fit the 5-factor model (RMSEA = 0.06; CFI = 0.86). The internal consistency reliabilities of scales were adequate: depression (α = .79), suicidal ideation (α = .84), posttraumatic stress (α = .74), substance use (α = .67), and family and/or community violence (α = .84). Consistent with prior theory and evidence, there were moderate to strong correlations among factors (range, r = 0.37-0.74). Sensitivity and specificity were strong, with an overall accuracy ranging from 78% to 85%. Patients who had above-scale cutoffs on depression, suicidal ideation, anxiety, and posttraumatic stress disorder symptoms were at least 4 times more likely to have other risk behaviors or stressors.²¹ In adapting the BHS for the ED setting, we assessed only urgent problems of interest to ED clinicians: depression, suicidal ideation, posttraumatic stress, substance use, and family and/or community violence. There are 37 required items in the computerized BHS-ED system and 14 follow-up questions, if indicated. The average amount of time spent by subjects taking the BHS is 10 minutes.

Because of our focus on depression and because the BHS depression scale had not yet been validated, we incorporated the Beck Depression Inventory–Fast Screen (BDI-FS) system into the BHS-ED system for the computer-based assessment (with permission from, and remuneration to, the copyrighting agency).²⁶ Consisting of 7 of the original 21 BDI items, the BDI-FS system screens for depression in patients reporting somatic and behavioral symptoms that may be attributable to biological, medical, or substance abuse problems. Validated in a medical setting, the BDI-FS system demonstrates high sensitivity and specificity, strong internal consistency, and it correlates well (r > .60) with other measures that assess depressive symptoms.

Medical Decision Logic, Inc (Baltimore, Maryland), provides the Web-technology platform to support the implementation and deployment of the BHS-ED system. The tool is built on the MDLogix Health Science Process Framework (HSPF), a Web 2.0 (and Web 3.0–enabled) technology platform that supports flexible scientific data specification, collection, reporting, management, and archiving. The HSPF also supports scientific workflows such as study protocols. Web 2.0 is a term generally used to represent the current generation of Web applications with rich and user-friendly interactive interfaces, such as provided by Google and Facebook. A key technology for this purpose is called AJAX (a technology popularized by Google and based on JavaScript and XML). Web 3.0 refers to the growing interest and capability for “intelligent” applications that “understand” the semantics of collected information to better confer users' desires. The Web platform also allows user sites to add questions to fit the needs of their practice.

Our primary outcomes were (1) the rate of adoption of the BHS-ED system into clinical practice, (2) the identification of psychiatric illness, and (2) the number of ED-based behavioral health assessments by social workers and/or psychiatrists that resulted from the screenings.

For all eligible patients during the study period, we obtained the following data from the ED electronic tracking system: age, sex, symptoms during triage, discharge diagnosis (up to 4 International Classification of Diseases, Ninth Revision [ICD-9 ], codes), disposition at discharge from ED, and follow-up instructions. The disposition of the patient at ED discharge (a free-text field) was then coded by 2 trained research assistants for the presence or absence of a referral for mental health treatment (κ = 0.93).

Paper copies of social work and psychiatry consultation forms are routinely scanned in their entirety into an electronic medical record viewing system. Two trained research assistants reviewed all social work and psychiatry consultation forms during the study period to determine whether a referral for mental health treatment was made after the consultation (κ = 0.75). Social workers also maintain a log sheet of all the patients they see in the ED. Two research assistants reviewed these forms to identify consultations for mental health that were not discovered by use of the above-mentioned methods (κ = 0.89). Coding discrepancies were resolved by the project coordinator, a doctoral-level psychologist.

A patient was considered to have a psychiatric illness or a behavioral problem that was “identified” in the ED if he or she received a psychiatric discharge diagnosis (ICD-9 codes 291-313, excluding codes 293 [transient mental disorders due to conditions classified elsewhere], 294 [persistent mental disorders due to conditions classified elsewhere], 299 [pervasive developmental disorders], and 305.1 [nondependent tobacco use disorder]). A second indicator of identification was if a patient received an ED-based behavioral health assessment and/or a referral. We categorized “ED-based behavioral health assessment” as any patient who received a consultation from a social worker or a psychiatrist to evaluate a psychiatric illness or a behavioral problem.

To measure the rate of adoption of the BHS-ED system, we compared the number of eligible patients in the ED with those patients who were asked to complete the screening process by use of the BHS-ED system. To measure the effect of the BHS-ED system on rates of identification and assessment of psychiatric illness, we performed intent-to-treat analyses comparing eligible patients before implementation and during implementation with respect to identification of and assessment for psychiatric diagnoses. Secondary analyses compared eligible adolescent patients prior to the implementation of the BHS-ED system with those patients who were asked by a clinical staff member to complete the screening process during implementation. We used the χ² test and the Fisher exact test to compare categorical variables for all outcome analyses, and we used the t test to compare the ages of patients in the 2 groups. For 80% power and an α of .05, we calculated the need to enroll 653 patients in each group to demonstrate an increase in identification rate from 2.5% to 5%.²⁷

In the 19 months before implementation of the BHS-ED system, 7722 patients from 14 to 18 years of age were seen in the main ED and were triaged according to whether they had urgent or nonurgent symptoms. During the implementation period, there were 4202 patients seen in the main ED (not fast track) who were classified as “urgent” or “nonurgent” by the triage nurse. Of these 4202 patients, 88 presented with a psychiatric illness or a behavioral health problem, 3 had missing diagnosis data, and 93 were developmentally delayed, did not speak English, had significant hearing or vision impairment, had been screened in the prior 2 weeks, or were determined by the medical team to be too ill or unable to sit at the computer to complete the screening process using the BHS-ED system. Thirty-nine patients presented during a time in the implementation period when the computer program or server was not functioning. Of the remaining 3979 eligible patients seen during the implementation period, 1327 (33.4%) were asked by a clinical staff member to complete the screening process using the BHS-ED system. Of these 1327 patients, 857 (64.6%) completed the screening process and 470 (35.4%) refused. All patients who began the screening process using the BHS-ED system completed it. The age, sex, race, ethnicity, and triage symptom category did not differ between the 3 groups: those patients who were not asked to complete the screening process using the BHS-ED system, those who were asked and who completed the screening process, and those who were asked but refused (Table 1).

An intent-to-treat analysis compared all eligible patients before (n = 7222) and during (n = 3979) implementation. A significant increase in the identification of patients with psychiatric illness was found during the implementation period, compared with the period before implementation (4.2% vs 2.5%; odds ratio [OR], 1.70; 95% confidence interval [CI], 1.38-2.10). Also, a significant increase in ED-based behavioral health assessments by a social worker or psychiatrist was found during the implementation period, compared with the period before implementation (2.5% vs 1.7%; OR, 1.47; 95% CI, 1.13-1.90). This finding suggests that even when not all patients can be screened, the implementation of this process can increase the odds of discovering occult psychiatric illnesses in the ED.

Secondary analyses compared the preimplementation group of patients with the 1327 patients who were asked by an ED nurse or technician to complete the screening process using the BHS-ED system, as well as with the 857 patients who actually completed the screening process. Compared with the preimplementation group of patients, those patients who were asked to complete the screening process were more likely to be identified as having a psychiatric illness (7.8% vs 2.5%; OR, 3.28; 95% CI, 2.57- 4.20) and to be assessed by a social worker and/or a psychiatrist (5.8% vs 1.7%; OR, 3.49; 95% CI, 2.62-4.65). Similarly, compared with the preimplementation group of patients, those patients who actually completed the screening process using the BHS-ED system were much more likely to be identified (10.5% vs 2.5%; OR, 4.58; 95% CI, 3.53-5.94) and assessed (8.3% vs 1.7%; OR, 5.12; 95% CI, 3.80-6.88).

Of the 857 subjects completing the screening process using the BHS-ED, 152 (17.7%), 29 (3.4%), and 8 (0.9%) reported mild, moderate, and severe depressive symptoms on the BDI-FS, respectively. Ninety-five (11.1%) reported suicidal thoughts within the past year, and 31 (3.6%) reported suicidal thoughts within the past 2 weeks. Nine of these patients endorsed having a suicide plan. Of these 9 patients who endorsed suicidal ideation, 2 were referred to an inpatient psychiatric hospital. The chief symptoms of these patients were abdominal pain and chest pain. All patients who reported suicidal ideation in the past 2 weeks were assessed further and/or referred to an additional mental health care facility from the ED (Table 2).

The need for and the value of mental health screening in ambulatory medical settings has been well established. Yet the translation of research-based screening procedures into standard medical practice for pediatric patients in the ED has been untested.^16,17 The present study demonstrates that a Web-based screening system has the potential to assist in this mission. During a 9-month period, the clinical staff incorporated this screening tool into their routine care for one-third of the eligible adolescent patients in a busy ED. Two-thirds of these eligible patients completed the screening process during their ED visit. Of those screened, more than 10% endorsed significant levels of current psychiatric distress, and slightly fewer than 10% received an evaluation by a social worker or psychiatrist. These data suggest that, without assistance from research support staff, ED clinicians can integrate a behavioral health screening system into their work flow and identify a substantial number of patients who may be in need of behavioral health services.

It is also clear that, with the 33% adoption rate, there is room for improvement in the screening process. Several barriers to screening have been identified. From the patient's perspective, the stigma of psychiatric illness, along with a concern for the clinician's ability to handle this personal information in a culturally sensitive manner, hampers the patient's and his or her family's acceptability of this process. In a previous study¹⁹ in our ED regarding this screening process, caregivers and patients suggested that the screening needed to occur earlier in the ED visit, to decrease the perception that the child was being “targeted.” In addition, they requested that the staff provide specific information about the meaning of the screening results and that the staff provide the information needed to increase referral acceptability.¹⁹ In a similar study,²⁰ ED medical providers expressed concern that adolescents who endorsed dangerous behavior and did not receive an appropriate response would “slip through the cracks,” and these ED medical providers worried that the mental health system could not support the burden of referrals that the BHS-ED could generate. Other barriers include lack of time due to work-flow demands, lack of standardized protocols, discomfort with discussing sensitive psychosocial problems, and uncertainty about what to do with the results.^28,29 From the administrator's perspective, the absence of reimbursement for psychiatric screening makes it difficult to mandate these procedures. Adoption rates may increase if this screening process was incorporated into job descriptions and if automated reminders were provided for staff.³⁰ In addition, it is assumed that more screenings would occur if there was improved collaboration between the ED medical providers and mental health follow-up services, and if there was better insurance reimbursement of medical providers for mental health services. The low adoption rate may have also contributed to the relatively small differences in identification and assessment of psychiatric illness before and after implementation of the screening process. Despite the marginal adoption rate found in our study, the BHS-ED system has several advantages over existing screening options. The Web-based platform addresses the problems associated with administration, scoring, interpretation, and data integration. The questions are self-administered and self-explanatory, with the use of skip logic to decrease the duration of the screening process for less severe patients and with the use of audio files for reading-impaired or illiterate patients. The program scores the data and generates a report for clinicians to use in real time.

Some potential limitations should be mentioned. One might question the veracity of adolescents who report on questionnaires. However, studies show that adolescents are honest on self-report measures and, in fact, prefer reporting mental health symptoms in a computer format rather than verbally with a clinician.^31-33 The percentage of patients who endorsed suicidal ideation in our sample (11%) was lower than that found in the Youth Risk Behavior Surveillance Survey (16%).³⁴ This may be a function of different sample characteristics or of the context of assessment (ED vs schools). Second, it is possible that medical staff used the BHS-ED as an indicated screening (eg, when they suspected problems). This would have served to inflate the identification rates in this sample. However, research team observation as well as postimplementation interviews with the nurses suggest that the administration of the screening varied more by staff person than by patient characteristics. Third, our study did not look at outpatient referrals or the success of those referrals. Future studies will examine this latter phase of the screening and referral processes.

To our knowledge, this is the first study to demonstrate that behavioral health screening can be adopted as a routine medical procedure for noncritical adolescents in a pediatric ED. Even with moderate use, important clinical information was obtained that led to further ED-based assessment for mental health care. Future research should explore how to resolve additional barriers to adopting the BHS-ED system, modify this system to other types of ED settings, and estimate the economic ramifications of ED-based adolescent psychiatric screenings.

Correspondence: Joel A. Fein, MD, MPH, Division of Emergency Medicine, The Children's Hospital of Philadelphia, 3501 Civic Center Boulevard, ED-CTRB, 9th Floor, Philadelphia, PA 19104 (fein@email.chop.edu).

Accepted for Publication: April 16, 2010.

Author Contributions: Dr Fein had full access to the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Fein, Pailler, Barg, Wintersteen, Tien, and Diamond. Acquisition of data: Fein, Pailler, Hayes, and Tien. Analysis and interpretation of data: Fein, Pailler, Barg, Wintersteen, and Diamond. Drafting of the manuscript: Fein, Pailler, Wintersteen, Hayes, Tien, and Diamond. Critical revision of the manuscript for important intellectual content: Fein, Pailler, Barg, Wintersteen, Tien, and Diamond. Statistical analysis: Pailler and Hayes. Obtained funding: Fein, Barg, and Tien. Administrative, technical, and material support: Pailler, Barg, Hayes, and Tien. Study supervision: Fein, Pailler, and Barg.

Financial Disclosure: Dr Tien is a shareholder and employee of Medical Decision Logic, Inc.

Funding/Support: Funded by grant H34MC04366 from the Emergency Medical Services for Children Program, Maternal and Child Health Bureau, Health Resources and Services Administration, US Department of Health and Human Services (Dr Fein, principal investigator).

Role of the Sponsor: The funding agency had no role in the design and conduct of the study; in the collection, analysis, and interpretation of the data; or in the preparation, review, or approval of the manuscript.

Previous Presentation: This paper was presented at the Annual Meeting of the Pediatric Academic Societies; May 3, 2008; Honolulu, Hawaii.

Additional Contributions: We would like to thank the nurses, technicians, social workers, and physicians of the ED at The Children's Hospital of Philadelphia for their support during this study.