An Electronic Screen for Triaging Adolescent Substance Use by Risk Levels

Importance  Screening adolescents for substance use and intervening immediately can reduce the burden of addiction and substance-related morbidity. Several screening tools have been developed to identify problem substance use for adolescents, but none have been calibrated to triage adolescents into clinically relevant risk categories to guide interventions.

Objective  To describe the psychometric properties of an electronic screen and brief assessment tool that triages adolescents into 4 actionable categories regarding their experience with nontobacco substance use.

Design, Setting, and Participants  Adolescent patients (age range, 12-17 years) arriving for routine medical care at 2 outpatient primary care centers and 1 outpatient center for substance use treatment at a pediatric hospital completed an electronic screening tool from June 1, 2012, through March 31, 2013, that consisted of a question on the frequency of using 8 types of drugs in the past year (Screening to Brief Intervention). Additional questions assessed severity of any past-year substance use. Patients completed a structured diagnostic interview (Composite International Diagnostic Interview–Substance Abuse Module), yielding Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) substance use diagnoses.

Main Outcomes and Measures  For the entire screen and the Screening to Brief Intervention, sensitivity and specificity for identifying nontobacco substance use, substance use disorders, severe substance use disorders, and tobacco dependence were calculated using the Composite International Diagnostic Interview–Substance Abuse Module as the criterion standard.

Results  Of 340 patients invited to participate, 216 (63.5%) enrolled in the study. Sensitivity and specificity were 100% and 84% (95% CI, 76%-89%) for identifying nontobacco substance use, 90% (95% CI, 77%-96%) and 94% (95% CI, 89%-96%) for substance use disorders, 100% and 94% (95% CI, 90%-96%) for severe substance use disorders, and 75% (95% CI, 52%-89%) and 98% (95% CI, 95%-100%) for nicotine dependence. No significant differences were found in sensitivity or specificity between the full tool and the Screening to Brief Intervention.

Conclusions and Relevance  A single screening question assessing past-year frequency use for 8 commonly misused categories of substances appears to be a valid method for discriminating among clinically relevant risk categories of adolescent substance use.

Substance use causes substantial morbidity and mortality among adolescents (age range, 12-17 years) and contributes to mental health disorders and negative social sequelae.¹ Early initiation of substance use is also associated with increased odds of developing a substance use disorder (SUD) and experiencing substance-related problems, even as an adult.^2,3 Screening adolescents for substance use and intervening immediately can reduce the burden of addiction and substance-related morbidity.^1,4

Pediatricians and other primary care physicians play a vital longitudinal role in the lives of children and adolescents and are a trusted source of medical information. As such, they may be uniquely positioned to influence their patients’ decisions regarding substance use. The American Academy of Pediatrics (AAP) and other professional organizations recommend that primary care physicians screen all adolescents for substance use and provide guidance tailored to the level of substance use as part of routine health care.

Research performed in primary care clinics and emergency departments suggests that positive reinforcement to delay substance use initiation for adolescents who have no past-year alcohol or drug use, brief medical advice to quit for those with past-year substance use but without associated problems,⁵ and brief interventions based on motivational interviewing targeted at reducing use^6,7 or engaging in treatment⁸ for adolescents who have developed a SUD are promising interventions. Teens with severe nicotine use disorder, previously termed nicotine dependence, may also benefit from pharmacological treatment.^9-12 A policy statement developed by the AAP recommends pediatricians follow up per the intervention outline noted above.¹³

To be practical in the busy medical office setting, screening must quickly and accurately triage adolescents into risk categories that determine the appropriate level of intervention. Brief structured tools that eliminate the need for lengthy clinical assessments for low-risk patients can spare precious clinical contact time. Several tools have been developed and validated for use with adolescents. The 6-question CRAFFT screening tool, a mnemonic acronym developed by Knight et al¹⁴ that stands for the first letter in a keyword for each of the tool's 6 key questions (car, relax, alone, forget, friends or family, trouble), was initially developed as an assessment tool to discriminate between low- and high-risk substance use among adolescents who report any past-year use of alcohol or drugs.^4,15 To identify adolescents with past-year substance use, 3 post hoc screening questions were later added (“In the past year, did you drink any alcohol [more than a few sips], smoke any marijuana or hashish, use anything else to get high?”). Although the 3 additional questions have face validity, their psychometric properties were not formally tested for alcohol or drugs, and they do not assess tobacco use.¹³ Furthermore, it is not known whether they are sensitive for identifying problem use of substances that adolescents may not consider drugs, such as over-the-counter medications, synthetic substances, herbal preparations, or prescription medications, misuse of which has increased.¹⁶

The objective of this study was to describe the psychometric properties of an electronic screen and brief assessment tool that triages adolescents into 4 actionable categories regarding their experience with nontobacco substance use: (1) no past-year alcohol or drug use, (2) past-year alcohol or drug use without a SUD, (3) mild or moderate SUD, and (4) severe SUD. The tool has 3 additional categories for tobacco use: (1) no tobacco use, (2) tobacco use, and (3) nicotine dependence.

We recruited a convenience sample of adolescent patients aged 12 to 17 years (mean age, 15.4 years) who presented for a medical evaluation at 1 of 3 sites at Boston Children's Hospital: the Adolescent/Young Adult Medical Practice, the Primary Care Center, and the Adolescent Substance Abuse Program. These 3 sites allowed for sufficient sampling across the age range and across substance use risk and diagnostic categories. Patients were excluded if they were non–English speaking, were medically or emotionally unstable on the day of the appointment, or had been in residential treatment for a SUD in the past 3 months. Eligible patients were invited to participate at the end of a primary care appointment or before their first appointment in a substance abuse program. Interested patients met with a research assistant (R.Z. and A.S.), who obtained written informed consent. A waiver of parental consent allowed participants independence in electing to participate in the study because lower-risk samples result when parental consent is required.¹⁷ Parents, if present, were coinformed during the consent procedure, and adolescents were encouraged to consult with them before deciding whether to participate. All participants were guaranteed full confidentiality in their responses unless a serious safety issue was indicated, in which case they met privately with a medical or mental health care professional after completion of the study. Participants received a small stipend ($5) in the form of a gift card. The study was granted a certificate of confidentiality from the National Institutes of Health and was approved by the Boston Children’s Hospital Institutional Review Board.

We designed a screen and brief assessment tool that began with a comprehensive stem question, based on the National Institute on Drug Abuse quick screen,¹⁸ assessing the frequency of past-year use (none, once or twice, monthly, weekly, almost daily, or daily) for 8 categories of substances commonly used by adolescents. Patients completed the screen and brief assessment tool from June 1, 2012, through March 31, 2013. Those who reported alcohol use were asked 1 question adapted from the Alcohol Use Disorders Identification Test on blackouts and alcohol-related injuries, 1 question on frequency of binge drinking, 1 question on combining substances, and 2 questions on quantity and frequency of alcohol use. The RAFFT questions (CRAFFT¹⁴ without the “C” question relating to riding in a car driven by someone who was intoxicated) were used to determine the likelihood of problems. We did not include the car question because most participants would be too young to drive and because mixing reports of driving while impaired and riding with an impaired driver could complicate interpretation. Participants who reported monthly or greater tobacco use were asked, “Have you used tobacco 1 or more times per day for 2 or more weeks?” to identify potential nicotine dependence. A skip pattern was used to ensure that only relevant questions were administered. The screen and brief assessment tool varied in length from the single 8-part frequency question up to a total of 18 questions that were administered on a tablet computer in a private location. Participants were randomized to self- or interview-administered screens. The mean time to completion was 32 seconds (range, 9-102 seconds). The questions are listed in the Figure.^18,19

The validation assessment included a research eligibility form, which recorded age, sex, race/ethnicity, number of parents in the household, and highest level of parent education. We used the Composite International Diagnostic Interview–Substance Abuse Module (CIDI-SAM)²¹ as our criterion standard for SUDs based on Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) (DSM-5) criteria²² for alcohol, marijuana, and other substances except tobacco, for which Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition) (DSM-IV) criteria²³ were used because the CIDI-SAM did not include a question on tobacco craving. Participants also completed a 90-day Timeline Follow-Back Calendar,²⁴ which recorded frequency of alcohol, marijuana, tobacco, and other drug use.

Using univariate analysis, we calculated frequencies for demographic factors and the 4 risk categories: (1) no past-year use, (2) past-year use without a SUD, (3) DSM-5 mild (2-3 criteria) or moderate (4-5 criteria) SUD, and (4) DSM-5 severe SUD (≥6 criteria). To assess the criterion validity of the screen and brief assessment tool, we calculated sensitivity and specificity for any (nontobacco) substance use, SUD, severe SUD, tobacco use, and DSM-IV dependence in the past 12 months. As part of post hoc analyses, we repeated these calculations for (nontobacco) SUD using only the frequency question for each substance. We refer to these frequency questions as the Screening to Brief Intervention (S2BI). Table 1 provides the definitions of substance use categories for each version of the tool. We also compared sensitivity and specificity for these categories for those who self-administered the screen (n = 102) and those who received the screen by a trained interviewer (R.Z. and A.S.) (n = 111). We used SUDAAN statistical software, version 11.0.0 (RTI International), with clinic site as a nest variable that accounted for correlated error from the site cluster sample design to estimate 95% CIs and to perform statistical tests for differences in survey administration mode.

Among 457 age-eligible patients scheduled for an outpatient clinic appointment, we excluded 117 because they did not speak English (n = 11), were medically or emotionally unstable at the time of the appointment (n = 52), were not developmentally able to assent or complete the survey (n = 11), had been in a residential treatment facility in the past 90 days (n = 20), or were deemed ineligible by the patient’s primary care physician on the day of the appointment for unspecified reasons (n = 23). A total of 340 patients were invited to participate in the study: 245 from the Adolescent/Young Adult Medical Practice, 51 from the Adolescent Substance Abuse Program, and 44 from the Primary Care Center. A total of 157 (64.1%), 37 (72.5%), and 22 patients (50.0%) enrolled in the study from each of those 3 clinics, respectively, for a total of 216 study participants. Technical problems caused incomplete screens for the first 3 participants, resulting in an analyzable sample of 213. Of the study participants, 142 were female (66.7%), which was reflective of the sex distribution of patients presenting to the primary care clinics. A total of 119 (55.9%) lived in a 2-parent home, and 117 (54.9%) had a parent with a bachelor’s degree or higher. Race/ethnicity was evenly distributed. The characteristics of the study sample are given in Table 2.

For nontobacco substance use, 123 participants (57.7%) reported no past-year substance use, 49 (23.0%) reported use but did not meet criteria for a SUD, 22 (10.3%) met criteria for a mild or moderate SUD, and 19 (8.9%) met criteria for a severe SUD. The screening and brief assessment tool, as originally conceived, has sensitivity and specificity of 90% (95% CI, 66%-97%) and 83% (95% CI, 76%-88%) for identifying a past-year SUD and 90% (95% CI, 66%-97%) and 91% (95% CI, 86%-94%) for identifying a severe SUD, respectively. Sensitivity and specificity did not differ between self-administration vs interview administration for any category of use or SUD.

Table 3 provides the prevalence of use for each substance and the sensitivity and specificity of the single past-year frequency question for rates of alcohol and cannabis use, alcohol and cannabis use disorders, any SUD, and severe SUD vs diagnosis of a SUD by the CIDI-SAM interview. Rates of SUDs could not be determined for 9 specific substances with the full screen and brief assessment tool because the 10 RAFFT questions do not distinguish substances. Sensitivity and specificity were high for all categories, ranging from 79% to 100%. No differences in sensitivity or specificity were found between self- and interview-administered screens (data not shown). Our tool identified more past-year substance use than the criterion standard; in total, 19 more participants reported past-year use compared with those administered the CIDI-SAM. Of those, 8 reported past-year alcohol or marijuana use and 11 reported past-year “nonmedical use of an over-the-counter medication.” Four participants who did not report past-year alcohol or marijuana use endorsed use of tobacco in the past year. All participants who reported use of illegal drugs, inhalants, herbal preparations, synthetic drugs, or misuse of prescription drugs in the past year also reported past-year alcohol and/or marijuana use. No participant reported past-year substance use on the CIDI-SAM but not on the electronic tool.

Table 3 also provides the prevalence of tobacco use as detected by the CIDI-SAM and the sensitivity and specificity of the screen and brief assessment tool for detecting tobacco use and DSM-IV nicotine dependence. As with other substances, sensitivity and specificity were high, ranging from 75% to 98%.

We describe an electronic tool that is brief and easy to administer to adolescents presenting for routine care. The single past-year frequency question from the S2BI was sensitive and specific for discriminating among 4 categories of substance use experience (no past-year use, use without a SUD, mild or moderate SUD, and severe SUD) for each substance. This screening strategy is similar to the single-question screen used with adults²⁵ and the National Institute on Alcohol Abuse and Alcoholism (NIAAA) youth alcohol screening guide, which triages risk level based on the frequency of past-year alcohol use.²⁶ The tool’s psychometric properties were similar regardless of the format of administration (ie, self-administered vs interview administered), suggesting that the tool can be administered either way to suit the needs of a particular medical setting.

The initial design of our tool included frequency screening questions and assessment questions selected from previously validated tools. However, we found that frequency screening questions alone resulted in similar psychometric properties as the full-length tool. Despite recommendations for universal screening as part of routine adolescent health care, self-reported screening rates as reported by physicians were very low in a study by Millstein and Marcell.²⁷ Another study⁵ found higher screening rates but also noted that most physicians do not use validated tools. Time constraints are one of the most frequently cited reasons for forgoing screening.^28-30 The S2BI, which consists of a single question for each substance screened and 2 questions for tobacco, could lower this barrier.

A unique quality of the S2BI is the ability to discriminate between mild or moderate and severe SUDs. The AAP guidelines recommend further evaluation whenever an adolescent has high-risk substance use.¹³ However, physician acumen for identifying patients with severe SUDs is poor.³¹ This finding suggests that many opportunities for referring adolescents to treatment are missed with standard practices. Less than 10% of adolescents with a SUD receive any treatment, and most who receive treatment are referred by the criminal justice system,³² with few coming from primary care. A tool that can accurately identify adolescents who meet criteria for severe SUD could be a step toward improving the rates of referral to treatment for this underserved population.

The S2BI identified more substance use than the CIDI-SAM interview; 8 participants reported past-year alcohol or marijuana use on the S2BI but not on the CIDI-SAM. The screening questions were based on the National Institute on Drug Abuse quick screen,³³ which asked, “In the past year, how many times have you [used alcohol]” followed by forced-choice frequency items. This is in contrast to the “yes or no” CIDI-SAM question, which was phrased as, “Have you had a drink containing alcohol in the past 12 months?” The “how many times” question, which is also recommended in the NIAAA youth alcohol screening guide,²⁶ appears to be more sensitive than the “have you ever” stem recommended in the AAP guidelines.³⁴ Participants who reported substance use only on the electronic screen were in the lowest frequency category, and they likely did not have a SUD (they were not administered the branching questions in the CIDI-SAM). An error that miscategorizes an adolescent as a nonuser would misdirect the physician to give positive reinforcement, which is intended to maintain the status quo, instead of brief advice, which is intended to reduce use. The clinical effect of giving positive reinforcement to adolescents who are occasional users is not known.

Although, to our knowledge, a link between screening and increased substance use has never been reported, a concern with using “how many times” as the stem question is that adolescents may think that physicians expect them to use substances. This could be problematic, especially for younger adolescents and those who have not initiated substance use. The NIAAA youth alcohol screening guide²⁶ recommends that physicians include a statement about the rarity of alcohol use by younger children in their positive reinforcement statement. A similar strategy could be used with our tool. In addition, our results suggest that the screen could be terminated for those who report no alcohol, marijuana, or past-year tobacco use because no participant reported use of another drug on our criterion standard without at least 1 of these 3.

Few participants who did not use alcohol or marijuana reported use of any other substance. Four individuals reported tobacco use without other substances, and 11 reported “nonmedical over-the-counter or prescription medication misuse” on our screening questions, although none of these individuals reported this use on the CIDI-SAM. It is possible that these participants did not understand the term nonmedical use. To limit this potential error, we therefore recommend administering the question about past-year alcohol, marijuana, and tobacco use to all adolescents and asking only those who respond positively about other substances. The S2BI is also compatible with the CRAFFT questions, which could be administered to adolescents who screen positive for a SUD to explore problems associated with substance use as the first step of a brief intervention. This approach needs further assessment.

This study had a number of strengths and some possible limitations. We recruited a diverse sample that represented both sexes, a mix of race/ethnicity, and a broad representation of the age range of interest. The sample included adequate numbers of adolescents in each substance use risk or diagnostic category to allow for psychometric analyses. Participation rates were moderate, ranging from 50% to 73%. The rate of substance use in adolescents aged 14 to 17 years presenting to primary care clinics in our final analytical sample is similar to a previous study³⁵ (44.7% vs 49.8%, P = .27) within the same age range at this hospital, suggesting that higher-risk adolescents were not preferentially opting out. Sixteen percent of our sample was recruited from a subspecialty substance abuse program for adolescents and had already been identified as using substances before completing the screen (although not all adolescents who are referred to this program are diagnosed as having a SUD). All these participants completed the screen before their initial evaluation appointment in the substance abuse program to reduce the likelihood of affecting their response to screen questions. Nonetheless, we recommend that the S2BI be tested in other settings to confirm our findings.

The S2BI uses a strategy similar to the NIAAA youth alcohol screening tool³⁶ and the single-item quick screen used for adults.²⁵ Our findings suggest that frequency screening questions are also a valid and efficient means of triaging alcohol and drug use into clinically meaningful risk levels in adolescents. The S2BI can thus be used to direct physicians to apply evidence-based brief intervention for adolescent substance use appropriate to the screen-identified risk level.

Accepted for Publication: April 24, 2014.

Corresponding Author: Sharon Levy, MD, MPH, Division of Developmental Medicine, Adolescent Substance Abuse Program, Boston Children’s Hospital, 300 Longwood Ave, Boston, MA 02115 (sharon.levy@childrens.harvard.edu).

Published Online: July 28, 2014. doi:10.1001/jamapediatrics.2014.774.

Author Contributions: Mr Sherritt and Ms Ziemnik had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Levy, Sherritt, Van Hook, Shrier.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Levy, Sherritt, Ziemnik, Shrier.

Critical revision of the manuscript for important intellectual content: Weiss, Sherritt, Spalding, Van Hook, Shrier.

Statistical analysis: Levy, Sherritt, Ziemnik.

Obtained funding: Levy, Van Hook.

Administrative, technical, or material support: Sherritt, Ziemnik, Spalding, Van Hook.

Study supervision: Levy, Weiss, Sherritt, Shrier.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was supported by grants 5K23DA019570-05S1 (Dr Levy), U10DA15831 (Dr Weiss), and K24DA022288 (Dr Weiss) from the National Institute on Drug Abuse.

Role of the Sponsors: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: Danielle Murphy, BA, Josef Korbel School of International Studies, Denver, Colorado (paid research assistant), Roman Pavlyuk, BA, Division of Developmental Medicine, Boston Children’s Hospital, Boston, Massachusetts (paid research assistant), Sakina Sojar, BS, University at Buffalo School of Medicine and Biomedical Sciences, Buffalo, New York (volunteer research assistant), and Emily Axel, BA, Miami University, Oxford, Ohio (volunteer research assistant), assisted with data collection.