Flowchart of participant recruitment and randomization.
eFigure. Screen of Drug Use (SoDU)
eTable. Statistical Properties of the Screen of Drug Use for Subgroups of Patients Based on Sex, Age, Racial/Ethnic Background, Marital Status, Educational Level, and Posttraumatic Stress Disorder Status
Tiet QQ, Leyva YE, Moos RH, Frayne SM, Osterberg L, Smith B. Screen of Drug UseDiagnostic Accuracy of a New Brief Tool for Primary Care . JAMA Intern Med. 2015;175(8):1371-1377. doi:10.1001/jamainternmed.2015.2438
Copyright 2015 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.
Illicit drug use is prevalent, and primary care provides an ideal setting in which to screen for drug use disorders (DUDs) and negative consequences of drug use (NCDU). Comprehensive reviews have concluded that existing drug use screening instruments are not appropriate for routine use in primary care.
To develop and validate a screening instrument for drug use.
Design, Setting, and Participants
We revised items drawn from existing screening instruments and conducted signal detection analyses to develop the new instrument. We approached 3173 patients at 2 primary care clinics in a US Department of Veterans Affairs health care system from February 1, 2012, through April 30, 2014. A total of 1300 (41.0%) patients consented to the study, of whom 1283 adults were eligible (mean [SD] age, 62.2 [12.6] years). In the last 12 months, 241 (18.8%) participants reported using illicit drugs or prescription medication for a nonmedical purpose, and 189 (14.7%) reported 1 or more NCDU. A total of 133 (10.4%) patients met DSM-IV criteria for a DUD. The sample was randomly divided first to develop the measure and then to validate it.
Main Outcomes and Measures
The Mini-International Diagnostic Interview was used as the criterion for DUDs, and the Inventory of Drug Use Consequences was used as the criterion for NCDU.
The screening instrument has 2 questions. The first is, “How many days in the past 12 months have you used drugs other than alcohol?” Patients meet that criterion with a response of 7 or more days. The second question asks, “How many days in the past 12 months have you used drugs more than you meant to?” A response of 2 or more days meets that criterion. The screening instrument was 100% sensitive and 93.73% specific for DUDs (643 patients); when replicated in the second half of the sample (640 patients), it was 92.31% sensitive and 92.87% specific. The screening instrument was 93.18% sensitive and 96.03% specific for NCDU (643 patients); when replicated in the second half of the sample (640 patients), it was 83.17% sensitive and 96.85% specific.
Conclusions and Relevance
The 2-item screen of drug use has excellent statistical properties and is a brief screening instrument for DUDs and problems suitable for busy US Department of Veterans Affairs primary care clinics.
Nearly 1 in 10 Americans currently uses illicit drugs.1 Annual costs of illicit drug use are estimated at more than $11 billion for health care and $182 billion for crime and lost productivity.2 Drug treatment is effective,3 but nearly 90% of those who need treatment for illicit drug use disorder (DUD) or alcohol use disorder did not receive specialty treatment in the past year.1 As the gateway to health care4,5 and provider of continuity services,6 primary care is an ideal setting in which to screen for DUD and drug-related problems and to provide brief behavioral therapies and referral to treatment, especially since drug use often goes unrecognized in primary care settings.7
Validated screening instruments for alcohol use disorders8 have been recommended for use in primary care7; however, reviews have identified weaknesses of existing screening instruments for DUDs and concluded that they were not appropriate for routine use.9- 12 Therefore, we developed and validated the Screen of Drug Use (SoDU) for primary care settings.
To develop and validate the new measure, we revised items drawn from existing drug use screening instruments and recruited 1300 patients from 2 primary care clinics at a US Department of Veterans Affairs (VA) medical center. We conducted item performance analyses using half of the sample (development sample) to select the best and most parsimonious items. We then validated the chosen items for diagnostic accuracy on the other half of the sample (validation sample) and examined the performance of the SoDU on subgroups of patients, including age, sex, race or ethnicity, marital status, educational level, and posttraumatic stress disorder (PTSD) status. The Stanford University School of Medicine Institutional Review Board reviewed and approved all study procedures, and the participants provided written informed consent.
This study recruited patients from February 1, 2012, through April 30, 2014, at 2 primary care clinics at a VA medical center in northern California. We included only current primary care patients who had visited the clinics within 2 weeks and compensated participants for their time.
Of the 3173 approached patients, 1518 refused, 355 scheduled an appointment but did not follow through (eg, did not attend or cancelled the appointment), and 1300 (41.0%) completed the informed consent process (Figure). Of the 1300 consented participants, we excluded 17 individuals for various reasons, including cognitive issues (n = 5), second recruitment of the same individual (n = 2), incomplete data (n = 6), and being the spouse of a patient (n = 4). We administered a cognitive functioning screening test13 to patients with difficulty summarizing key points of the study during the consent process to determine eligibility.
Research staff recruited patients in the primary care waiting areas and posted flyers in the clinics for patients to contact research staff. When conducting in-person recruitment, research staff approached all patients in the waiting areas to reduce risk of selection bias. Trained research staff conducted interviews in a private setting using a computer-assisted protocol to reduce errors, administration time, and missing data. For reliability purposes, we asked participants’ permission to audiotape the interviews; 1167 participants (91.0%) consented to audio recording. We ascertained the accuracy of the data through random review of 130 (11.1%) of the interviews by 2 of us (Y.E.L. and B.S.).
The interview included questions on demographic information, drug screening measures, and the criteria instruments. Demographic information included age, sex, racial and ethnic background, educational level, and marital and relationship status. Drug screening instruments included 6 measures: Two-Item Conjoint Screen (TICS)14; Cut down, Annoyed, Guilty, Eye-opener–Adapted to Include Drugs (CAGE-AID)15,16; Drug Use Disorders Identification Test (DUDIT)17; 10-item Drug Abuse Screening Test (DAST)18; Alcohol, Smoking and Substance Involvement Screening Test (ASSIST)5; and Single-Question Screening Test (SQST).19
Guided by previous reviews,9,12 we conducted the following item revisions. We streamlined questions in conjoint screening instruments (eg, TICS, CAGE-AID) that asked about both alcohol and drugs (eg, “…have you drunk or used drugs more than you meant to?”) to focus on drug use only. We revised lifetime (eg, CAGE-AID) and 3-month (eg, ASSIST) measures to assess drug use in the past 12 months. We consolidated measures (eg, ASSIST) that assessed each category of drug separately (eg, marijuana, cocaine) to assess all illicit and nonprescribed drugs as a combined question and revised all measures to elicit responses on a continuum of number of days in the past 12 months (ie, “how many days in the past 12 months …”).
We provided the participants the following instruction: “When the word ‘drug’ is used, it includes various types of drugs, such as marijuana, tranquilizers, barbiturates, cocaine, etc. ‘Drug’ does not refer to a medication used in the manner it was prescribed or recommended. Include only your use of marijuana if you are using it above and beyond what it was recommended for by a doctor or a state-licensed health care provider, or if you are using it recreationally.”
To examine test-retest reliability, 100 randomly selected participants repeated the drug screening measures between 4 and 10 days following initial administration.
We included 2 criteria: the Mini-International Neuropsychiatric Interview (MINI)20 for DUD and the Inventory of Drug Use Consequences (InDUC)21 for negative consequences of drug use (NCDU; considered independent of whether criteria were met for a DUD). The MINI is a short, structured, diagnostic interview designed to be administered by lay interviewers with limited training and has good concordance with the Composite International Diagnostic Interview22,23 and the Structured Clinical Interview for DSM-IV Disorders.20,24 The MINI is compatible with both the DSM-IV and the International Classifications of Diseases, 10th Revision (ICD-10).20,22 We established interrater reliability (κ >.95) with 2 interviewers (Y.E.L. and B.S.) who separately interviewed 20 individuals before actual data collection began. Based on the 1167 audiotaped interviews, we randomly selected 130 interviews to check for interrater reliability. Reliability at the item level was maintained at κ >.95, and perfect interrater reliability was maintained for DUD diagnosis.
The MINI assessed both illicit and prescribed medications in 8 major categories of substances: stimulants, cocaine, narcotics, hallucinogens, inhalants, marijuana, tranquilizers, and miscellaneous. Participants were shown a list of substances and read the following instructions: “I am going to show you and read to you a list of street drugs. The list also includes some medicines. Please only describe your use of a listed medicine if the medicine was not prescribed to you by a doctor or other qualified medical providers.” For example, the category of stimulants included amphetamines, “speed,” “crystal meth,” “crank,” “rush,” Dexedrine (dextroamphetamine), Ritalin (methylphenidate), and diet pills. Based on the MINI protocol, dextroamphetamine, methylphenidate, and diet pills were considered prescribed medications. Other prescribed medications were included under narcotics (morphine, methadone, Dilaudid [hydromorphone], Demerol [pethidine], codeine, Percodan [oxycodone], Darvon [dextropropoxyphene], Oxycontin [oxycodone], Vicodin [hydrocodone and paracetamol, hydrocodone and acetaminophen], and Percocet [oxycodone and paracetamol]), tranquilizers (Quaalude [methaqualone], Seconal [“reds”], Valium [diazepam], Xanax [alprazolam], Librium [chlordiazepoxide], Ativan [lorazepam], Dalmane [flurazepam], Halcion [triazolam], barbiturates, and Miltown [meprobamate]), and miscellaneous (steroids and nonprescription sleep or diet pills). The MINI then assessed symptoms of drug use based on DSM-IV and ICD-10 criteria. A drug abuse or dependence diagnosis based on the DSM-IV was considered positive for a DUD for this criterion.
We used the InDUC to assess the second criterion of NCDU in the past 12 months, using 37 questions representing 4 domains: impulse control (eg, “got into trouble because of my drug use”), social responsibility (eg, “missed days of work or school”), physical (eg, “sick or vomited”), and interpersonal (eg, “friendship or close relationship has been damaged”). The domain scales have excellent internal consistency reliability,25 good to excellent test-retest reliability, and interclass correlations ranging from 0.68 to 0.92.21 Internal consistency was excellent in the current sample (Cronbach α, 0.98). Participants who indicated they had experienced drug use consequences on any question were considered as having NCDU.
We conducted analyses to identify the best test characteristics of items from the pool of items drawn from existing instruments. Using the development sample data (n = 643), we relied on signal detection based on receiver operating characteristic (ROC) curve analytical techniques26 to select the best items and cut points that were most closely associated with the criteria. Signal detection using ROC curve analytical techniques has been found to be superior to logistic regression in identifying homogeneous subgroups of high-risk individuals.27 Recursive-partitioning methods, including signal detection, have greater power to detect interactions between predictors than do logistic regression analyses. Furthermore, the iteration in this method is based on an empirical construction of interactions among predictors as opposed to only interactions specified by investigators.26 In addition, because all items measure drug use and drug-related behaviors, there is a strong potential of multicollinearity among the items. Signal detection using ROC curve analytical techniques are particularly appropriate because multicollinearity has very little effect on the results.26,27
We used signal detection methods to identify predictor variables (ie, each item of the drug screening instruments) that divided the sample into subgroups of individuals with different likelihoods of meeting or not meeting the criterion (eg, DUD). For each predictive item, the software (ROC5; publicly available at http://web.stanford.edu/~yesavage/ROC.html) made a cut point at each level of the item (ie, each number of days); the sample was divided into 2 subgroups based on each cut point of the predictive item, and a measure of efficiency (sensitivity and specificity, with equal weight) of the predictive item in relation to the criterion was calculated and tested for statistical significance using a 2 × 2 χ2 test. The software calculated and compared the efficiency of all predictive items at all cut points. The software then identified the most efficient cut point of a predictive item (compared with all potential cut points of that item and all potential cut points of all other predictive items) and used this value to divide the sample into 2 subgroups. The software repeated the process within each of the 2 subgroups until there were too few people in a subgroup (<10 people) or until the most efficient cut point was nonsignificant.
We calculated statistical properties on the development sample, which included sensitivity, specificity, positive and negative likelihood ratios, efficiency, positive test rate, and area under the ROC curve. To validate the SoDU, we recalculated the statistical properties of the instrument to examine how well they held up in a new patient sample (validation sample; n = 640).
The overall sample was predominantly male, older, and non-Hispanic white (Table 1). Most participants had some post–high school education and were not married. In the last 12 months, 241 (18.8%) participants reported using illicit drugs or prescription medication for a nonmedical purpose (based on the MINI), and 189 (14.7%) reported 1 or more NCDU. Based on the MINI, 133 (10.4%) patients met DSM-IV criteria of a DUD for using illicit drugs and/or prescription medications (116 [9.0%] dependence and 17 [1.3%] abuse). Among the 133 patients who met criteria for a DUD, 76 patients met criteria only for illicit drug use, 9 met criteria only for prescription medication misuse, and 48 met criteria for both illicit drug use and prescription medication misuse. The most common DUDs were marijuana (51 [4.0%]), stimulants (50 [3.9%]), cocaine (42 [3.3%]), and narcotics (34 [2.7%]). Also based on the MINI, 164 (12.8%) participants met criteria for an alcohol use disorder (132 [10.3%] dependence and 32 [2.5%] abuse), and 191 (14.9%) met criteria for PTSD.
The development and validation samples did not differ significantly in terms of age, sex, educational level, marital status, and rates of DUD, NCDU, alcohol use disorder, and PTSD (Table 1). The development sample had a significantly lower percentage of non-Hispanic white individuals than the validation sample.
Signal detection analyses on data from the development sample identified the 2 items and their cut points for the SoDU (eFigure in the Supplement) that provided the best predictive performance for both the DUD and NCDU criteria. Specifically, if the response to item 1 (“How many days in the past 12 months have you used drugs other than alcohol?”) is 7 or more days, the SoDU is considered positive for DUD and NCDU and the second item is not needed. If the response to item 1 is 6 or fewer days, the second item asks, “How many days in the past 12 months have you used drugs more than you meant to?” A response to the second question of 2 or more days indicates a positive result (0 or 1 for a negative result). For DUD (Table 2), the SoDU was 100% sensitive and 93.73% specific; for NCDU, it was 93.18% sensitive and 96.03% specific.
We tested the SoDU on the validation sample (Table 2). In that sample, the SoDU was 92.31% sensitive and 92.87% specific for DUD and was 83.17% sensitive and 96.85% specific for NCDU. Test-retest reliability of the SoDU based on 100 participants retested after 1 week resulted in a κ of 0.9.
We examined the statistical properties of the SoDU for subgroups of patients (age, sex, racial or ethnic background, marital status, educational level, and PTSD status) on both the DUD and NCDU criteria (eTable in the Supplement). Sensitivity and specificity of the SoDU were comparable across all subgroups of patients on both criteria. Other criteria (eg, positive and negative likelihood ratios, rate of positive test) were also comparable among these subgroups of patients.
The SoDU is a practical screening instrument and merits consideration for routine use in primary care settings similar to the VA. The SoDU has sensitivity and specificity for DUD either comparable with or higher than other drug-screening instruments that have been examined in primary care settings, such as the ASSIST,28 DAST,19 and the SQST.19 The SoDU is also shorter and less time consuming than comparable drug-screening instruments (eg, ASSIST, DAST, DUDIT, and CAGE-AID).5,16- 18 The SoDU consists of just 2 questions, the second of which can be omitted if the patient meets the threshold on the first question (≥7 days of drug use in the past 12 months). Furthermore, the SoDU uses simple, easy-to-understand language (written at a fifth-grade level). Specifically, patients are asked about the drugs they used other than alcohol and about using drugs more than they meant to. These questions have face validity and are easily interpreted and answered by patients. In contrast, for example, the SQST asks patients whether they used an illegal drug or used a prescription medication for nonmedical reasons. The term illegal appeared to confuse many participants in our study. This confusion may contribute to the SQST’s relatively low specificity (eg, 74%).19
In addition, the SoDU is effective in detecting NCDU. All patients who have a positive screening result for DUD will also have a positive screening result for NCDU. However, screening for NCDU is more inclusive because some patients who do not meet criteria for a DUD diagnosis have NCDU and will have a positive screening result for this criterion. Physicians and primary care clinics need a screening instrument to detect subdiagnostic levels of drug use (eg, experimenting with drugs but not meeting DSM criteria) because these individuals are likely to benefit from early preventive intervention.9 The SoDU shows comparable or superior statistical properties in detecting NCDU in the primary care setting compared with instruments such as the DAST19 and SQST.19 It is also the only instrument that has been validated and shown comparable performance for multiple patient subgroups based on both the criteria of DUD and NCDU.
The large sample size of this study (N = 1283, with 133 individuals meeting DUD criteria) provides some confidence that the findings are stable rather than an artifact of sample irregularity. In contrast, the ASSIST was validated on a sample of 150 individuals,28 and the DAST19 and SQST19 were examined in 286 individuals. The cross-validation of the performance of the SoDU with a new sample of primary care patients (ie, validation sample) is another strength of the present study not found in the development of other measures (eg, ASSIST, SQST).5,19
The efficiency indices and positive test rates also support the SoDU’s promise as an instrument feasible for use in busy primary care settings. The SoDU has high efficiencies for both DUD and NCDU, with prevalence of less than 15% of DUD and NCDU in this sample. In addition, the SoDU’s positive test rate was 16% for both criteria in both samples, in which more than 10% of patients met criteria for a DUD and close to 15% met criteria for NCDU. In a population such as the studied clinics in which the prevalence of DUD and NCDU are 10% and 15%, respectively, this screening instrument would alert primary care clinic staff to follow up on only 16% of patients. Previous studies have not reported the efficiency and positive test rates of other measures, and therefore a comparison could not be made.
Another strength of the SoDU is the continuous response format (ie, number of days) as opposed to dichotomous (yes or no) responses. The continuous response format provides the option to adjust the threshold (number of days of drug usage) up or down to meet the needs of clinical settings when either false-positive or false-negative is preferred. Future research could validate the SoDU for such needs.
Primary care settings could easily integrate the SoDU into routine practice. Medical assistants or nurses could administer the SoDU, or patients could electronically complete the measure using a kiosk or tablet computer during check-in. For patients with a positive screening result, clinical personnel could engage the patients in a brief dialogue to decide whether second-stage screening, brief intervention, referral to behavioral specialists embedded in the primary care clinic, or referral to substance use treatment is necessary. Brief behavioral therapies with multiple sessions (rather than a single session29,30) may benefit patients with DUD. Future studies should examine which implementation strategies have optimal effect for using the SoDU in diverse primary care settings.
This study has several limitations. Biological data, such as toxicologic screening tests, were not available for corroboration of DUD. Valid biological data for a 12-month period for all substances, including prescription medications, would strengthen the criterion standards of this study. In addition, the criteria of DUD and NCDU relied on patients’ report rather than clinical interviews by trained professionals. Also, signal detection using ROC curve analytical techniques is useful for identifying at-risk individuals, but other methods are better for identifying predictors of risk and for testing hypotheses.27 Although the patient characteristics matched those of the population of the studied clinics (older, approximately 70% non-Hispanic white, and 4% women) as well as the general VA population31 and the performance of the SoDU did not differ across sex, age, and racial or ethnic groups, the majority of the patients in this sample were primarily older men and non-Hispanic white. Similarly, the study sample consisted of veterans who used the VA for primary care; therefore, the extent to which the results would generalize to a nonveteran sample is unclear. While the SoDU performed well with subgroups of patients in this sample, future validation studies should examine how well it performs in non-VA, diverse populations, including individuals who are cognitively impaired, have low literacy, and do not speak English, as well as in populations with greater representation of women and younger men.
The SoDU appears to be a suitable screening instrument for both DUD and NCDU in VA primary care settings. It is clear and brief, consisting of 1 or potentially 2 questions. Its sensitivity and specificity are excellent for both criteria and sustained for diverse veteran patient groups varying in terms of age, sex, race and ethnicity, marital status, educational level, and PTSD status among VA primary care patients. The SoDU would be a valuable addition to VA health care system screening tools owing to its excellent efficiency and low positive test rates. In the context of increased attention to psychosocial issues in VA primary care venues, it may be time to revisit the value of routinely screening for DUD.
Accepted for Publication: April 19, 2015.
Corresponding Author: Quyen Q. Tiet, PhD, Center for Innovation to Implementation and the National Center for PTSD, Dissemination and Training Division, VA Palo Alto Health Care System, 795 Willow Rd, Menlo Park, CA 94025 (firstname.lastname@example.org).
Published Online: June 15, 2015. doi:10.1001/jamainternmed.2015.2438.
Author Contributions: Drs Tiet and Leyva had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analyses.
Study concept and design: Tiet, Moos.
Acquisition, analysis, or interpretation of data: Tiet, Leyva, Frayne, Osterberg, Smith.
Drafting of the manuscript: Tiet, Leyva, Moos, Frayne.
Critical revision of the manuscript for important intellectual content: Tiet, Leyva, Moos, Osterberg, Smith.
Statistical analysis: Tiet, Leyva.
Obtained funding: Tiet.
Administrative, technical, or material support: Tiet, Leyva, Osterberg, Smith.
Study supervision: Tiet, Moos.
Conflict of Interest Disclosures: None reported.
Funding/Support: Dr Tiet reported receiving grant IIR 10-347 from the Department of Veterans Affairs, Health Services Research and Development.
Role of the Funder/Sponsor: The Department of Veterans Affairs, Health Services Research and Development, had no role in the design, methods, subject recruitment, data collections, analysis, and preparation of manuscript.
Disclaimer: The views expressed herein are those of the authors and do not necessarily reflect the views of the US Department of Veterans Affairs.
Additional Contributions: Katharine A. Bradley, MD, MPH, VA Puget Sound Health Care System; Tyson H. Holmes, PhD, Stanford University School of Medicine; Eve Carlson, PhD, National Center for PTSD, Dissemination and Training Division, VA Palo Alto Health Care System; and Janet Schmidt, PhD, National Center for PTSD, Dissemination and Training Division, VA Palo Alto Health Care System, provided consultations to improve the conceptualization of the publication of the study. They were not compensated for their contributions. John Chardos, MD, Primary Care Clinic, VA Palo Alto Health Care System; Allen Hullinger, MHPA, Primary Care Clinic, VA Palo Alto Health Care System; and Julie Bergman, MSN, Primary Care Clinic, VA Palo Alto Health Care System, helped facilitate the acquisition of data. They were not compensated for their contributions. We also thank the staff and patients at the primary care clinics who participated in this study