The Quality of Highly Regarded Adolescent Substance Abuse Treatment Programs: Results of an In-depth National Survey

Background  Drug and alcohol abuse and dependence are the most prevalent causes of adolescent morbidity and mortality in the United States. Effective, accessible treatment for adolescents with substance abuse problems is urgently needed.

Objective  To conduct the first systematic evaluation of the quality of highly regarded adolescent substance abuse treatment programs in the United States.

Methods  An advisory panel of 22 experts defined 9 key elements of effective treatment for adolescent substance abuse based on a review of the literature. In-depth telephone and written surveys were conducted with 144 highly regarded adolescent substance abuse treatment programs identified by panel members and by public and private agencies. There was a 100% response rate to the initial interviews, and a 65% response rate to the follow-up surveys. The open-ended survey responses were coded by defining 5 components deemed to be crucial in addressing each of the 9 key elements, and quality scores were calculated overall and for each of the 9 key elements.

Results  Out of a possible total score of 45, the mean score was 23.8 and the median was 23. Top-quartile programs were not more likely to be accredited. The majority of programs scored at least 4 of a possible 5 on only 1 of the 9 key elements (qualified staff). The elements with the poorest-quality performance were assessment and treatment matching, engaging and retaining teens in treatment, gender and cultural competence, and treatment outcomes.

Conclusions  Most of the highly regarded programs we surveyed are not adequately addressing the key elements of effective adolescent substance abuse treatment. Expanded use of standardized assessment instruments, improved ability to engage and retain youths, greater attention to gender and cultural competence, and greater investment in scientific evaluation of treatment outcomes are among the most critical needs. Expanding awareness of effective elements in treating adolescents will lead the way to program improvement.

Drug and alcohol abuse and dependence are the most prevalent causes of adolescent morbidity and mortality in the United States. Consequences of adolescent substance abuse can include academic failure, social and familial disruption, overdose, automobile accidents, increased risk for human immunodeficiency virus infection and sexually transmitted diseases, and arrest and incarceration.^1-4

Effective, accessible drug treatment programs for adolescents are urgently needed. Only 10% of the estimated 1.4 million adolescents (aged 12-17 years) with an illicit drug problem are receiving treatment, compared with 1 in 5 adults.⁵ Many substance abuse treatment programs that were initially designed for adults fail to address the needs of adolescents. Compared with adults, adolescents have higher rates of dual diagnosis,⁶ different developmental needs,⁷ and higher rates of binge and opportunistic use.⁸ Assessing an adolescent's treatment needs poses significant challenges. The Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) criteria for substance abuse and dependence, which were developed for adults, have significant limitations when applied to adolescents.⁹

Research on the effectiveness of treatment for adolescents is still a new field, with relatively few scientifically rigorous studies published to date.^10,11 Only 2 of 38 Federal Drug Treatment Improvement Protocols have addressed adolescents.^12,13 We conducted the first systematic evaluation of the quality of highly regarded adolescent substance abuse treatment programs in the United States. This assessment was initiated to develop a guide that would define program quality and help parents, physicians, juvenile court judges, and school counselors make informed treatment choices. We conducted interviews with 144 programs that were recommended as exemplary by experts in the field and by public and private agencies. Using the results of these interviews, our study describes whether these programs include those key elements identified by experts as effective adolescent substance abuse treatment. We also describe the program characteristics that were and were not associated with the highest-quality programs.

We reviewed the literature on adolescent substance abuse treatment and assembled an advisory panel of 22 experts, including 10 leading researchers, 9 practitioners from nationally recognized treatment programs, and 3 senior federal policymakers. Structured telephone interviews averaging 45 minutes were conducted with the 14 panel members who were currently conducting or funding research. Based on these interviews, a report identifying and describing 9 key elements was sent to all 22 panel members prior to their first meeting on June 1, 2001. At the meeting, consensus on the key elements was reached by this larger body of experts. On April 10, 2002, a smaller working group of the panel refined the key elements (Figure 1), and an explanatory justification, based on research and clinical experience, was generated for each element.

Directors of all 50 US states' alcohol and drug abuse agencies (as well as the District of Columbia) were mailed a request to identify 5 adolescent treatment programs in their jurisdictions that they considered exemplary. There was a 92% response rate and 65% (33 states) provided a list of up to 6 programs, which yielded a total of 126 recommended programs. Fourteen state directors either provided lists of all programs without identifying exemplary ones (stating there were no programs in their states which they would describe as exemplary), or said they had no criteria for evaluating the effectiveness of adolescent programs. In addition, expert panel members recommended a total of 20 adolescent programs to which they would refer family members or friends. Twenty-two national organizations and federal agencies were also contacted, including the American Medical Association, the American Academy of Pediatrics, and the National Institute on Drug Abuse, which yielded 59 recommended programs. After accounting for duplicate recommendations and excluding those programs that were not adolescent-only substance abuse treatment programs, this process identified a total of 144 highly regarded programs nationally.¹⁴

A 2-part initial survey instrument was created, consisting of questions derived from the literature review and panel discussion. This survey was tested with 5 programs to assess time required and ease of use. The first part was a written questionnaire that was faxed to each program to collect basic information on services offered, client demographics, and costs. The surveys were returned by mail, telephone, or fax. This was followed by a structured, recorded telephone interview with representatives (primarily the executive, program, or clinical directors) from each program that included detailed open-ended questions about treatment services and program practices as they related to the 9 key elements. The initial fax survey and telephone interviews were conducted between June 2001 and February 2002, with a 100% response rate.

Responses from these interviews were compiled into a draft profile for each program. Approximately 10 to 12 months later, the programs were faxed this draft profile along with additional open-ended questions to clarify their previous answers, obtain further information on key elements that were not covered in sufficient depth in the original telephone interviews, and to review the draft profile for accuracy. These follow-up surveys had a 65% response rate (107 responders and 37 nonresponders). The initial and follow-up survey results were combined to produce a final profile of each program. For nonresponders, a final letter was both mailed and faxed with another copy of the profile and the follow-up questions.

The open-ended survey responses regarding each of the 9 key elements were coded by defining 5 components, in the form of yes-or-no questions, deemed to be crucial in addressing each element (the 45 component questions are available from the authors). For each of the 144 programs, a researcher reviewed survey responses for all 45 components to determine whether or not the program adequately fulfilled the requirements of the components. For responses that were unclear, a second researcher reviewed the response and coding was determined by consensus among the reviewers. Thirty-two of the component questions were collected in the initial telephone interviews and 13 collected in the follow-up fax survey. For each program, scores were calculated as the number of components fulfilled in total (out of a possible 45) and for each of the 9 key elements (out of a possible 5). An alternative summary score based only on responses to the initial telephone interview was also calculated. Program characteristics that were measured included geographic region, program age, types of services offered, and accreditation.

Data analysis was conducted using Excel (Microsoft Corporation, Redmond, Wash) and Stata 6 (Stata Corporation, College Station, Tex). Results are reported as percentages of program responses. Differences between the percentage responses by program subgroups were compared using the χ² test with PEPI version 3 (Stone Mountain, Ga). The reliability of the summary score was tested using Cronbachs α with standardized items.¹⁵

The 144 highly regarded adolescent-only substance abuse treatment programs were broadly distributed in region, program age, setting, approach, and accreditation status (Table 1). Out of a possible total score of 45 components, the mean score was 23.8 and the median was 23. The reliability of the overall scale was 0.79, as measured by Cronbachs α with standardized items. The highest score achieved by any program was 39. Only 19 programs (13%) satisfied more than two thirds of the 45 components and 64 programs (44%) fulfilled fewer than half the components (Figure 2). Results were generally consistent using an alternative summary score based only on responses to the initial telephone interview; 55 programs (38%) fulfilled fewer than half the components, and the composition of programs whose scores were in the top quartile changed by only 7 programs.

There were some differences between the characteristics of programs whose scores were in the top quartile and other programs (Table 1). Top-quartile programs were more likely to be 20 years old or more, and were somewhat less likely to be less than 10 years old. Top-quartile programs were also more likely to offer Multidimensional Family Therapy and somewhat more likely to employ the Therapeutic Community approach. Top-quartile programs were not more likely to be accredited. There were no significant differences between bottom-quartile programs and the other programs.

The majority of programs did not perform well on most of the 9 key elements (Figure 3). On only 1 element—qualified staff—did the majority of programs score at least 4 of a possible 5. None of the 144 programs scored at least 4 on all 9 key elements. Of a possible total score of 5 on each element, the mean (median) scores on each component were: assessment and treatment matching, 2.7 (3); comprehensive, integrated treatment approach, 2.7 (3); family involvement in treatment, 3.0 (3); developmentally appropriate program, 3.3 (3); engaging and retaining teens in treatment, 2.5 (3); qualified staff, 3.5 (4); gender and cultural competence, 1.8 (2); continuing care, 3.1 (3); and treatment outcomes, 1.2 (1).

Accredited programs were somewhat more likely to score at least 4 on assessment and treatment matching but were similar to unaccredited programs on most other elements (Table 2). For engaging and retaining teens in treatment, accredited programs were less likely to score at least 4 than programs without accreditation, although the difference was not significant.

The elements with the poorest overall performance were assessment and treatment matching, engaging and retaining teens in treatment, gender and cultural competence, and treatment outcomes. More than 50 different instruments were being used by programs to screen and assess clients. Less than half (45%) of the programs reported using a standardized substance abuse instrument or a clinical interview, 15% reported using a standardized mental health instrument, and 10% reported using both a standardized substance abuse instrument or clinical interview and a standardized mental health instrument.

For engaging and retaining teens in treatment, 39% of the programs reported an emphasis on building a therapeutic alliance between staff and clients; 41% reported utilizing motivational enhancement techniques, such as motivational interviewing; and 48% reported incorporating positive reinforcements to provide incentives for client participation.

For gender and cultural competence, 35% of programs reported providing content that differs for male and female patients; 24% of programs were designed to meet the needs of minorities; and 12% of programs were designed to meet the needs of gay and lesbian adolescents.

For treatment outcomes, 44% of the programs reported not collecting any data related to client outcomes and 35% reported analyzing their own internally gathered data. Fewer than 10% of programs have been the subject of a scientifically rigorous follow-up of the program's effect on client outcomes.

Most of the 144 highly regarded programs we surveyed are not addressing the key elements of effective adolescent substance abuse treatment. More than 40% of the reviewed programs fulfilled fewer than half of the 45 components that make up the key elements, and only 3% of programs fulfilled four fifths of these components. However, high scores were achieved on individual key elements by several programs in our sample, suggesting that implementing the key elements in practice is already within the reach of existing programs.

Older programs appear to have higher quality scores. However, few other program characteristics are associated with high quality. Quality scores were predominantly based on answers to questions that measured processes of care, which can be more sensitive than outcome measures when comparing the quality of individual health care programs or providers.¹⁶ Programs that offer the Multidimensional Family Therapy or Therapeutic Community approaches have highly structured treatment processes and higher quality scores. Recent studies^17,18 that used aggregate data from several programs concluded that no 1 particular treatment modality is associated with superior outcome. Accreditation, which focuses on client safety and dignity, does not necessarily ensure higher quality performance.

The generally unimpressive results for most of the key elements are noteworthy because we deliberately sought out programs that were well regarded. On only 1 key element did the majority of programs fulfill 4 or more components. The low scores observed for assessment and treatment matching were especially disappointing, because proper assessment provides a road map for developing an effective treatment plan tailored to adolescents' specific needs. Programs often reported that they relied on questionnaires developed in-house that may not have been tested for reliability and validity. We recommend that the National Association of State Alcohol and Drug Abuse Directors encourages all directors to establish formal adolescent assessment protocols that will require every program that receives state funds to use a standardized assessment instrument, such as the Substance Abuse Subtle Screening Inventory, Personal Experience Screening Questionnaire, Comprehensive Addiction Severity Index for Adolescents, or the Global Assessment of Individual Needs.¹²

The overall poor performance on engaging and retaining teens in treatment is also disturbing. Denial among adolescents about their drug problems is high, and few seek treatment on their own.⁷ A positive therapeutic alliance between the adolescent, the counselor, and staff members is an essential aspect of treatment.¹⁹ Programs need creative techniques to engage and retain adolescents in treatment by making activities relevant to their concerns; treatment for adolescents should have tangible, concrete stages of process and outcomes if teenagers are to remain engaged.¹⁴

Poor performance on gender and cultural competence is a concern for several reasons. Recent research points to significant differences between the characteristics and treatment needs of male and female adolescent drug users.^20-22 Same-sex group sessions provide an opportunity to focus on issues that might be difficult to discuss in mixed groups. While only limited research has been directed toward cultural differences,²³ our expert panel believes that a lack of understanding of these differences may limit the ability to treat minority youth.²⁴ Sensitivity to sex and cultural differences helps develop a successful therapeutic alliance between the teenager and the counselor, which facilitates behavior change. This alliance is especially important for gay and lesbian adolescents and adolescents with mixed racial and cultural identities, who might not be willing or able to address key aspects of their identity.

Poor performance in treatment outcomes is less surprising. Adolescent drug treatment research is a relatively new field. Given the high cost of conducting rigorous outcome evaluations, few of even the best programs in the country can be expected to undertake such studies on their own.

Our study has several limitations. First, because the sample was not designed to be representative of the field of adolescent treatment programs as a whole, the results do not allow for direct inferences about programs nationwide. Second, the program data were self-reported, raising the possibility that for some programs actual services provided were not consistent with the survey responses. However, programs would be expected to overreport, rather than underreport, the quality of services provided. Third, 35% of programs sampled did not respond to the follow-up portion of the survey, although results that included the follow-up responses were consistent with results based only on responses to the initial survey. Fourth, while the 9 key elements were developed through expert consensus and constitute a significant conceptual advance for the field of adolescent substance abuse treatment, future research to validate these constructs with more complete responses may suggest the need for further refinement or modification. Moreover, in the summary score, each of the 45 components was weighted equally. Although the reliability of the summary score was high, it might be increased if the key elements and their components could be weighed by importance. Because the level and quality of information that would be required to do this does not yet exist, we report results by key element as well as by summary score results.

It is critical that more adolescent substance abuse treatment programs adopt standardized assessment tools to ensure that adolescents are evaluated and matched properly. Programs must incorporate proven techniques that will initially engage teenagers in order to sufficiently build therapeutic alliances. We believe these alliances will motivate them to remain in treatment. It is also vital that the field's leaders build awareness of the importance of gender and culture issues and generate support for research that can build a stronger evidence base on outcomes to informed practice. From the broader societal perspective, rigorous research on treatment outcomes can help target scarce treatment dollars more effectively.

Parents, physicians, juvenile court judges, and school counselors face difficulties in referring adolescents to appropriate treatment programs. Accreditation has been relied upon by some to determine appropriateness for treatment for adolescents. However, we have shown that accreditation is not a useful measure of quality. Adolescent treatment program quality needs to be measured directly, and the 9 key elements featured in this study can help fill this role. They represent a distillation of the latest thinking and research on the ingredients that contribute to successful adolescent substance abuse treatment. They constitute a new, potentially powerful tool for the development of quality measures by external agencies. Our findings establish an important point of reference for measuring progress in the field in the years ahead.

Article

There are still relatively few drug treatment programs specifically designed for adolescents and very little rigorous research conducted that compares the effectiveness of different types of adolescent treatment. This is the first comprehensive assessment of a large number of highly regarded adolescent treatment programs across the United States.

This study identifies 9 key elements of effectiveness that provide a framework for evaluating the quality of individual programs. This framework will encourage programs to improve services as well as stimulate research in the field.

Correspondence: Rosalind Brannigan, MPH, Drug Strategies, 1755 Massachusetts Ave NW, Washington, DC 20036-2103 (dspolicy@aol.com).

Accepted for publication April 19, 2004.

This research was supported by a grant from the Robert Wood Johnson Foundation. The funding agreement ensured the authors' independence in designing the study, interpreting the data, and writing and publishing the report.

We thank Jeremy Gans, John M. Walsh, MPP, and Jacqueline Yannacci, MPP, for their expert assistance with data analysis and manuscript preparation.