Effectiveness of Training Therapists to Deliver An Individualized Mental Health Intervention for Children With ASD in Publicly Funded Mental Health Services: A Cluster Randomized Clinical Trial

Key Points

Question  Does training community mental health therapists to deliver an intervention influence challenging behaviors in children with autism spectrum disorder?

Findings  In a cluster randomized community effectiveness trial, 202 children with autism spectrum disorder whose therapists received training and ongoing consultation in evidence-based strategies showed greater reductions in challenging behaviors across 18 months compared with children whose therapists delivered usual care. Observer ratings of therapist fidelity mediated the intervention effects.

Meaning  This study supports the effectiveness of an intervention designed specifically for the targeted community service context and highlights the need to develop and test implementation strategies targeting high-fidelity therapist delivery.

Importance  Publicly funded mental health services play an important role in addressing co-occurring mental health problems in children with autism spectrum disorder (ASD); however, therapists report lacking training to effectively serve this complex population.

Objective  To test the effectiveness of training community therapists in An Individualized Mental Health Intervention for ASD (AIM HI) on challenging behaviors across 18 months among children with ASD and identify moderators and mediators of any intervention effects.

Design, Setting, and Participants  Cluster randomized trial conducted in 29 publicly funded outpatient and school-based mental health programs in southern California from 2012 to 2017. Programs were randomized to receive immediate AIM HI training or provide usual care followed by receipt of AIM HI training. Therapist participants were recruited from enrolled programs, and child participants were recruited from participant therapists’ caseloads. Data were analyzed from 202 children with ASD who were aged 5 to 13 years.

Interventions  The AIM HI protocol is a package of parent-mediated and child-focused strategies aimed to reduce challenging behaviors in children with ASD who are 5 to 13 years old. It was designed for delivery in publicly funded mental health services based on a systematic assessment of therapist training needs and child clinical needs. The therapist training and consultation process takes approximately 6 months and includes an introductory workshop, 11 structured consultation meetings as the therapist delivers AIM HI with a current client, and case-specific performance feedback from trainers.

Main Outcomes and Measures  Child participants were assessed for challenging behaviors using the Eyberg Child Behavior Inventory (ECBI) and Social Skills Improvement System (SSIS) Competing Problem Behaviors scales based on parent report at baseline and at 6-month intervals for 18 months. Outcomes were analyzed using intent-to-treat models.

Results  In total, 202 children with ASD (mean [SD] age, 9.1 [2.4] years; 170 [84.2%] male; 121 [59.9%] Latinx) were eligible, enrolled, and included in the analyses. Statistically significant group by time interactions for the ECBI Intensity (B = −0.38; P = .02) and ECBI Problem (B = −1.00; P = .005) scales were observed, with significantly larger decreases in ECBI Intensity scores in the AIM HI group (B = −1.36; P < .001) relative to the usual care group (B = −0.98; P < .001) and a significantly larger decrease in ECBI Problem scores in the AIM HI group (B = −1.22; P < .001) relative to the usual care group (B = −0.20; P = .29). Therapist fidelity moderated these intervention effects.

Conclusions and Relevance  The present findings support the effectiveness of training therapists to deliver the AIM HI model to children with ASD receiving publicly funded mental health services.

Trial Registration  ClinicalTrials.gov identifier: NCT02416323

The number of individuals with autism spectrum disorder (ASD) has increased substantially during the past 2 decades, with an estimated 1 in 59 school-aged children meeting criteria for ASD.¹ Community service systems, including publicly funded mental health (MH) services, are faced with addressing the complex needs of these children. The federal Interagency Autism Coordinating Committee² has called for research to scale up evidence-based interventions in community service settings and to improve psychosocial interventions for core symptoms and co-occurring conditions in ASD. One related direction of study is community effectiveness research for school-age children with ASD, which is important given the paucity of research for this group.

Publicly funded MH services play an important role in caring for school-age children with ASD³ because approximately 70% of these children meet criteria for at least 1 non-ASD co-occurring psychiatric disorder.^4-7 Children with ASD represent 10% to 14% of children in the psychiatrically referred population,⁸ and MH therapists indicate that children with ASD represent a significant proportion of their caseloads.⁹ The primary presenting problems in MH services are challenging behaviors⁹ and more than 80% of children in MH services meet criteria for at least 1 disorder associated with challenging behaviors (eg, attention-deficit/hyperactivity disorder or oppositional defiant disorder).¹⁰ Furthermore, challenging behaviors are often the expressed symptoms of ASD and non-ASD psychiatric disorders.^11,12 There are significant concerns about the lack of MH therapist training to work with the ASD population^9,13 and broader concerns about the overall quality of MH services for children with ASD.¹⁴

Although a number of well-established, evidence-based behavioral intervention strategies have been shown to be efficacious in reducing behavior problems in children with ASD,^15-18 there is limited training and delivery in MH settings.¹⁴ There have also been no systematic, large-scale studies examining the effectiveness of such strategies within publicly funded MH services.¹⁸ The most relevant research has shown the efficacy of adapted anxiety interventions^19,20 when delivered in research settings and the effectiveness of interventions addressing social communication skills within the context of early intervention services^21-23 and in public schools.^24-26 Furthermore, understanding child characteristics associated with responses to interventions delivered in routine care is critical in ASD given the heterogeneity of ASD symptomology and variable responses to the intervention.²⁷ Finally, there is significant variability in community therapist fidelity to ASD interventions, and therapist fidelity is associated with child outcomes.^28,29

Quiz Ref IDIn response to the need for an intervention protocol that could be delivered in publicly funded MH services for children with ASD presenting with challenging behaviors, An Individualized Mental Health Intervention for ASD (AIM HI)³⁰ clinical intervention and corresponding therapist training protocol were developed. The AIM HI protocol was developed based on a systematic needs assessment of the clinical needs of children with ASD receiving MH services and the training needs of MH therapists.⁹ The purpose of the current cluster randomized controlled community effectiveness trial was to build on the promising findings from a pilot study supporting the feasibility of training MH therapists and positive changes in child behaviors^31-33 by assessing the effectiveness of training therapists in AIM HI on changes in child behaviors across 18 months and examining potential moderators (child characteristics) and mediators (therapist fidelity) of intervention effects.

A cluster randomized waitlist control design was used to examine the effectiveness of AIM HI training on child challenging behaviors (Figure 1). Publicly funded outpatient clinics and school-based MH programs in San Diego and Los Angeles counties were invited to enroll in the study (trial protocol in Supplement 1). Programs were eligible if they provided psychotherapy services to children, including those with ASD. After programs enrolled in the study, they were randomized to immediate (wave 1) AIM HI training or to a waitlist usual care (UC) condition/delayed (wave 2) AIM HI training condition. After programs in the waitlist control completed the UC condition, they were invited to participate in the wave 2 AIM HI condition if they had new children with ASD served in their program. Program randomization was conducted by an independent statistical investigator (S.R.) and occurred at the program level to prevent contamination at the therapist level. Randomization was stratified by agency size using randomly permuted blocks according to a computer-generated assignment sequence prepared in advance by the study statistician (S.R.). During the first year of recruitment (2012), programs were randomized using a 1:1 ratio of UC/wave 2 AIM HI to wave 1 AIM HI. Based on an imbalance in the mean number of participants enrolled in UC/wave 2 vs AIM HI wave 1 during the first year, we switched to a restricted randomization approach^34,35 for subsequent years. Specifically, the randomization ratio was recalibrated from 1:1 to 2:1, that is, 2 programs randomized to UC/wave 2 for every program randomized to wave 1 to achieve better balance in the sample size by condition. Following randomization, therapist/child dyads were recruited for participation. Once the UC waitlist control programs moved over to wave 2, a new round of therapist/child recruitment was conducted. Eight of the UC therapists participated in wave 2, and no child participated in both conditions. The University of California, San Diego Institutional Review Board approved this study, and informed written consent was obtained from all participants or their guardians.

Therapists were recruited from enrolled programs, and children were recruited from the caseloads of participating therapists. See the Table for a summary of program, therapist, and child/family characteristics. Following the enrollment of 29 programs and randomization, therapists were recruited from within enrolled programs. Therapists were eligible if they (1) were a trainee or staff providing psychotherapy services in an enrolled program, (2) anticipated working in the program for at least the next 7 months, and (3) had a least 1 eligible child on their current caseload. Of 172 therapist participants, most (85.5%) were women, and the therapists had a mean (SD) age of 34.0 (8.1) years. The mean (SD) number of years of experience was 5.43 (4.77) and ranged from 0 to 24 years.

After initial enrollment, therapists referred children for an eligibility assessment conducted by study staff. Children were eligible if they met the following 5 criteria: (1) were aged 5 to 13 years old at the time of recruitment; (2) spoke English or Spanish as their primary language; (3) had an existing ASD diagnosis on record, and (4) exhibited clinically significant ASD symptoms on at least 1 of 2 standardized ASD diagnostic measures: the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2)³⁶ or the Social Responsiveness Scale, Second Edition (SRS-2)³⁷; and (5) exhibited at a least 1 challenging behavior. Of 202 child participants, most (170 [84.2%]) were male, 121 (59.9%) were Latinx, and the children had a mean (SD) age of 9.1 (2.4) years. In total, 190 children (94%) met criteria on both the ADOS-2 and SRS-2, and 12 children (6%) met criteria based on the SRS-2 and had previously received a diagnosis of ASD from a community health care professional specializing in ASD diagnostic assessment.

The AIM HI protocol is a package of parent and child directed strategies aimed to reduce challenging behaviors in children with ASD aged 5 to 13 years and designed specifically for delivery in publicly funded MH services. The AIM HI protocol contains well-established behavioral intervention strategies, which include collaborating with the child’s parent to identify patterns in the child’s challenging behaviors and actively teaching (through modeling, behavioral rehearsal, reinforcement, and in-home practice) positive alternative skills for children and complementary strategies for parents to help prevent the occurrence of challenging behaviors and to promote the child’s use of alternative skills. The AIM HI protocol also includes strategies to adapt psychotherapy structure for this population to facilitate engagement and skill building (eg, maximizing predictability). A minimum of 13 sessions is required, and it typically takes approximately 6 months to complete the AIM HI protocol steps.³¹

Quiz Ref IDThe AIM HI therapist training model includes active, direct, and explicit instructional methods applied through the following 4 components: (1) an introductory training workshop, (2) 11 (9 group and 2 individual) structured consultation/feedback meetings (1 hour in length) with an AIM HI trainer who is supervised by the AIM HI developer during a 6-month period with didactic instruction and case-specific feedback, (3) performance feedback from trainers based on review of the therapists’ therapy session recordings with participating clients, and (4) access to a therapist manual, printed and electronic forms, and resource website.

Therapist participants in the waitlist control condition delivered routine care to participating children. The therapists submitted recordings of their therapy sessions to the research team for the 6-month observation period and completed all study measures; no AIM HI training or clinical feedback was provided to therapists in the control condition.

Baseline assessments were conducted by the research team (which included among others C.C. and M.B.-E.) at each participating program site. In addition to the ADOS-2 and SRS-2 eligibility assessments, the baseline assessment included a measure of cognitive functioning (the Wechsler Abbreviated Scale of Intelligence-Second Edition³⁸ or the Differential Ability Scales-II³⁹ based on the child’s age), parent report of challenging behaviors (the Eyberg Child Behavior Inventory [ECBI]⁴⁰ and the competing behavior scale of the Social Skills Improvement System [SSIS]⁴¹), and parent report of sociodemographic characteristics. The ECBI and SSIS competing problem behavior were the preidentified primary outcome measures, with higher scores indicating greater challenging behaviors. Participants received a $40 gift card for their time for completion of the baseline assessment. Research staff contacted parent participants by telephone at 6-, 12-, and 18-month follow-up visits to administer the follow-up measures, including the ECBI and SSIS; caregivers received a $25 gift card after each completed assessment for their time (up to $115 across all time points). Caregivers were contacted for each follow-up assessment regardless of whether their child was still in treatment with a study therapist; the proportion of children in treatment across conditions was 73.3%, 25.9%, and 17.8% at 6, 12, and 18 months, respectively. There was no difference in the likelihood that the child would remain in treatment by study condition.

Trained coders naive to study condition coded in-session therapist and child behaviors observed via video-recorded sessions collected during the 6 months training/consultation or usual care observation period. Recordings were randomly selected from 2-month windows representing the beginning (months 1-2), middle (months 3-4), and end (months 5-6), and the mean of the scores calculated across coded sessions for the 6-month period. The mean (SD) number of coded sessions per child was 7.5 (3.3) sessions. Therapist fidelity scores included 2 adherence composites reflecting the extent to which the therapist was observed to use strategies expected in most AIM HI sessions (1 comprising AIM HI strategies delivered to caregivers, and 1 comprising strategies directed to children) and 4 effectiveness ratings (session structure, continuity across sessions, pursuit of child skill, and pursuit of caregiver skill).

Given the clustered nature of the data, multilevel modeling with random intercepts and slopes was used as the primary statistical analysis procedure: repeated measures (level 1) nested within children/therapists (level 2) nested within programs (level 3). To test for intervention effects across time, a group (usual care vs AIM HI) by time (baseline and 6, 12, and 18 months) cross-level interaction was assessed. If a statistically significant (P < .05) interaction term was evident, simple slopes analyses were conducted. To test for trajectory differences between groups on the primary outcomes, we conducted 2 sets of analyses. First, we compared usual care to AIM HI wave 1 only. Then, we compared usual care to the full AIM HI sample (wave 1 and wave 2 combined). This later, larger data set was used for moderation and mediation analyses. To test for moderators (eg, child characteristics) of the group by time interaction, the group by time by moderator cross-level interactions were statistically evaluated. To determine if therapist fidelity (adherence and effectiveness) scores mediated the intervention effect for outcomes at 6, 12, or 18 months, the longitudinal mediation approach of Cole and Maxwell⁴² was used (Figure 2). The asymmetric confidence interval (CI) was used to formally test for mediation⁴³; CIs that did not contain the value 0 were considered statistically significant mediated effects. These models also included unstructured autoregressive effects for the target outcomes. All analyses used an intent-to-treat approach and multilevel modeling analyses using the maximum likelihood robust estimation procedure implemented in the Mplus software.⁴⁴ The estimation procedure adjusted for missing data and non-normality of the outcome variables. Preceding the formal multilevel model analyses, we identified variables (eg, autism severity) that were significantly associated with missing data patterns. These variables were subsequently entered into the multilevel models to account for missing data under the missing at random assumption.⁴⁵ The power analysis program RMASS2⁴⁶ was used to estimate the sample size necessary to find statistically significant (α = .05) intervention effects given the prospective design. These power analyses assumed an effect size (d) of 0.40 at the last time point and an overall attrition rate of 20%. Moreover, clustering at the program level was accounted for with the design effect, with an mean number of therapists per program assumed to be 7 and an intraclass correlation coefficient of 0.05. Given these assumptions, 206 programs/therapists nested with 29 clinics/therapists were needed at the beginning of the study to detect the predicted intervention effect with 80% power.

Intraclass correlation coefficients (ICCs) showed significant variability at the child/therapist level for the ECBI Intensity (ICC = 0.69), ECBI Problem (ICC = 0.67), and the SSIS Competing Problem Behaviors (ICC = 0.69) scales. However, minimal variability was evident at the program level for all outcomes (0.01-0.02). Significantly higher autism severity scores at baseline for those that had missing data vs not (mean [SD], 82.49 [10.85] vs 79.06 [11.22]) and in the usual care (43.5%) relative to the AIM HI group (25%) were found (all P < .05). Moreover, statistically significant baseline differences for the outcomes were found for 2 study variables; higher autism severity scores were associated with higher scores on both the ECBI Intensity (B = 0.17, P = .003) and Problem (B = 0.16, P = .04) scales at baseline. Moreover, significant differences in Problem scale scores were found as a function of primary service setting, with higher scores in clinic settings relative to school settings (B = −3.73, P = .001). These variables were entered as covariates variables in multilevel modeling.

In the initial analysis comparing usual care to wave 1 only, there was a statistically significant group by time interaction for the Intensity (B = −0.36, P = .02, d = −0.10, 95% CI for d = −0.19 to −0.02) and Problem (B = −0.98, P = .009, d = −0.28, 95% CI for d = −0.49 to −0.07) scales. Simple slope analyses found a significantly steeper decline in ECBI Intensity scores in the AIM HI group (B = −1.32, P < .001) relative to the usual care group (B = −0.96, P < .001) and a significantly steeper decline in ECBI Problem scores in the AIM HI group (B = −1.21, P < .001) relative to the usual care group (B = −0.14, P = .78). There was no statistically significant difference in the group by time interaction for the SSIS Competing Problem Behaviors scale (P > .05).

Quiz Ref IDIn the subsequent analyses using both wave 1 and wave 2 data (Figure 3 and Figure 4), there was a statistically significant group by time interaction for the Intensity (B = −0.38, P = .02, d = −0.13, 95% CI for d = −0.24 to −0.03) and Problem (B = −1.00, P = .005, d = −0.36, 95% CI for d = −0.61 to −0.11) scales. Simple slope analyses found a significantly larger decrease in ECBI Intensity scores in the AIM HI group (B = −1.36, P < .001) relative to the usual care group (B = −0.98, P < .001) and a significantly larger decrease in ECBI Problem scores in the AIM HI group (B = −1.22, P < .001) relative to the usual care group (B = −0.20, P = .29). There was no statistically significant difference in the group by time interaction for the SSIS Competing Problem Behaviors scale.

Quiz Ref IDChild characteristics (age, sex, race/ethnicity, household income, IQ, and ASD severity) did not significantly moderate outcome trajectories on the ECBI scales (all P > .05, all R² < .01). Treatment continuity (MacKinnon 95% asymmetric CI, −5.34 to −0.67) and session structure (MacKinnon 95% asymmetric CI, −4.75 to −0.15) both significantly mediated the intervention effects for ECBI Problem behavior at 6 months. Specifically, children in the intervention group (relative to the UC group) had significantly higher continuity across sessions (B = 1.79, P < .001, R² = 0.59) and session structure (B = 2.18, P < .001, R² = 0.56), which in turn was associated with lower ECBI Problem scores at 6 months (for continuity: B = −1.54, β = −0.13 P = .04; for session structure: B = −1.35, β = −0.16, P = .01). No additional significant mediation effects were found for ECBI scores.

In this cluster randomized trial, we tested the effectiveness of training therapists who were providing publicly funded MH services to deliver a package of evidence-based strategies for ASD. Our results showed that children whose therapists received AIM HI training and ongoing consultation showed greater decreases in the frequency (ECBI Intensity scale) and severity (ECBI Problem scale) of parent-reported challenging behaviors across 18 months compared with children whose therapists delivered usual care. Effect sizes estimates indicated small to medium differences between groups. Child characteristics showed no moderating effects. Observer-rated indicators of therapist delivery of AIM HI within sessions mediated these intervention effects. There were no significant differences in the outcome trajectories on a broader assessment of problems on the SSIS.

This study makes a number of important contributions. First, to our knowledge, this is the first community effectiveness trial of an ASD psychosocial intervention tested within the context of publicly funded MH services. The limited number of ASD effectiveness trials^29,47 and community-based studies^21,22,24,48 that have been conducted were in education, early intervention, or specialized ASD service settings. Second, the development of AIM HI is unique among ASD interventions because it was designed specifically for the end users and the targeted community service setting based on a comprehensive needs assessment in collaboration with relevant stakeholders.^9-11 The present study results support the effectiveness of a model designed to fit the needs and structure of community services when delivered by therapists who do not specialize in ASD. The significant decreases in frequency and severity of challenging behaviors suggests that AIM HI was effective in reducing the problem behavior symptoms it was designed to improve. Although modest, the ECBI effect sizes are consistent with or higher than those in trials of psychosocial interventions for youth⁴⁹ compared with usual care with similar designs (eg, youth who were clinically referred, the majority of sample from ethnic minority backgrounds, and parent/family focused interventions). The lack of differences in outcome trajectories for a broader assessment of problem behaviors (SSIS Competing Problem Behaviors scale) may relate to the scale content (a broad array of behaviors considered to influence social skill development).

A strength of this study is the large and diverse sample of therapists who are representative of the MH workforce (eg, master of arts–level licensed and trainees) and clients (primarily ethnic minority and low socioeconomic status with a large proportion of caregivers having limited English). Although the sample of 202 children is relatively small given the heterogeneity of ASD, the lack of differences in intervention effects by child characteristics (race/ethnicity, language, and income) and setting (clinic vs school) is promising and may support the effectiveness of AIM HI with diverse populations and in multiple settings.

Methodologic strengths include the use of a cluster randomized trial, which addresses pragmatic concerns related to contamination of data. In addition, collecting data on effectiveness (child symptoms) and implementation (therapist fidelity) responds to calls for hybrid research designs to accelerate the translation of evidence-based practices into routine care.⁵⁰ The finding that therapist fidelity related to session structure and continuity across sessions, using the criterion standard method of observational coding, mediated child outcome trajectories suggests the need for development and testing of implementation strategies to enhance therapists’ fidelity to deliver evidence-based interventions such as AIM HI. It is possible that therapists with higher scores on session structure and continuity across sessions engage in more focused skill-building activities, which results in reductions in challenging behaviors. Enhancing fidelity is the primary focus of 2 linked trials currently under way that aim to test implementation strategies across 2 ASD community-based interventions (AIM HI and Classroom Pivotal Response Teaching).⁵¹

Quiz Ref IDThe present study findings should be considered within the context of a number of limitations. First, the outcomes measured were limited to parent report of child challenging behaviors. The next steps are to examine changes in child behaviors reported by teachers and changes in parent functioning. Second, given the complexity of obtaining data within a community trial, there were a number of missing data points, particularly in the usual care group. This limitation was minimized through an analytic approach accounting for missing data. Third, although the researchers conducting the outcome assessments were blinded to the condition, clients/caregivers were not. Fourth, because the study simultaneously tested the AIM HI intervention protocol and therapist training model, it was not possible to disentangle the effectiveness of the intervention from that of the training. Future study could control for therapist training by comparing training in 2 active treatment groups to further understand the effectiveness of AIM HI on child outcomes.

The findings of this community trial support the effectiveness of training therapists in publicly funded MH services, who are not ASD specialists, to deliver evidence-based strategies to reduce challenging behaviors among children with ASD. The study findings highlight the importance of developing and testing implementation strategies aimed to facilitate high therapist fidelity in delivering evidence-based interventions.

Accepted for Publication: November 27, 2018.

Corresponding Author: Lauren Brookman-Frazee, PhD, Department of Psychiatry, University of California San Diego, 9500 Gilman Dr, La Jolla, CA 92093-0812 (lbrookman@ucsd.edu).

Published Online: March 6, 2019. doi:10.1001/jamapsychiatry.2019.0011

Author Contributions: Drs Brookman-Frazee and Roesch had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Brookman-Frazee, Baker-Ericzen.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Brookman-Frazee, Chlebowski, Baker-Ericzen.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Roesch.

Obtained funding: Brookman-Frazee.

Administrative, technical, or material support: Brookman-Frazee, Chlebowski, Baker-Ericzen, Ganger.

Supervision: Brookman-Frazee, Baker-Ericzen.

Conflict of Interest Disclosures: None reported.

Funding/Support: This work was supported by Research Project Grant R01MH094317 from the National Institute of Mental Health.

Role of the Funder/Sponsor: The National Institute of Mental Health had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The views expressed in this article are solely those of the authors and do not reflect the endorsement or the official policy or position of the National Institute of Mental Health.

Data Sharing Statement: See Supplement 2.