Training Models for Implementing Evidence-Based Psychological Treatment: A Cluster-Randomized Trial in College Counseling Centers

Key Points

Question  Does training college (university) counseling center therapists in an evidence-based treatment (interpersonal psychotherapy) using the train-the-trainer method vs an expert training method result in improved fidelity (adherence and competence)?

Findings  In this cluster-randomized trial that included 184 therapists from 24 college counseling centers, results indicated within group improvements in both adherence and competence; only competence differed between groups, favoring the train-the-trainer condition.

Meaning  Results support the effectiveness of the train-the-trainer approach; further, given its potential capability to train more therapists over time, it has the potential to facilitate widespread dissemination of evidence-based treatments.

Importance  Progress has been made in establishing evidence-based treatments for psychiatric disorders, but these are not often delivered in routine settings. A scalable solution for training clinicians in evidence-based treatments is needed.

Objective  To compare 2 methods of training college (university) counseling center therapists to treat psychiatric disorders using interpersonal psychotherapy. The hypothesis was that the train-the-trainer condition would demonstrate superior implementation outcomes vs the expert condition. Moderating factors were also explored.

Design, Setting, and Participants  This cluster-randomized trial was conducted from October 2012 to December 2017 in 24 college counseling centers across the United States. Therapist participants were recruited from enrolled centers, and student patients with symptoms of depression and eating disorders were recruited by therapists. Data were analyzed from 184 enrolled therapists.

Interventions  Counseling centers were randomized to the expert condition, which involved a workshop and 12 months of follow-up consultation, or the train-the-trainer condition, in which a staff member from the counseling center was coached to train other staff members.

Main Outcomes and Measures  The main outcome was therapist fidelity (adherence and competence) to interpersonal psychotherapy, as assessed via audio recordings of therapy sessions. Therapist knowledge of interpersonal psychotherapy was a secondary outcome.

Result  A total of 184 therapists (mean [SD] age, 41.9 [10.6] years; 140 female [76.1%]; 142 white [77.2%]) were included. Both the train-the-trainer–condition and expert-condition groups showed significant within-group improvement for adherence to interpersonal psychotherapy (change: 0.233 [95% CI, 0.192-0.274] and 0.190 [0.145-0.235], respectively; both P < .001), with large effect sizes (1.64 [95% CI, 1.35-1.93] and 1.34 [95% CI, 1.02-1.66], respectively) and no significant difference between conditions. Both groups also showed significant within-group improvement in interpersonal therapy competence (change: 0.179 [95% CI, 0.132-0.226] and 0.106 [0.059-0.153], respectively; both P < .001), with a large effect size for the train-the-trainer condition (1.16 [95% CI, 0.85-1.46]; P < .001) and a significant difference between groups favoring the train-the-trainer condition (effect size, 0.47 [95% CI, 0.05-0.89]; P = .03). Knowledge of interpersonal psychotherapy improved significantly within both groups (effect sizes: train-the-trainer, 0.64 [95% CI, 0.28-0.99]; P = .005; expert, 0.69 [95% CI, 0.38-1.01]; P < .001), with no significant difference between groups. The significant moderating factors were job satisfaction for adherence (b, 0.120 [95% CI, 0.001-0.24]; P = .048) and competence (b, 0.133 [95% CI, 0.001-0.27]; P = .048), and frequency of clinical supervision for competence (b, 0.05 [95% CI, 0.004-0.09]; P = .03).

Conclusions and Relevance  Results demonstrate that the train-the-trainer model produced training outcomes comparable with the expert model for adherence and was superior on competence. Given its potential capability to train more therapists over time, it has the potential to facilitate widespread dissemination of evidence-based treatments.

Trial Registration  ClinicalTrials.gov Identifier: NCT02079142

Remarkable progress has been made in establishing evidence-based treatments (EBTs) for psychiatric disorders. However, when individuals receive treatment in routine settings, it is typically not an EBT.^1-3 As Insel noted, “We have powerful, evidence-based psychosocial interventions, but they are not widely available…A serious deficit exists in training.”^4(p131) Standard approaches to training typically consist of provision of a manual and a workshop delivered by an expert.⁵ The influence of this approach on skills is short-lived if follow-up consultation does not occur.^6-9 A practical, scalable, effective means of training therapists to implement EBTs is needed. One option for which there is a strong theoretical case¹⁰ for changing therapist behavior is the train-the-trainer approach,⁵ which centers around the development of a trainer who then trains therapists in the setting and serves as an internal coach. Although this approach has some evidence, past research has had limitations (eg, small samples, no comparison group, no assessment of implementation outcomes).^11-13

In this study, we compared the implementation outcomes of 2 methods of training therapists to treat depression and eating disorders on college (university) campuses using interpersonal psychotherapy (IPT). Counseling centers were chosen because they are staffed by therapists of various backgrounds in a manner similar to many community settings. The demand for treatment, also similar to community settings, often outweighs available resources, resulting in barriers to EBT implementation and little time for training. Moreover, this method meant that it was possible to recruit a sufficiently large sample of centers for a cluster-randomized design.

The 2 training methods were an external expert consultation model involving a workshop, manual, and 12 months of expert follow-up consultation (called the expert condition) and a train-the-trainer model in which a staff member from the counseling center was coached to train other staff members to implement IPT (called the trainer condition). Because a trainer is embedded in the site, therapists can continuously be trained over a prolonged period, providing sustainability of benefits, including potential cost-effectiveness, compared with the expert model. This model may also be a particularly good fit for college counseling centers, which typically have new trainees entering each year. The primary outcome was therapist fidelity to IPT assessed via audio recordings of therapy sessions, with assessments in 2 dimensions: adherence to the procedures of IPT and level of competence in applying these procedures. Changes in therapist knowledge of IPT were a secondary outcome.

We selected IPT, which is similar in structure to other EBTs, to disseminate because it is an EBT for 2 of the most common psychiatric disorders seen in college counseling centers, depression and eating disorders^14-18; IPT is thus a transdiagnostic treatment, making it a so-called best-buy intervention.³ Treatments with a wide clinical range, such as IPT, may have increased therapist adoption.¹⁹ Interpersonal psychotherapy is also a particularly good match for young adults, because interpersonal issues are common.^20-23

We hypothesized that the trainer condition would demonstrate superiority in these implementation outcomes compared with the expert condition. We also explored factors associated with and factors moderating the outcomes, including therapist-level and site-level characteristics.

Colleges were eligible to participate if they had a counseling center, at least 3 interested therapists, and a staff member who consented to serve as the study director. This individual provided organizational-level data. Of the 24 consenting directors, 11 also participated as therapists, and 1 participated as a trainer in the trainer condition.

Therapists were eligible if they treated students at the counseling center at least 25% of the time. Study staff obtained written consent from them, and they were enrolled between October 2012 and April 2014. Therapists were asked to obtain written informed consent from up to 2 student patients (eligible if they were 18 years or older and presenting with symptoms of depression and/or eating disorders [excluding anorexia nervosa, given that IPT is not an EBT for this diagnosis]). Therapists decided if the patients met these criteria using their clinical judgment or usual assessment procedures during each of the 2 study phases: baseline (ie, prior to training in IPT) and after training. Student data collection occurred from September 2014 to December 2017. At each site in the trainer condition, 1 trainer was selected by the study’s director at that center. Directors were informed that they should select someone with interest in the project, proficiency as a therapist and supervisor, and a stable position within the center. The study received institutional review board approval at the coordinating sites and each of the participating colleges.

We provided a 2-day workshop in IPT conducted by a study team member with expertise in IPT at each site randomized to the expert condition. Therapists were provided with a treatment manual, and the workshop involved a detailed review of key principles and procedures of IPT, using slides, plus role plays and case examples to demonstrate IPT treatment phases. Therapists at each site had the opportunity to engage in a 1-hour consultation call with the study team member who conducted the workshop every month for up to 12 months after the workshop to monitor treatment quality, provide feedback, and track patient progress.

As a group, trainers attended 2 separate workshops at 1 of the study team sites; the first 2-day workshop was identical in content to the workshop provided in the expert condition and was designed to teach participants to conduct IPT. The second workshop provided training in training others in IPT.

After participation in the first workshop, each trainer returned to their site and was encouraged to treat up to 2 patients with IPT, audio-recording each session. The study team member who conducted the IPT training reviewed a selection of recorded sessions from each case treated by the trainers and provided feedback regarding treatment quality, which modeled for these individuals how to provide such feedback.

The goal of the second workshop was to prepare trainers to train others in IPT. Problem-solving was offered for potential barriers to conducting the training at their sites and providing ongoing consultation. Trainers were provided with video-recorded role plays demonstrating the treatment phases for use in conducting training, as well as standardized IPT checklists and forms designed to facilitate trainees’ use of IPT. This portion of the study, including the 2 workshops for the trainers and the trainer’s practice treating up to 2 patients and receiving feedback, spanned approximately 6 months. Once that phase was complete, trainers were encouraged to train therapists at their sites.

Once trainers had trained their colleagues in IPT, they were encouraged to meet weekly with their trainee colleagues for 1-hour group consultations to monitor treatment quality, provide feedback, and track patient progress. Trainers were also encouraged to join monthly group implementation review calls with the study team member who conducted their training and trainers from other sites.

Additional details of both conditions have been previously described.²⁴ More details on the study methods are also available in the Trial Protocol in Supplement 1.

Three measures were used to assess training outcomes during each assessment interval (ie, baseline and posttraining implementation): therapist adherence to, competence in, and knowledge of IPT. The 2 dimensions of fidelity (adherence [the extent to which treatment is delivered as outlined] and competence [the skill with which the intervention is implemented])²⁵ were assessed from audio recordings of therapy sessions by raters blind to the participant condition, using the IPT Fidelity Rating Scale, which was adapted (to assess both adherence and competence) from the measure developed for the Veterans Health Administration IPT Training Program, which was established through expert consensus.²⁶ Items relevant to the IPT phase being delivered were scored as either 0 or 1 for adherence (absent or present), and for competence, the item was scored as 0 (unsatisfactory or incompetent; this grade was automatically given if the therapist was nonadherent), 1 (good enough or satisfactory), or 2 (at a high level of quality), with the means of ratings being calculated to generate overall scores. We note that it was not expected that therapists would use every technique in a single session. Raters also determined whether the session could be better characterized as cognitive-behavioral therapy, dialectical behavior therapy, psychodynamic therapy, or motivational interviewing. Additional information is in the eAppendix in Supplement 2. Two audio recordings were rated for each patient from session 1 and a randomly chosen later session so that a selected sample of therapist behavior could be rated. Raters were a senior study team member and 5 graduate students and study staff members who were blinded to condition and study phase and who had received training in IPT. All raters received their training from the same study team member with expertise in IPT who had trained therapists in the expert condition and trainers in the trainer condition; in addition, the raters also received training from a senior study team member in rating audio recordings for IPT fidelity. This method of training raters and auditing audio recordings for fidelity to IPT has demonstrated validity and good reliability (ie, intraclass correlation coefficients ranging from 0.78-0.96) in other clinical trials.^27,28 Interrater reliability (2-way mixed-effects intraclass correlation), calculated from a subset of 9 audio recordings of posttraining tapes rated by multiple raters, was 0.72 (95% CI, 0.46-0.91), which is consistent with prior work.^27,28 Knowledge of IPT was assessed from 20 multiple-choice questions developed for this study.

Therapist characteristics assessed at baseline included age, sex, race/ethnicity, degree, years employed in the present position, attendance at a prior IPT workshop or class (yes or no), experience using IPT with patients in the past year (yes or no), and job satisfaction. The Evidence-Based Practice Attitude Scale²⁹ total score was used to ascertain the individual’s degree of acceptance of EBTs. Finally, job satisfaction was assessed with the 14-item job-satisfaction scale, which demonstrates construct validity and acceptable internal consistency.³⁰

Site-level assessments included information concerning the site director: age, experience, race/ethnicity, and degree, as well as the number of students at the site, therapist-to-student ratio, clinical supervision frequency (rated on a scale of 1 [indicating 0 times per week] to 7 [>2 times per week]), and 1 item assessing staff training and educational priorities (rated on a scale of 1 to 5, with 1 indicating not at all and 5 indicating a very great extent).

After baseline assessment, 24 sites were randomly allocated to the 2 conditions by the data coordinating center using a computer program, matched on therapist-to-student ratio. Matching was achieved based on simple randomization. A power analysis was conducted based on the primary longitudinal analysis at the therapist level. With a medium effect size (Cohen d, 0.50), 112 therapists (56 per group) were needed to reach power of 0.80.

We used standard linear mixed effects modeling^31,32 to estimate changes from baseline to posttraining assessments in primary and secondary outcomes. In line with the intention-to-treat principle, we included all randomized therapists in the analyses, as long as data were available from at least 1 of the repeated assessments. For all model estimations, we used maximum likelihood embedded in Mplus version 8 (Muthén & Muthén).³³ Given the moderate sample size, standard errors were estimated using robust maximum likelihood, which is more robust to deviation from parametric assumptions. For the model specification, we used a random-intercept mode, assuming linear change over time. Finally, focusing on key outcomes of interest (adherence and competence), we examined potential moderators in the mixed-effects modeling framework, applying the analytical criteria for detecting moderators in line with the MacArthur approach.^34,35 Effect sizes are reported as Cohen d, and other results are reported in the form of b, a nonstandardized coefficient from the regression of the slope on training condition (centered), baseline covariate (centered), and their interaction. The threshold of significance was P < .05, 2-sided.

Figure 1 shows the CONSORT diagram. Among 215 college counseling sites that were contacted regarding the study, we had no documented response from 92 sites; 59 expressed initial interest but did not enroll; 34 declined; and 6 enrolled but dropped out before randomization, citing inadequate resources to continue. Hence, 24 sites were randomized with the therapist-to-student ratio for each institution taken into account. From those sites, 184 therapists were enrolled, with 89 allocated to the expert condition and 95 to the trainer condition. During the pretraining period, 105 therapists audio-recorded sessions with 1 to 2 patients to allow estimation of the primary outcome. After training, 87 therapists audio-recorded sessions with 1 to 2 patients. There were no significant baseline differences between those who audio-recorded sessions compared with those who did not. In total, we received audio recordings from 115 therapists before and/or after training. Pretraining recordings indicated that 9 of 97 therapists (9.3%) were using an EBT (either cognitive behavioral therapy or dialectical behavior therapy). There was no evidence of the use of IPT. Ratings of nonspecific factors of establishing rapport and using a collaborative approach were high both before and after training.

Site, site director, and therapist characteristics are shown in Table 1 and Table 2. A diverse sample of small institutions (with 2500 students) to large institutions (with 51 000 students) and private (n = 7) and public (n = 17) institutions entered the study. Therapist-to-student ratios ranged from 1 therapist for 337 students to 1 therapist for 2900 students at various single institutions, indicating different abilities between institutions to provide services.

Most directors and therapists were white (19 of 24 directors [79.2%]; 142 of 184 therapists [77.2%]), and as expected, directors tended to be older than therapists (mean [SD] age: directors, 49.5 [8.7] years; therapists, 41.9 [10.6] years) and to be more likely to have a doctoral degree (directors: 22 of 24 [91.7%]; therapists, 110 of 184 [59.8%]). Nearly one-quarter of therapists (43 of 184 [23.4%]) reported having had taken a class/workshop in IPT before training and more than one-third of therapists (70 of 184 [38.0%]) reported having used IPT in the past year. Attitudes toward evidence-based practice were in the midrange (mean [SD], 2.8 [0.5] on a scale of 0 to 4), and mean job satisfaction was high (mean [SD], 5.9 [0.6] on a scale of 1 to 7).

Both training groups showed significant within-group improvement for IPT adherence (trainer group: 0.233 [95% CI, 0.192-0.274]; expert group: 0.190 [0.145-0.235]; both P < .001) with large effect sizes (trainer group: 1.64 [95% CI, 1.35-1.93]; expert group: 1.34 [1.02-1.66]), but there was no significant difference between conditions (Table 3). Both groups also showed significant within-group improvement in IPT competence (trainer group: 0.179 [95% CI, 0.132-0.226]; expert group: 0.106 [0.059-0.153]; both P < .001), with a large effect size for the trainer condition (1.16 [95% CI, 0.85-1.46]; P < .001) and a medium effect size for the expert condition (0.69 [95% CI, 0.38-0.99]; P < .001), as well as a significant difference between groups favoring the trainer condition (0.47 [95% CI, 0.05-0.89]; P = .03). Adherence and competence were highly correlated both before and after training (pretraining: Spearman ρ, 0.85; P < .001; posttraining: Spearman ρ, 0.83; P < .001). Within-group knowledge of IPT improved significantly for both groups (effect sizes: trainer, 0.64 [95% CI, 0.28-0.99]; P = .005; expert, 0.69 [95% CI, 0.38-1.01]; P < .001), with no significant difference between groups.

Six pretraining variables were examined as moderating factors of training on both adherence and competence: knowledge of IPT, job satisfaction, clinical supervision frequency, staff training and educational priorities, evidence-based practice attitude, and use of IPT in the past year. Job satisfaction significantly moderated the outcome of training on adherence (b, 0.120 [95% CI, 0.001-0.24]; P = .048). Figure 2 shows the overall outcome of training with respect to adherence favoring the trainer group, together with job satisfaction dichotomized. This depicts a greater difference between conditions for therapists with high job satisfaction. The trainer condition was also associated with greater competence regardless of job satisfaction, although the outcome with respect to competence was greater in the group with high job satisfaction (b, 0.133 [95% CI, 0.001-0.27]; P = .048). In addition, the frequency of clinical supervision afforded by the site moderated the outcome of training with respect to therapist competence, with a higher frequency of supervision showing greater training outcome with respect to competence (b, 0.05 [95% CI, 0.004-0.09]; P = .03). Several pretraining variables were positively associated with therapist adherence to IPT, irrespective of condition. These included evidence-based practice attitude (b, 0.10 [95% CI, 0.04-0.17]; P = .002), site staff training and educational priorities (b, 0.06 [95% CI, 0.02-0.09]; P = .003), and clinical supervision frequency (b, 0.05 [95% CI, 0.02-0.07]; P < .001). Evidence-based practice attitude (b, 0.10 [95% CI, 0.02-0.18]; P = .01) and staff training and educational priorities (b, 0.06 [95% CI, 0.02-0.10]; P = .001) were significantly associated with competence.

After training, both the expert and trainer groups showed significant improvement in IPT adherence, with large effect sizes but no significant group difference. This suggested that the trainer condition was as effective as the expert condition at improving adherence. Both groups also evidenced significantly increased IPT competence after training. The trainer condition, showing a large effect size, was significantly more effective than the expert condition in enhancing competence.

Assessing adherence alone is inadequate. It is essential to determine the competence with which interventions are implemented—in other words, to measure doing the right things well^25,36—hence the importance of the findings regarding the greater effectiveness of the trainer model in developing IPT competence. Knowledge of IPT improved significantly for both conditions, with no significant difference between them.

The therapists in both conditions showed little evidence of fidelity in applying IPT at baseline. Nevertheless, 23.4% of therapists self-reported having previously taken an IPT workshop or class, and 38% reported having used IPT in the past year. However, study data indicate that, despite therapists’ self-reported experience with and use of IPT, the use of this approach was not demonstrated. In addition, ratings of all sessions revealed that other possible EBTs were rarely used before or after training, aligning with other work that has demonstrated that EBTs are rarely used in clinical practice.^1-3,37,38 In contrast with the lack of implementation of specific EBTs, therapists showed consistently high fidelity with respect to nonspecific elements.

As per Figure 2, job satisfaction and frequency of clinical supervision emerged as significant moderating factors. High job satisfaction was associated with a greater difference between conditions on adherence, and both high job satisfaction and higher frequency of clinical supervision showed greater training effect sizes on therapist competence, particularly in the trainer condition. In line with the moderator findings, models of training transfer suggest that both trainee characteristics (including job satisfaction) and aspects of the work environment (including frequency of clinical supervision) influence how well the trainee applies and maintains newly learned skills.^5,6,39,40

Findings should be interpreted in light of limitations. We cannot generalize the findings from the selected sample of sites and therapists to college counseling centers as a whole, although we had a large number of diverse, participating colleges (eg, geographic diversity and public and private institutions). Although we were able to examine moderating factors, we did not evaluate the mechanisms by which the trainer model resulted in greater competence in IPT compared with expert training. Future research is needed to examine the specific processes underlying this result, which may then lead to increased efficiency and effectiveness of the trainer model as well sustainability in either training model. Finally, we cannot specifically generalize these findings to other EBTs but anticipate the trainer model could work equally well for other approaches. Other important future research directions include establishing standards for acceptable fidelity levels in community settings, investigating whether fidelity can be increased without substantially increasing the time for training, determining cost-effectiveness of the training approaches, assessing whether the trainer model actually facilitates training of more trainees over time, and further tailoring the treatment protocol to match the reality of college counseling centers (ie, a low limit in the number of sessions).

Results demonstrate that the trainer model produced training outcomes comparable with the expert model with respect to adherence and actually demonstrated superiority with respect to competence. Given its potential capability to train more therapists over time, the trainer model has the potential to facilitate widespread dissemination of EBTs like IPT.

Accepted for Publication: August 27, 2019.

Corresponding Author: Denise E. Wilfley, PhD, Department of Psychiatry, Washington University School of Medicine, Mailstop 8134-29-2100, 660 S Euclid Ave, St Louis, MO 63110 (wilfleyd@wustl.edu).

Published Online: November 6, 2019. doi:10.1001/jamapsychiatry.2019.3483

Correction: This article was corrected on November 27, 2019, to add an affiliation for Ramesh Raghavan, MD, PhD (Silver School of Social Work, New York University, New York). The article has been corrected online.

Author Contributions: Dr Wilfley had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Wilfley, Agras, Welch, Jo, Raghavan, Proctor, Wilson.

Acquisition, analysis, or interpretation of data: Wilfley, Agras, Fitzsimmons-Craft, Bohon, Eichen, Jo, Wilson.

Drafting of the manuscript: Wilfley, Agras, Fitzsimmons-Craft, Bohon, Jo, Wilson.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Bohon, Jo.

Obtained funding: Wilfley, Agras, Proctor, Wilson.

Administrative, technical, or material support: Wilfley, Agras, Fitzsimmons-Craft, Bohon, Eichen, Welch, Wilson.

Supervision: Wilfley, Fitzsimmons-Craft, Eichen, Welch, Proctor.

Conflict of Interest Disclosures: Drs Wilfley, Agras, Fitzsimmons-Craft, Bohon, Welch, Raghavan, Proctor, and Wilson reported grants from National Institute of Mental Health during the conduct of the study. Dr Eichen reported grants from the National Institute of Diabetes and Digestive and Kidney Diseases during the conduct of the study. Dr. Proctor reported grants from Washington University in St Louis during the conduct of the study. Dr Jo reported grants from the National Institutes of Health during the conduct of the study. No other disclosures were reported.

Funding/Support: This study is supported by National Institute of Mental Health (NIMH) (grant R01 MH095748 [Drs Wilfley, Agras, and Wilson]). This study was also supported by the National Institutes of Health (grants T32 HL130357, K08 MH120341, K23 DK114480, and K23 MH106794).

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official view of the National Institutes of Health.

Data Sharing Statement: See Supplement 3.

Additional Contributions: We thank all of the participating counseling centers, directors, therapists, and student patients. Dorothy Van Buren, PhD, also contributed immensely to the conduct of this trial. Dr Van Buren passed away in 2018. She had retired in 2018 from the position of Associate Professor within the Department of Psychiatry at Washington University School of Medicine. For 27 years, she served as a clinical psychologist, supervisor, and researcher in the field of childhood obesity and eating disorders. She was compensated by study funding for her contributions.