eTable. Zero-Order Correlations Between Predictors and Outcome Variables
Beidas RS, Marcus S, Aarons GA, Hoagwood KE, Schoenwald S, Evans AC, Hurford MO, Hadley T, Barg FK, Walsh LM, Adams DR, Mandell DS. Predictors of Community Therapists’ Use of Therapy Techniques in a Large Public Mental Health System. JAMA Pediatr. 2015;169(4):374-382. doi:10.1001/jamapediatrics.2014.3736
Few studies have examined the effects of individual and organizational characteristics on the use of evidence-based practices in mental health care. Improved understanding of these factors could guide future implementation efforts to ensure effective adoption, implementation, and sustainment of evidence-based practices.
To estimate the relative contribution of individual and organizational factors on therapist self-reported use of cognitive-behavioral, family, and psychodynamic therapy techniques within the context of a large-scale effort to increase use of evidence-based practices in an urban public mental health system serving youth and families.
Design, Setting, and Participants
In this observational, cross-sectional study of 23 organizations, data were collected from March 1 through July 25, 2013. We used purposive sampling to recruit the 29 largest child-serving agencies, which together serve approximately 80% of youth receiving publically funded mental health care. The final sample included 19 agencies with 23 sites, 130 therapists, 36 supervisors, and 22 executive administrators.
Main Outcomes and Measures
Therapist self-reported use of cognitive-behavioral, family, and psychodynamic therapy techniques, as measured by the Therapist Procedures Checklist–Family Revised.
Individual factors accounted for the following percentages of the overall variation: cognitive-behavioral therapy techniques, 16%; family therapy techniques, 7%; and psychodynamic therapy techniques, 20%. Organizational factors accounted for the following percentages of the overall variation: cognitive-behavioral therapy techniques, 23%; family therapy techniques, 19%; and psychodynamic therapy techniques, 7%. Older therapists and therapists with more open attitudes were more likely to endorse use of cognitive-behavioral therapy techniques, as were those in organizations that had spent fewer years participating in evidence-based practice initiatives, had more resistant cultures, and had more functional climates. Women were more likely to endorse use of family therapy techniques, as were those in organizations employing more fee-for-service staff and with more stressful climates. Therapists with more divergent attitudes and less knowledge about evidence-based practices were more likely to use psychodynamic therapy techniques.
Conclusions and Relevance
This study suggests that individual and organizational factors are important in explaining therapist behavior and use of evidence-based practices, but the relative importance varies by therapeutic technique.
Implementation science frameworks posit that individual (eg, knowledge and attitudes) and organizational (eg, culture and climate) characteristics affect the provision of evidence-based practices (EBPs) in general health and mental health care.1 Little is known about the relative contributions of these 2 sets of characteristics. This study estimates the relative contribution of individual and organizational factors on therapists’ use of cognitive-behavioral therapy (CBT), family therapy, and psychodynamic therapy techniques within the context of a large-scale effort to increase the use of CBT in an urban public mental health system.
Literature supports the role of individual and organizational factors in the provision of children’s mental health services. For example, individual factors, such as attitudes toward EBPs,2,3 predict the extent to which therapists deliver EBPs as designed. Similarly, organizational factors, such as organizational culture (ie, shared employee perceptions around expectations and norms)4 and organizational climate (ie, psychological effect of the work environment on individual well-being),5 have been linked to quality of services6 and youth mental health outcomes.6,7 Previous research has largely focused on individual or organizational factors. Both sets of studies find evidence of the predictive validity of their constructs of interest. Individual and organizational factors are correlated,8 making it difficult to disentangle the contributions of each set. Furthermore, different outcomes have been examined in these 2 sets of studies, making it challenging to compare results.
To address these issues, we measured the association between individual and organizational characteristics and the use of therapy techniques in therapists in an urban public mental health system engaged in a large-scale effort to increase the use of CBT. We explore the relative contribution of individual and organizational characteristics on therapist self-reported use of CBT, family therapy, and psychodynamic therapy techniques, all 3 being widely endorsed techniques used by therapists in usual care.9 Cognitive-behavior therapy has a large body of evidence to support its effectiveness in addressing youth psychiatric disorders.10 Family therapy is effective for some youth psychiatric disorders, particularly when family-oriented components are combined with CBT techniques.11 Psychodynamic therapy has little evidence to support its effectiveness for youth psychiatric disorders.10
Since 2007, the Philadelphia Department of Behavioral Health and Intellectual disAbility Services has supported the implementation of therapy techniques in the public mental health system based on the principles of CBT. Implementation support includes a full-time city employee who coordinates implementation and training and consultation by treatment developers.
There are more than 100 community mental health agencies in Philadelphia that provide outpatient services to youth (Cathy Bolton, PhD, written communication, January 3, 2013). We used purposive sampling to recruit the 29 largest child-serving agencies, which serve approximately 80% of youth receiving publically funded mental health care. Of these 29 agencies, 18 (62%) agreed to participate. In addition, 1 agency involved in EBPs efforts asked to participate, resulting in a final sample of 19 agencies (23 sites, 130 therapists, 36 supervisors, and 22 executive administrators). Each site rather than each agency was treated as a distinct organization because of different leadership structures, locations, and staff. In this article, we refer to site as organization. The leader of each organization was invited to participate as the executive administrator. There were no exclusion criteria for participation. Of the organizations enrolled in this study, 16 had participated in city-sponsored EBPs initiatives.
All procedures were approved by the City of Philadelphia and University of Pennsylvania institutional review boards. We approached the executive administrator of each organization for participation. Executive administrators completed their questionnaires using REDcap, a secure Web-based application that supports online data collection.12 For supervisors and therapists, we scheduled a 1-time, 2-hour meeting at each organization, at which we provided lunch, obtained written informed consent, and completed data collection. Approximately 60% of therapists employed by the 23 organizations participated in the study, which collected data from March 1 through July 25, 2013. Participants received $50.
Executive administrators provided information on their age, sex, ethnicity/race, and educational background. Therapist and supervisor demographics were assessed using the Therapist Background Questionnaire,13 a 21-item demographics questionnaire. We also asked participants to report on employment status (ie, fee for service or salaried).
Therapist attitudes were assessed using the Evidence-Based Practice Attitude Scale,14 a 15-item self-report questionnaire that assesses constructs related to appeal the of EBPs, requirements to use EBPs, general openness to new practices, and divergence between EBPs and usual practice. Each subscale is a mean of the items within that factor, which is measured on a continuum from 0 (not at all) to 4 (very great extent). The Evidence-Based Practice Attitude Scale has good internal consistency15 and validity.16
Therapist knowledge about EBPs was measured using the Knowledge of Evidence-Based Services Questionnaire,17 a 40-item self-report instrument. Knowledge is measured on a continuum from 0 to 160, with higher scores indicative of more knowledge of evidence-based services for youth. Psychometric data suggest temporal stability, discriminative validity, and sensitivity to training.17
Supervisors provided information on the number of therapists in their unit and their employment status. From that information, we determined program size (ie, number of therapists in their unit) and percentage of fee-for-service staff. We also assessed the number of years the organization had formally been involved in city-sponsored EBPs initiatives.
Organizational culture and climate were measured from the perspectives of therapists, supervisors, and executive administrators using the Organizational Social Context Measurement System,18 a 105-item measure of the social context of mental health organizations. Organizational culture includes proficiency, rigidity, and resistance, and climate includes engagement, functionality, and stress. Proficient cultures are those in which therapists prioritize the well-being of clients and are expected to keep up competencies. Rigid cultures are those in which therapists have little autonomy and discretion. Resistant cultures refer to ones in which therapists are expected to be apathetic. Engaged climates refer to ones in which therapists think they can accomplish worthwhile tasks and remain invested in their work. Functional climates are ones in which therapists think they are able to get their job done effectively. Stressful climates refer to ones in which therapists feel emotionally exhausted. Organizational culture and climate are measured with t scores with a mean of 50 and an SD of 10 based on a normed sample of 100 community mental health clinics.18 The Organizational Social Context Measurement System has strong psychometric properties.19
Implementation climate was measured from the perspective of therapists, supervisors, and executive administrators, using the Implementation Climate Scale,20 an 18-item scale that measures climate for EBPs implementation, including focus on EBPs, educational support for EBPs, recognition for using EBPs, rewards for using EBPs, selection of staff for EBPs, and general organizational openness. Each subscale is a mean of the items within that factor, which is measured on a continuum from 0 (not at all) to 4 (very great extent). Psychometrics are strong and have excellent reliability and validity.20
Implementation leadership was measured from the perspective of therapists, rating their direct supervisor with the Implementation Leadership Scale,21 a 12-item scale that measures leader behaviors relevant to implementation of EBPs, including proactive, knowledgeable, supportive, and perseverant leadership. Each subscale is a mean of the items within that factor, which is measured on a continuum from 0 (not at all) to 4 (very great extent). The Implementation Leadership Scale has strong psychometric properties and excellent internal consistency and validity.21
Therapists’ use of CBT, family therapy, and psychodynamic therapy techniques was measured using the Therapy Procedures Checklist–Family Revised (TPC-FR),22 a 62-item self-report checklist. Therapists were asked to respond in reference to a representative client who they were currently treating. Therapists could endorse using strategies from all 3 families of techniques. Each dependent variable is a mean of the items that fit within that factor, which is measured on a continuum from 1 to 5, with 1 indicating rarely; 2, seldom; 3, sometimes; 4, often; and 5, most of the time. Higher scores are indicative of more use of the set of techniques. Factor structure has been confirmed, test-retest reliability is strong, and the instrument is sensitive to within-therapist changes in technique use.22,23
Organizational measures are constructed by aggregating individual responses within the organization, if there is enough agreement among individuals. To determine agreement, we used mean within-group correlation statistics.24,25 On all organizational variables, statistics were above the suggested 0.60 level25,26; therefore, participant responses to organizational constructs were averaged within each organization. Missing data for independent and predictor variables were minimal (<10%); series means were imputed for missing predictor variables.
We used 3 sets of linear mixed-effects regression models to determine the associations of individual and organizational factors (independent variables) with self-reported use of CBT, family therapy, and psychodynamic therapy techniques (dependent variables). All linear mixed-effects models included random intercepts for organization to account for nesting of therapists within organizations and fixed effects for individual and organizational factors. Therapist factors included participant demographics (sex, age, clinical experience, and employment status [ie, fee for service or salaried]), attitudes (Evidence-Based Practice Attitude Scale subscales), and knowledge (Knowledge of Evidence-Based Services Questionnaire total score). Organizational factors included organization demographics (cumulative years organization participated in city-sponsored EBPs initiatives, program size [ie, number of therapists in the unit], and organization type [percentage of fee-for-service staff]), implementation climate (Implementation Climate Scale subscales), implementation leadership (Implementation Climate Scale subscales), and organizational culture and organizational climate (Organizational Social Context Measurement System subscales). Dependent variables included use of CBT, family therapy, and psychodynamic therapy techniques. Therapist case mix and therapist race/ethnicity were initially included in the model as covariates but were removed because associated coefficients were not statistically significant.
Analyses were conducted using PROC MIXED in SAS statistical software, version 9.0 (SAS Institute Inc). Four separate models were conducted for each of the dependent variables. In the first unconditional model, only the organization random effect was included to provide an estimate of the organizational and residual variance. This model allows us to estimate how much of the total variance could possibly be attributed to the organization and to calculate the intraclass correlation coefficient. In the second model, only individual fixed effects were included to estimate the total variance attributable to individual fixed effects. This model allowed us to calculate the proportion of the total variation in the model explained by individual factors. In the third model, only organizational fixed effects were included to estimate the total variation attributable to organizational fixed effects. In the fourth model, both individual and organizational fixed effects were included; these models are reported on in the Results section. Our goal was to understand how much of the overall variation in our dependent variable(s) was explained by the set of organizational and individual factors, respectively (ie, separately). The proportion of variation explained by individual factors was calculated by subtracting the total variation from the individual factors model (model 2) from the variance of the unconditional model (model 1) and dividing by the total variation from model 1 (ie, %Vartherapist = [Varunconditional − Vartherapist]/Varunconditional); the same technique was used separately to calculate the proportion of variation for organizational factors (ie, %Varorganization = [Varunconditional – Varorganization]/Varunconditional).27 Our analyses of the contributions of individual and organizational factors focus on the unique effect of each factor after controlling for all other factors in the model.
Table 1 provides demographic information about the therapists. Of the 22 executive administrators, 11 (50%) were male; they identified as Asian (2 [9%]), Hispanic/Latino (3 [14%]), African American (4 [18%]), white (12 [55%]), multiracial (2 [9%]), or missing ethnicity/race (2 [9%]). Of the 36 supervisors, 25 (69%) were female; they identified as African American (6 [17%]), white (20 [56%]), Hispanic/Latino (5 [14%]), other (1 [3%]), or missing ethnicity/race (4 [11%]).
Table 1 gives the descriptive statistics for all variables included in the models. The eTable in the Supplement is a correlation matrix that documents correlations between predictors and outcomes.
Table 2 gives the model parameters. Organizational factors accounted for 23% of the variance in therapists’ use of CBT techniques; therapist factors accounted for 16%. Three organizational variables were associated with use of CBT techniques. Therapists in organizations that had participated for fewer years in city-sponsored EBPs initiatives, had more resistant cultures, and had more functional climates were more likely to use CBT techniques. Two therapist variables were associated with the use of CBT techniques. Older therapists and therapists with more open attitudes toward new practices were more likely to use CBT techniques.
Organizational variables accounted for 19% of the variance in the use of family therapy techniques; therapist variables accounted for 7%. Two organizational variables were associated with use of family therapy techniques. Organizations using more fee-for-service staff and organizations with more stressful climates were more likely to use family therapy techniques. One individual variable was associated with use of family therapy techniques: women were more likely to use family therapy techniques.
Individual factors accounted for 20% of the variance in the use of psychodynamic therapy techniques; organizational factors accounted for 7%. Two individual factors were associated with the use of psychodynamic therapy techniques. Therapists with more divergent attitudes on the perceived difference between EBPs and current practices and less knowledge about EBPs were more likely to use psychodynamic therapy techniques.
For use of CBT techniques, individual and organizational factors collectively explained 30% of the overall variation; for use of psychodynamic therapy strategies, individual and organizational factors collectively explained 18% of the overall variation; for use of family therapy strategies, individual and organizational factors collectively explained 26% of the overall variation (data not shown).
This study provides information on what predicts therapists’ use of therapy techniques in a large public mental health system supporting implementation of CBT. Organizational factors accounted for more of the variance in therapists’ use of CBT and family therapy techniques. Conversely, individual factors accounted for more of the variance in therapists’ use of psychodynamic therapy techniques. Both CBT and family therapy are evidence-based techniques for childhood disorders, whereas psychodynamic therapy techniques have less rigorous evidence to support them.10 Taken collectively, these findings suggest that organizational factors are more likely to drive use of EBPs, whereas therapist attributes are more likely to drive use of non-EBPs therapy techniques.
Consistent with the literature,16 therapists with more open attitudes were more likely to use CBT techniques. However, inconsistent with the literature,14 older therapists were more likely to use CBT techniques. Older therapists may have more experience changing treatment modalities according to demand because they have been in the system longer. Paradoxically, organizations that had spent fewer years participating in city-sponsored EBPs initiatives were more likely to use CBT techniques. It is possible that organizations that seek out CBT training have therapists less experienced in CBT. Alternatively, it is possible that organizations spending more years participating in EBPs initiatives experience EBPs fatigue (ie, innovation fatigue28) or the stress involved with competing demands of difficult human service jobs coupled with lack of clarity of how these initiatives fit with the role of therapist. Further research is needed to understand potential unintended consequences of EBPs efforts, such as innovation fatigue.
Organizations with more resistant cultures and more functional climates were more likely to have therapists who endorsed using CBT techniques. Organizations with more resistant cultures may be more likely to participate in initiatives to increase use of innovation. Therapists’ perceptions that they are functioning effectively appear to contribute to more use of CBT techniques. Organizations with more stressful climates and more fee-for-service staff were more likely to have therapists who used family techniques. It is possible that organizations with more stressful climates serve a more chaotic population, suggesting the potential usefulness of family therapy techniques, which are indicated for youth with psychiatric disorders and chaotic family environments.29
Consistent with the literature, therapists who were less knowledgeable about EBPs and held more divergent attitudes toward EBPs were more likely to use psychodynamic therapy techniques.30 Also consistent with previous findings,9 therapists reported using CBT and other therapy techniques (ie, eclecticism) at the same time, suggesting a potential exnovation problem. Exnovation refers to “the process whereby an organization decides to divest itself of an innovation that it had previously adopted.”31 Further research is needed to understand how organizations can plan for EBPs that fits with existing practices.
Of interest, implementation climate and leadership, constructs hypothesized to be related to implementation outcomes,8,32 did not predict use of EBPs. However, the current study did not examine more complex interactive or mediational processes to account for how these constructs may work together with molar culture or climate to predict the outcome used in this study.33
Some study limitations should be noted. First, the primary outcome variables are based on therapists self-report of use of therapeutic techniques, and therapists are not always accurate reporters of use of therapeutic techniques.34,35 Second, we did not have 100% participation, creating a potential selection bias at the therapist and organizational levels. Third, we used a random intercepts–only model and did not allow the slopes to vary by organization because of sample size limitations.
These findings offer important implications. Organizational variables accounted for more of the variance than individual variables in predicting use of EBPs, suggesting that organizational-level implementation strategies36 will be more effective in increasing the use of EBPs than implementation strategies that directly target the therapist. Individual factors account for more of the variance than organizational factors in therapists’ use of psychodynamic therapy techniques. Efforts to implement EBPs may need to be accompanied by efforts to encourage therapists to divest themselves of outdated practices. Although these 2 activities may be seen as 2 sides of the same coin, the results of this study suggest that the process of exnovation may be driven by different factors than the process of implementation. Of perhaps equal importance, the variables included in our measurement model represent many of the constructs posited to predict implementation, and yet separately they accounted only for a maximum of 23% of the overall variation in outcomes, suggesting that there are a number of unmeasured constructs. Finally, this study highlights the need for prospective studies to test the relative contributions and interactions of individual- and organization-focused implementation strategies on adoption, fidelity, and sustainment of EBPs.
These findings suggest the nuanced effect of individual and organizational factors on therapists’ use of therapy techniques. This study suggests that both where you work and who you are matter in understanding therapists’ behavior in context and that improving the effectiveness of implementation strategies should consider both approaches.37
Accepted for Publication: December 21, 2014.
Corresponding Author: Rinad S. Beidas, PhD, Department of Psychiatry, University of Pennsylvania Perelman School of Medicine, 3535 Market St, 3015, Philadelphia, PA 19104 (firstname.lastname@example.org).
Published Online: February 16, 2015. doi:10.1001/jamapediatrics.2014.3736.
Author Contributions: Dr Beidas had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Beidas, Marcus, Aarons, Evans, Hadley, Barg, Mandell.
Acquisition, analysis, or interpretation of data: Beidas, Aarons, Hoagwood, Schoenwald, Hurford, Walsh, Adams.
Drafting of the manuscript: Beidas, Marcus, Aarons, Walsh.
Critical revision of the manuscript for important intellectual content: Aarons, Hoagwood, Schoenwald, Evans, Hurford, Hadley, Barg, Adams, Mandell.
Statistical analysis: Beidas, Marcus, Mandell.
Obtained funding: Beidas.
Administrative, technical, or material support: Aarons, Evans, Hurford, Hadley, Barg, Walsh, Adams.
Study supervision: Beidas, Aarons, Hurford, Barg, Mandell.
Conflict of Interest Disclosures: Dr Beidas reported receiving royalties from Oxford University Press and serving as a consultant for Kinark Child and Family Services. Dr Marcus reported receiving grant support from Ortho-McNeil-Janssen Pharmaceuticals Inc and Forest Research Institute and serving as a consultant to AstraZeneca and Alkermes. Dr Schoenwald reported being a stakeholder and board member of MST Services, LLC. No other disclosures were reported.
Funding/Support: This study was supported by grant K23 MH099179 and an Implementation Research Institute fellowship (2012-2014) from the National Institute of Mental Health (Dr Beidas); the Implementation Research Institute, George Warren Brown School of Social Work, Washington University, St Louis, Missouri; grant R25 MH080916 from the National Institute of Mental Health; and the Quality Enhancement Research Initiative, Department of Veterans Affairs Contract, Veterans Health Administration, Office of Research and Development, Health Services Research and Development Service.
Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.
Additional Contributions: The following experts provided their time and input on this project: Marc Atkins, PhD (University of Illinois at Chicago), Ross Brownson, PhD (Washington University in St Louis), David Chambers, DPhil (National Cancer Institute), Bruce Chorpita, PhD (UCLA), Charles Glisson, PhD (University of Tennessee), Nicholas Ialongo, PhD (Johns Hopkins University), John Landsverk, PhD (Oregon Social Learning Center), Enola Proctor, PhD (Washington University in St Louis), and Ronnie Rubin, PhD (Community Behavioral Health). The Department of Behavioral Health and Intellectual DisAbility Services and the Evidence Based Practice and Innovation group also supported this project. No compensation was provided to the contributors.