Can assigning patients to therapists with empirically determined strengths in treating the patients’ specific mental health problem(s) (ie, measurement-based matching) improve the outcomes of naturalistic psychotherapy compared with case assignment as usual?
In this 2-arm, double-blind randomized clinical trial including 48 therapists and 218 outpatients, measurement-based matching promoted significantly greater reductions in patients’ general symptomatic and functional impairment, global psychological distress, and domain-specific impairment on patients’ most elevated presenting problem over 16 weeks postintake.
In this study, mental health care was enhanced by prospectively assigning patients to empirically good-fitting therapists, which requires minimal disruptions within a mental health care system.
Psychotherapists possess strengths and weaknesses in treating different mental health problems, yet performance information is rarely harnessed in mental health care (MHC). To our knowledge, no prior studies have tested the causal efficacy of prospectively matching patients to therapists with empirically derived strengths in treating patients’ specific concerns.
To test the effect of measurement-based matching vs case assignment as usual (CAU) on psychotherapy outcomes.
Design, Setting, and Participants
In this randomized clinical trial, adult outpatients were recruited between November 2017 and April 2019. Assessments occurred at baseline and repeatedly during treatment at 6 community MHC clinics in Cleveland, Ohio. To be eligible, patients had to make their own MHC decisions. Of 1329 individuals screened, 288 were randomized. Excluding those who withdrew or provided no assessments beyond baseline, 218 patients treated by 48 therapists were included in the primary modified intent-to-treat analyses.
Therapist performance was assessed pretrial across 15 or more historical cases based on patients’ pre-post reporting across 12 problem domains of the routinely administered Treatment Outcome Package (TOP). Therapists were classified in each domain as effective (on average, patients’ symptoms reliably improved), neutral (on average, patients’ symptoms neither reliably improved nor deteriorated), or ineffective (on average, patients’ symptoms reliably deteriorated). Trial patients were randomly assigned to good-fitting therapists (matched group) or were assigned to therapists pragmatically (CAU group). There were multiple match levels, ranging from therapists being effective on the 3 most elevated domains reported by patients and not ineffective on any others (highest) to not effective on the most elevated domains reported by patients but also not ineffective on any domain (lowest). Therapists treated patients in the matched and CAU groups, and treatment was unmanipulated.
Main Outcomes and Measures
General symptomatic and functional impairment across all TOP domains (average z scores relative to the general population mean; higher scores indicate greater impairment), global distress (Symptom Checklist-10; higher scores indicate greater distress), and domain-specific impairment on each individual’s most elevated TOP-assessed problem.
Of 218 patients, 147 (67.4%) were female, and 193 (88.5%) were White. The mean (SD) age was 33.9 (11.2) years. Multilevel modeling indicated a match effect on reductions in weekly general symptomatic and functional impairment (γ110 = −0.03; 95% CI, −0.05 to −0.01; d = 0.75), global distress (γ110 = −0.16; 95% CI, −0.30 to −0.02; d = 0.50), and domain-specific impairment (γ110 = −0.01; 95% CI −0.01 to −0.006; d = 0.60), with no adverse events.
Conclusions and Relevance
Matching patients with therapists based on therapists’ performance strengths can improve MHC outcomes.
ClinicalTrials.gov Identifier: NCT02990000
Mental illness is a major public health problem,1-4 and even among people who engage mental health care (MHC), more than 60% do not benefit meaningfully from care received.5,6 These estimates mainly derive from naturalistic studies where outcomes are collected on large patient samples. Such patient-generated data also yield information on MHC therapists, which reveals that therapists differ in their average effectiveness (the between-therapist effect).7 For example, irrespective of treatment type, above-average therapists are up to twice as effective as below-average therapists.8-10 Thus, research underscores that improvements in MHC can occur not just by using evidence-based interventions, but also by harnessing therapist performance data.11
To this end, multidimensional outcome measures that assess distinct symptom and functioning domains can also assess within-therapist differences in treating different mental health problems, and research has demonstrated that most MHC therapists have performance strengths and weaknesses. In 1 study, therapists who were, on average, reliably effective in treating patients with a given presenting problem had large positive pre-post effect sizes (d ranged from 1.00 to 1.52 across problem domains), whereas therapists who were reliably ineffective in treating a given problem had large negative effect sizes (d ranged from −0.91 to −1.42 across domains).5 Moreover, underscoring that therapists possess distinct strengths and weaknesses, there were relatively low correlations among their domain-specific competencies. In another study, such effects were stable; therapists who demonstrated effectiveness or ineffectiveness in 1 wave of cases largely continued being effective or ineffective in those same domains over a second wave.12 These data suggest that in any population of therapists, there is an opportunity to facilitate precision care through therapist specialization. Were therapists to specialize toward their measurement-informed skills and away from their shortcomings, it follows that population-level outcomes could be improved and harm could be reduced. Such personalized case assignment could be a readily scalable complement to efforts at improving patient-treatment matching or therapists’ in-session interventions.
Historically, though, MHC stakeholders (ie, patients, therapists, and administrators) are unaware of therapists’ outcome-based report cards, which represents a critical gap.11,13 Without exploiting such information, suboptimal improvement rates may partly owe to a lack of measurement-based approaches to case assignment. That is, because current assignments are typically nonpersonalized and based on convenience or self-defined therapist expertise (which is often overestimated or inaccurate14,15), it is largely left to chance whether patients will be assigned to therapists who are historically exceptional or average at treating the patients’ primary problems vs being below average in these areas. Conversely, there may be an advantage to intentionally matching patients to therapists based on therapist-level outcome data, and MHC patients have endorsed such matching as a valued priority.16,17
Accordingly, we developed a personalized match system based on therapist performance report cards determined with a multidimensional outcomes tool—the Treatment Outcome Package (TOP).18 By collecting TOP data from enough patients treated by a given therapist, this outcomes tool can establish the domains in which that therapist is stably effective (historically, on average, their patients’ symptoms reliably improved), neutral (historically, on average, their patients’ symptoms neither reliably improved nor deteriorated), or ineffective (historically, on average, their patients’ symptoms reliably deteriorated).5,12 As routine patient outcomes monitoring continues to become more commonplace at the individual level,11 such de facto therapist data can be generated within any MHC system with little to no additional burden, and used to implement our patient-therapist match algorithm. Specifically, after completing the TOP at intake, patients are assigned to therapists with previously established strengths in treating their primary problem(s).
For the first time, to our knowledge, the present study tested the efficacy of patient-therapist matching with an individual-level, double-blind, randomized clinical trial within a community MHC system. Adult outpatients were randomly assigned to matching or case assignment as usual (CAU), and treatment was delivered naturalistically. We hypothesized that patients assigned to empirically well-matched therapists would report greater reduction in general symptomatic and functional impairment (general impairment), global psychological distress, and domain-specific symptomatic or functional impairment on their most elevated (ie, severe) presenting problem domain (domain-specific impairment).
The trial was conducted across 6 clinics in a single health care system in Cleveland, Ohio. A total of 288 patients were enrolled from November 2017 to April 2019. The trial was approved by the institutional review board of the University of Massachusetts, Amherst, overseen by a data and safety monitoring board, and is registered at ClinicalTrials.gov. Written informed consent was obtained for all participants. The protocol is in Supplement 1. The study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.
Participants and Procedures
To be included, therapists needed 15 or more historical cases with outcomes data to establish their pretrial performance profiles (exceeding the recommended minimum of 5 cases)19 and needed to agree to keep roster openings until meeting their target number of trial cases (approximately 6). Therapists provided written informed consent before treating patients in both matched and CAU conditions and were unaware of their patients’ condition assignments. This design minimized administrative disruptions and allowed us to test the within-therapist effect of matching—within a given therapist’s study-based caseload, did patients who were matched have better outcomes than those who were not matched?
All adult patients aged 18 to 70 years who naturally presented during the trial period were eligible, as the only exclusion criterion was not making one’s own MHC decisions. Once eligibility was confirmed, an intake specialist presented study information, including the focus on examining different case-assignment methods (although both therapists and patients would be unaware of what those methods were). Interested participants received a link to an online consent form and baseline TOP assessment to inform the match system. The project coordinator then allocated consenting patients to a condition based on a preestablished randomization sequence created with an online generator, which was concealed until participants were enrolled. The project coordinator was unaware of therapists’ effectiveness report cards. Demographic characteristics were collected for both patients and therapists by self-reporting. Race/ethnicity and sexual orientation were self-reported according to predefined categories. Reporting race/ethnicity and sexual orientation data allowed us to provide information about the generalizability of the results in accordance with the Patient-Centered Outcomes Research Institute’s methodology standards.
Pretrial, enrolled therapists were classified as effective, neutral, or ineffective on each of 12 outcome domains. This classification was based on an established procedure that involved several steps.5,10,12 First, in a reference sample of approximately 28 000 adult outpatients receiving naturalistic MHC in diverse settings (ie, outpatient clinics, hospitals, residential centers, or day treatments), machine learning determined the most important patient-level predictors of TOP-based change from pretreatment to posttreatment. For each TOP domain, the resulting best-fitting model generated normative, risk-adjusted change rates. Second, for any new patient, the algorithm compared their personally expected change, based on the aforementioned risk-adjusted norms, with their actual change on each TOP domain. Finally, these data were examined at the therapist level. For each therapist and each TOP domain, we calculated an 86% CI for the patients’ mean difference from the risk-adjusted expected outcomes. Then, we used the reliable change index to determine whether therapists’ domain-specific mean patient change rates exceeded the scale’s measurement error.20 Therefore, within each domain, an effective therapist was one whose patients, on average, reliably exceeded their expected outcome, a neutral therapist was one whose patients, on average, generally neither exceeded nor fell short of their expected improvement (ie, they changed as would be predicted within the 86% CI), and an ineffective therapist was one whose patients, on average, reliably fell short of their expected change. With the match system primed, new trial patients completed the TOP at intake to inform the intervention. Patients in the matched group were then assigned to a therapist based on an algorithm-generated shortlist of multiple clinicians according to 5 match levels, ranging from highest to lowest:
Therapist effective on patient’s 3 most elevated domains, not ineffective on any domain.
Therapist effective on patient’s single most elevated domain, not ineffective on any domain.
Therapist effective on patient’s 3 most elevated domains, ineffective on at least 1 other.
Therapist effective on patient’s single most elevated domain, ineffective on at least 1 other.
Therapist not effective on any elevated domain, but also not ineffective on any domain.
With the shortlist, the project coordinator could navigate from highest to lowest match until there was a therapist who met other necessary parameters for a patient (eg, accepted their insurance). Patients in the CAU group were assigned to therapists through typical pragmatic procedures, such as therapist availability. Following assignment, treatment was delivered naturalistically across both conditions.
The TOP,18 which was used both for pretrial therapist classification and for 2 primary outcomes during the trial, includes 58 items assessing 12 symptomatic or functional domains in terms of how much of the time the person has experienced each of the following specific concerns in the past 2 weeks on a scale ranging from 0 (none) to 6 (all): depression, quality of life, mania, panic or somatic anxiety, psychosis, substance misuse, social conflict, sexual functioning, sleep, suicidality, violence, and work functioning. The general impairment index represents the mean of the domain-specific z scores (ie, SD units relative to the general population mean). The domain-specific impairment index represents individual z scores throughout treatment for each patient’s most elevated baseline clinical scale. Higher scores represent greater impairment (eg, 0 represents the general population mean, 2 represents impairment 2 SDs above the general population mean). The Symptom Checklist-10 (SCL-10)21 was used to assess global psychological distress. The 10 items are rated from 0 to 4 with a total score range of 0 to 40 (higher scores indicate greater distress). Patients completed the TOP and SCL-10 at baseline and biweekly, either through actual termination (ie, end of their treatment) or a maximum of 16 treatment weeks.
A power analysis using the formula presented by Raudenbush and Liu,22 as incorporated in the Optimal Design program version 3.01,23 revealed that we needed 44 therapists and 211 patients to achieve a power of 0.80 to detect moderate condition effects (d = 0.50) on linear outcome change rates. Accounting for 25% attrition, our enrollment target was 281 patients. For this naturalistic trial, attrition connoted unusable cases in the primary analyses, that is, patients who consented but either actively withdrew consent to analyze their data after starting treatment, passively withdrew by never beginning treatment or receiving a therapist assignment, or provided no outcomes data beyond baseline (lost to follow-up).
To test condition effects, we used hierarchical linear modeling (HLM),24 as facilitated by HLM software version 7 (Scientific Software International).25 To address missing data, HLM uses maximum likelihood estimation,24,26 resulting in a modified intent-to-treat (mITT) sample for our primary analyses, whereby all participants were retained who completed at least 1 outcome assessment beyond baseline, which equated to all nonattrited patients as per the 3 attrition categories described above and in Figure 1. (Had any randomized patients failed to start treatment but completed outcome assessments within the study period, we would have included them; however, there were no such cases.) Specifically, 3-level HLMs estimated within-patient (level 1), between-patient (level 2), and between-therapist (level 3) differences. At level 1, a linear change trajectory was fit to each individual’s outcome scores. Because of natural variability in treatment length, we centered time at baseline so that the intercept represented patients’ outcome level at the time point when all patients had a score. Random patient-level intercepts and slopes and a random therapist-level intercept were included across all models. Therapist-level random slopes were included if they were significant and/or improved model fit. After selecting the best-fitting growth model, we added assignment condition as a level 2 predictor of between-patient (within-therapist) differences in outcome, while accounting for global between-therapist differences at level 3. eAppendix 1 in Supplement 2 shows the equation. Results are presented with 95% CIs and 2-tailed P values. Significance was set at P < .05. Effect size is represented as Cohen d modified for a multilevel context,25 which represents the number of SDs on the patient-level outcome by which the 2 groups differed.
Of 48 therapists, 35 (73%) were female. Mean (SD) age was 48.0 (14.0) years. Therapists treated a mean (SD) of 4.54 (2.42) patients, with influences from varied theoretical orientations (Table 1). Therapists’ pretrial effectiveness report cards were based on pre-post TOP data from a mean (SD) of 28.48 (3.00) patients. Overall, therapists had a mean (SD) of 1.56 (1.66) strengths (domains classified as effective) and 0.96 (1.65) weaknesses (domains classified as ineffective; no therapists were ineffective on 9 or more domains); 42 (87.5%) had at least 1 strength on which they could be matched or at least 1 weakness that could be avoided, and 6 (12.5%) were classified as neutral (neither effective nor ineffective) across all domains, allowing them to be matched at level 5 (eTables 1 and 2 in Supplement 2). Only 5 therapists (10.4%) were ineligible to treat patients in the match condition (ie, they were not classified as effective on any domains and were ineffective in at least 1 domain; eAppendix 2 in Supplement 2 shows analyses investigating the effect of this subgroup on the results, which was negligible).
Patient Flow and Descriptive Information
As per Figure 1, 288 patients were randomized after providing consent during the naturalistic intake process. Despite the minimal ask of eligible patients at baseline, of those who declined to participate, most simply wanted to be assigned to a therapist without having to leave the intake call to go online to consent and complete the TOP (moreover, without knowing the nature of the assignment methods being tested, patients had limited incentive to participate outside of financial compensation). Importantly, though, the study sample was closely aligned with the participating MHC network’s average utilization data with regard to demographic characteristics, thereby increasing confidence in its representativeness. After allocation, 70 patients attrited for a final mITT sample of 218 for our primary analyses. This effective sample did not differ from attrited patients on clinical and demographic variables (eAppendix 3 in Supplement 2). Table 2 presents baseline patient characteristics for the mITT sample. Patients in the CAU (n = 119) and matched (n = 99) conditions did not differ on any demographic or clinical variables (eAppendix 3 in Supplement 2). Among 218 patients, 147 (67.4%) were female, and 193 (88.5%) were White. The mean (SD) age was 33.9 (11.2) years. Patients provided a median (SD) of 12.14 (6.10) weeks of data, and 83 (38.1%) provided data through the full 16 treatment weeks.
Consistent with the intended manipulation, compared with patients in the CAU group and to a large degree, significantly more patients in the matched group were matched at higher levels (χ24 = 49.47; P < .001; Cramer V = 0.48). Specifically, 43 patients in the CAU group (36.1%) were assigned to a therapist who was not a match at any level and was ineffective in at least 1 problem domain. Of the remaining 76 patients in the CAU group who received a chance match, 55 (72.4%) were matched at level 5, 2 (2.6%) at level 4, none at level 3, 17 (22.4%) at level 2, and 2 (2.6%) at level 1. In the match condition, 58 patients (58.6%) were matched at level 5, 4 (4.0%) at level 4, none at level 3, 28 (28.3%) at level 2, and 9 (9.1%) at level 1 (eFigure in Supplement 2). Finally, we tested the effects of different match levels on general impairment (eAppendix 4 in Supplement 2); results indicated that each level of matching outperformed no match, with higher levels showing the largest effects: d = 1.25 when patients were matched with therapists effective in treating the patients’ top 3 elevated domains, d = 1.00 when patients were matched with therapists effective at treating the patients’ single-most elevated domains, and d = 0.75 when patients were matched with therapists not ineffective in any domain.
All outcomes were normally distributed except domain-specific impairment, which was positively skewed (skewness = 2.12). Given that this variable included both negative and positive numbers, we added a constant before log transforming it (skewness = −0.18). Also, patients for whom substance misuse, suicidal ideation, or violence was the most elevated domain had more extreme values than patients for whom the most elevated domain was 1 of the other 9. Therefore, in the domain-specific impairment model, we included as covariates dummy-coded indicators of when a participant’s most elevated problem was substance misuse, suicidal ideation, or violence. For context, there were no condition differences in the mean (SD) number of sessions attended (match: 5.80 [3.54]; CAU: 5.61 [3.02]; t216 = −0.42; SE = 0.44; P = .68) or in the mean (SD) number of weeks in the study (match: 11.09 [5.99]; CAU: 11.81 [6.20]); t216 = 0.61; SE = 0.83; P = .39).
Condition Effects on Primary Outcomes
Table 3 shows all fixed effects, and eTable 3 in Supplement 2 shows all random effects and model fit information. To a moderate to large degree, patients in the matched group experienced greater weekly reductions in their general impairment compared with patients in the CAU group (γ110 = −0.03; 95% CI, −0.05 to −0.01; P = .02; patient-level d = 0.75). Figure 2 shows that this effect was clinically significant with the matched patients, on average, ending treatment in the nonclinical range (ie, average week 17 model-estimated z score = −0.03, or below the mean impairment level shown by individuals in non–treatment-seeking samples), and patients in the CAU group, on average, ending treatment in the clinically impaired range (ie, average week 17 model-estimated z score = 0.35). To a moderate degree, patients in the matched group experienced greater weekly reductions in psychological distress compared with patients in the CAU group (γ110 = −0.16; 95% CI, −0.30 to −0.02; P = .03; patient-level d = 0.50). Finally, to a moderate degree, patients in the matched group experienced greater weekly reductions in domain-specific impairment compared with patients in the CAU group (γ110 = −0.01; 95% CI, −0.01 to −0.006; P = .01; patient-level d = 0.60), controlling for which domain was elevated. No adverse events were reported.
To ensure that our results were not unduly influenced by our preregistered analytic plan for using the mITT sample, we conducted a supplemental analysis for the general impairment outcome. For this model, we included 48 of 51 patients who were lost to follow-up (3 of these patients did not complete the TOP at baseline and therefore had no outcomes data to include) for a sample size of 266. (For ethical reasons, we did not include the 4 patients who withdrew consent to analyze their data; we also could not include the 15 patients who never began treatment or received a therapist assignment, as these patients had no therapist identification variable.) For this sample of 266 patients, the match effect on general impairment reduction remained the same (γ110 = −0.03; 95% CI, −0.05 to −0.01; P = .01).
In this 2-arm, double-blind randomized clinical trial, we tested the effectiveness of a personalized match system based on therapists’ effectiveness report cards vs CAU prior to naturalistic MHC. As predicted, matched patients demonstrated greater reductions in 3 outcomes compared with patients in the CAU group. With most therapists treating patients in both conditions, our multilevel models tested a within-therapist effect of patient-specific matching, while accounting for general between-therapist differences in effectiveness across all patients in the therapists’ caseloads. That is, relative to their own average outcome, a given therapist achieved better outcomes when treating a matched patient vs a CAU patient. Thus, measurement-based matching represents a readily scalable innovation that can complement other evidence-based efforts to improve MHC.
These results suggest that measurement-based therapist report cards can help redirect MHC toward therapists’ strengths.11 Analogous to specialization movements in medicine,27,28 such precision care in mental health can have a substantive positive effect. Importantly, dispelling the possibility that the match effect was solely a function of shared method variance, TOP-based matching also had a positive effect on SCL-10–based distress.
Notably, the good fit in this study came not from changing what the therapists did in their treatment, but rather who they treated. Capitalizing on whatever it is that a therapist historically does well when treating patients with certain mental health problems, the current data indicate that our match system can improve the effectiveness of that care, even with neither therapist nor patient being aware of their match status. To be used in this masked manner, the match system only requires that a multidimensional outcome tool be administered at baseline and follow-up—a practice that is becoming more common29—so that therapist performance data can prime matched case assignment without threatening clinician treatment autonomy. That said, it is plausible (although it requires testing) that there might be an even bigger effect were patients and therapists to be aware of their match status, perhaps through additive mechanisms, such as increased outcome expectation.30 Whether masked or aware, patients have indicated that a MHC system’s strategic use of therapist performance information would be a highly valued action.16,17
This study had limitations. First, the naturalistic design resulted in limited information on treatment, and future research should examine what it is that well-performing therapists do when treating patients in a given strength domain. Second, the setting limited our control over a balanced design in which each therapist saw an equal number of patients in the matched group and in the CAU group. Therefore, we cannot fully rule out that between-therapist differences in effectiveness could have influenced the results, although both our crossed design and supplemental analyses suggested that this was unlikely. Third, with only 22% of eligible patients participating, it is possible this subgroup differed from the general population of treatment-seeking adults, which could reduce generalizability. However, it is plausible that participation was restricted simply because of the limited incentive to participate in research. Thus, future implementation work should examine ways in which multidimensional assessment can be most effectively incorporated into a system’s modus operandi. Fourth, although all clinics in this trial were part of a single health care system, and clinicians may have worked fluidly between them in some instances, it is still possible that there were clinic-level effects on matching. Such additional nuance should be the focus of future work. Fifth, generalizability beyond a predominantly White, predominantly heterosexual sample within a specific MHC system may be limited. Sixth, although matching can theoretically be facilitated by any multidimensional outcome tool, this study only tested 1. Seventh, we report here only the main effects of 1 type of matching; future research should focus on potential moderators of (eg, patient motivation) or additions to (eg, also matching on cultural identities) the match system and other match or precision innovations in general.
This trial established a minimalist method of precision MHC. Namely, MHC can be substantially improved by using therapist performance data to determine who they treat. This method provides stakeholders (ie, patients, therapists, and administrators) a choice (as a shortlist) for optimizing care beyond chance levels, while also minimizing ineffectiveness.
Accepted for Publication: April 18, 2021.
Published Online: June 9, 2021. doi:10.1001/jamapsychiatry.2021.1221
Corresponding Author: Michael J. Constantino, PhD, Department of Psychological and Brain Sciences, University of Massachusetts, Amherst, 612 Tobin Hall, 135 Hicks Way, Amherst, MA 01003-9271 (firstname.lastname@example.org).
Author Contributions: Dr Constantino had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Constantino, Boswell, Kraus.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Constantino, Boswell, Coyne, Swales.
Critical revision of the manuscript for important intellectual content: Constantino, Boswell, Coyne, Kraus.
Statistical analysis: Constantino, Boswell, Coyne.
Obtained funding: Constantino, Boswell.
Administrative, technical, or material support: All authors.
Supervision: Boswell, Kraus.
Conflict of Interest Disclosures: Dr Constantino reports a grant from Patient-Centered Outcomes Research Institute and nonfinancial support from a data and safety monitoring board during the conduct of the study. Dr Boswell reports grant support from PCORI paid to the University of Massachusetts. Ms Coyne reports consulting fees from Center for Research on Families during the conduct of the study. Dr Swales reports grant support from PCORI paid to the University of Massachusetts. Dr Kraus reports grant support from PCORI paid to the University of Massachusetts; being the founder, president, and chief scientific officer of Outcome Referrals, Inc, which owns and processes the Treatment Outcome Package; and is the inventor of US Patent No. 7,873,525 entitled, “Method and system for outcome based medical referrals.” No other conflicts were reported.
Funding/Support: Research reported in this article was funded through a Patient-Centered Outcomes Research Institute Award (IHS-1503-28573).
Role of the Funder/Sponsor: The funder provided general administrative consultation over the conduct of the study. The funder was not involved in the design of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The statements in this article are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute, or its Board of Governors or Methodology Committee.
Meeting Presentation: This work was accepted for presentation at the International Society for Psychotherapy Research (SPR) Annual Conference; June 2020; Amherst, Massachusetts. (Conference canceled due to COVID-19).
Data Sharing Statement: See Supplement 3.
Additional Contribution: Felicia Romano, BS, University of Massachusetts, Amherst, served as the project coordinator and was essential to participant recruitment, data collection, and management. Ms Romano was compensated for her work. The study was conducted at Psychological and Behavioral Consultants in the greater Cleveland, Ohio, area.
Institute of Medicine. Improving the Quality of Health Care for Mental and Substance-Use Conditions. National Academies Press; 2006.
World Health Organization. ICD-11: International Statistical Classification of Diseases and Related Health Problems. 11th ed. World Health Organization; 2019.
MJ. The efficacy and effectiveness of psychotherapy. In: Lambert
MJ, ed. Bergin & Garfield’s Handbook of Psychotherapy and Behavior Change. 6th ed. Wiley; 2013:169-218.
DC. Removing very low-performing therapists: a simulation of performance-based retention in psychotherapy. Psychotherapy (Chic)
. 2015;52(3):329-336. doi:10.1037/pst0000023
MJ. Ethical implications of routine outcomes monitoring for patients, psychotherapists, and mental health care systems. Psychotherapy (Chic)
. 2019;56(4):459-469. doi:10.1037/pst0000246
DR. Accuracy of therapist self-perceived effectiveness as a determinant of between-therapist effects. Paper presented at: Annual meeting of the Society for Psychotherapy Research; July 4, 2019; Buenos Aires, Argentina.
R. Mental health care consumers’ relative valuing of clinician performance information. J Consult Clin Psychol
. 2018;86(4):301-308. doi:10.1037/ccp0000264
JR. Validation of a behavioral health treatment outcome and assessment tool designed for naturalistic settings: the Treatment Outcome Package. J Clin Psychol
. 2005;61(3):285-314. doi:10.1002/jclp.20084
AS. Hierarchical Linear Models: Applications and Data Analysis Methods. 2nd ed. Sage; 2002.
PD. Missing data. In: Marsden
JD, eds. Handbook of Survey Research. 2nd ed. Emerald Group Publishing; 2010:631-657.
et al. Keeping psychologists in the driver’s seat: four perspectives on quality improvement and clinical data registries. Psychotherapy (Chic)
. 2020;57(4):562-573. doi:10.1037/pst0000227
JF. A meta-analysis of the association between patients’ early treatment outcome expectation and their posttreatment outcomes. Psychotherapy (Chic)
. 2018;55(4):473-485. doi:10.1037/pst0000169