Investigation of Active Ingredients Within Internet-Delivered Cognitive Behavioral Therapy for Depression: A Randomized Optimization Trial

Key Points

Question  Which specific components within internet-delivered cognitive behavioral therapy (CBT) for depression are active ingredients that reduce symptoms?

Findings  In this randomized optimization trial that included 767 adults with depression, 6 treatment components (activity scheduling, thought challenging, relaxation, concreteness training, functional analysis, and self-compassion training) did not show a significant main effect on depression. However, the presence of the absorption component outperformed its absence in reducing depression at 6-month follow-up.

Meaning  The findings suggest that the majority of treatment benefit from internet-delivered CBT is likely to accrue from factors common to all CBT components and/or from generic factors common to all therapies, with the possible exception of absorption.

Importance  There is limited understanding of how complex evidence-based psychological interventions such as cognitive behavioral therapy (CBT) for depression work. Identifying active ingredients may help to make therapy more potent, brief, and scalable.

Objective  To test the individual main effects and interactions of 7 treatment components within internet-delivered CBT for depression to investigate its active ingredients.

Design, Setting, and Participants  This randomized optimization trial using a 32-condition, balanced, fractional factorial optimization experiment (IMPROVE-2) recruited adults with depression (Patient Health Questionnaire−9 [PHQ-9] score ≥10) from internet advertising and the UK National Health Service Improving Access to Psychological Therapies service. Participants were randomized from July 7, 2015, to March 29, 2017, with follow-up for 6 months after treatment until December 29, 2017. Data were analyzed from July 2018 to April 2023.

Interventions  Participants were randomized with equal probability to 7 experimental factors within the internet CBT platform, each reflecting the presence vs absence of specific treatment components (activity scheduling, functional analysis, thought challenging, relaxation, concreteness training, absorption, and self-compassion training).

Main Outcomes and Measures  The primary outcome was depression symptoms (PHQ-9 score). Secondary outcomes include anxiety symptoms and work, home, and social functioning.

Results  Among 767 participants (mean age [SD] age, 38.5 [11.62] years; range, 18-76 years; 635 women [82.8%]), 506 (66%) completed the 6-month posttreatment follow-up. On average, participants receiving internet-delivered CBT had reduced depression (pre-to-posttreatment difference in PHQ-9 score, −7.79 [90% CI, −8.21 to −7.37]; 6-month follow-up difference in PHQ-9 score, −8.63 [90% CI, −9.04 to −8.22]). A baseline score–adjusted analysis of covariance model using effect-coded intervention variables (−1 or +1) found no main effect on depression symptoms for the presence vs absence of activity scheduling, functional analysis, thought challenging, relaxation, concreteness training, or self-compassion training (posttreatment: largest difference in PHQ-9 score [functional analysis], −0.09 [90% CI, −0.56 to 0.39]; 6-month follow-up: largest difference in PHQ-9 score [relaxation], −0.18 [90% CI, −0.61 to 0.25]). Only absorption training had a significant main effect on depressive symptoms at 6-month follow-up (posttreatment difference in PHQ-9 score, 0.21 [90% CI, −0.27 to 0.68]; 6-month follow-up difference in PHQ-9 score, −0.54, [90% CI, −0.97 to −0.11]).

Conclusions and Relevance  In this randomized optimization trial, all components of internet-delivered CBT except absorption training did not significantly reduce depression symptoms relative to their absence despite an overall average reduction in symptoms. The findings suggest that treatment benefit from internet-delivered CBT probably accrues from spontaneous remission, factors common to all CBT components (eg, structure, making active plans), and nonspecific therapy factors (eg, positive expectancy), with the possible exception of absorption focused on enhancing direct contact with positive reinforcers.

Trial Registration  isrctn.org Identifier: ISRCTN24117387

Major depressive disorder is a common psychiatric disorder¹ ranked as the second leading contributor of years lived with disability.² Despite antidepressant medication and cognitive behavioral therapy (CBT) being the best evidence-based treatments, they are associated with remission rates of under 33% and limited sustained recovery (50%-80% relapse or recurrence).³

Understanding the active mechanisms of how psychotherapy works is a priority to advance treatment efficacy^3-8 by enabling the content and delivery of interventions to be optimized to make them more potent and efficient. Psychological treatments are complex interventions made up of multiple components and varying in structure and modality of delivery, each of which potentially influences outcome. While the parallel-group comparative randomized clinical trial (RCT) is the gold standard for establishing the relative efficacy of one intervention vs another or a control, this experimental design is limited at investigating the specific mechanisms of how complex psychological interventions work because it can only compare the overall effects of each intervention package. All treatment components are aggregated and confounded together in the comparison of treatment packages, such that the main effects and interactions of individual treatment components cannot be disentangled. To address this, we need alternatives to the parallel-group comparative RCT, that is, equally rigorous experimental designs that enable testing with causal inference of the effects of the presence or absence of individual therapeutic elements within a complex intervention.^8-10

One alternative is a factorial optimization trial framed within the multiphase optimization strategy (MOST).^10-19 MOST is a principled and comprehensive framework for optimizing and evaluating interventions,^11-15 is well-validated, and has proven value in health behavior research.^11,20-26 MOST uses efficient experimentation before a parallel-group comparative RCT, typically via factorial designs,^13-15,27 to identify which of a set of candidate components is effective and should be retained in an optimized intervention to be evaluated subsequently. Factorial experiments allow one to explore main effects of components and interactions among components within a treatment package,^27,28 which is necessary for developing a mechanistic understanding of therapy and for methodically enhancing and simplifying complex interventions.²⁸

Factorial designs have examined different types of support during psychological interventions,^29,30 but none to our knowledge have investigated treatment components within CBT for adult depression. This trial was, to our knowledge, the first factorial experiment to examine the efficacy of components within internet-delivered CBT for depression.^31,32 The component selection reflected well-established treatment elements within CBT as identified by the Delphi technique³³ (activity monitoring and scheduling, thought challenging, and applied relaxation), functional analysis as a mainstay of behavioral activation,^34-36 and recent innovations (concreteness, compassion, and absorption).^37-40 Internet-delivered CBT was selected for treatment reach and scalability, to reduce drift from treatment protocols, and to ensure patients only received the relevant standardized treatment content, minimizing the risk of patients receiving unallocated components.

This factorial approach enables an enhanced test of the relative contribution of specific vs common treatment factors¹⁰ within psychotherapy without requiring a prohibitive number of participants for statistical power. An unresolved debate concerns the extent to which psychotherapies work through nonspecific factors shared across all therapies (ie, pan-therapy common factors, such as hope and therapeutic alliance) vs elements specific to a particular therapy,^41-43 whether using specific strategies (eg, thought challenging) or features common to all components in that therapy (in CBT, the cognitive behavioral model and homework). Disentangling specific from common treatment effects has been limited by a lack of well-powered experimental studies⁴³ and difficulties in creating appropriate placebo controls that genuinely match an active psychotherapy for credibility and structural equivalence.⁴⁴ Well-designed factorial trials can address this issue. In a balanced factorial experiment, such as the current design, for any component (eg, relaxation), the aggregate of the 16 conditions in which it is present are equivalent for treatment credibility, structure, delivery, rationale, therapist contact, all other components, and therapist allegiance with the aggregate of the 16 conditions in which it is absent (Table 1). Thus, any observed main effect would be conservative evidence for a specific factor driving therapeutic benefit beyond pan-therapy common factors and factors common to all CBT components.

By comparing the presence vs absence of each component, this factorial design examines the main effect of each component on outcomes. Consistent with the Pareto principle and prior studies,²¹ we hypothesized that components and interactions would vary in effect size, with many having insignificant effects (ie, not all specific components are active ingredients in CBT).

This study was a stratified, block-randomized, single-blind, optimization trial randomized at the patient level (ISRCTN24117387). Ethical approval was provided by the National Health Service (NHS) research ethics board. Written informed consent was obtained from participants. The trial was conducted between July 7, 2015, and December 29, 2017, and followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.⁴⁵

The trial design included 7 experimental factors, each corresponding to a CBT component, and each factor had 2 levels (presence of component coded as +1 vs absence coded as −1; ie, effect coded). We used a balanced fractional factorial design (2IV^7,2) (Table 1), which is a special case of the factorial design in which logistics and expense are managed by including only a carefully selected subset of experimental conditions. Only 25% of the conditions required by a full factorial were included. The experiment was designed without a no-treatment control condition, making it suitable for implementation in a clinical service. Fractional factorial designs alias, or combine, effects; each main effect is aliased with selected 3-way interactions or higher-order interactions. To interpret any observed effects, we assumed that prespecified 2-way interactions and all 3-way and higher-order interactions were negligible in size. For each main effect, half of the sample was randomized to presence and half was randomized to absence of the factor (Table 1). A factorial experiment should not be considered a multi-arm parallel-group comparative group RCT because its logic is different. The full sample size is used to determine each main effect and interaction, making it highly efficient for power and sample size because each effect estimate involves all the conditions (half of the cells aggregated for presence of component and half of the cells aggregated for absence of component). Full details of the trial design and protocol are given in Watkins et al³¹ and Supplement 1.

Participants were recruited via a UK NHS Improving Access to Psychological Therapies service or direct internet self-referral. Eligibility criteria were age 18 years or older, registered with a general practice, meeting criteria for depression operationalized by Patient Health Questionnaire–9 (PHQ-9) scores of 10 or higher,⁴⁶ not currently receiving psychotherapy, and if receiving antidepressant medication, the dose was stable for at least 4 previous weeks (eAppendix 1 in Supplement 2). All eligible participants were invited to a telephone-delivered screening and baseline interview including the Structured Clinical Interview for DSM-5 sections on current and past depressive episodes conducted by trained research assistants and to complete self-report questionnaires via online survey, email, or mail. Similar follow-up assessments took place at 3 months (posttreatment) and 6 months after randomization.

All eligible participants who provided written informed consent were given access to an introductory internet-delivered CBT module, which included a mood diary and basic psychoeducation about depression. Only participants who completed this module were randomized to ensure participants tried and were engaged with internet-delivered CBT before committing. Participants were randomized with equal probability to all 32 conditions by a permuted block randomization program delivered independently by the Peninsula Clinical Trials Unit. Randomization was stratified by severity of current depression (moderate [PHQ-9 score 10 to <20] vs severe [PHQ-9 score ≥20), antidepressant use (receiving vs not receiving British National Formulary–recommended therapeutic dose), and referral source (NHS vs internet self-referral). All outcome assessors and data analysts were blind to treatment allocation. Statistical analyses were carried out blinded for randomization.

The intervention was hosted on an established, secure internet treatment platform. Participants received between 1 and 7 specific components (mean, 3-4) organized into discrete modules depending on the randomized condition. Therapists provided brief written online asynchronous feedback for each module to improve retention and adherence.⁴⁷ To ensure treatment integrity and fidelity, each patient could access only the specific modules allocated, constraining patients and therapists to the relevant treatment protocol. We counterbalanced the sequential order of the modules across all conditions to ensure that each component occurred equally early or late in therapy across participants.

The 7 component modules reflected core elements identified within CBT⁴⁸ for depression and CBT innovations,^37-40 with each hypothesized to specifically target distinct mechanisms. Within a behavioral model of CBT, activity scheduling and absorption training were hypothesized to increase response-contingent positive reinforcement³⁵ by respectively increasing frequency of and direct contact with positive reinforcers,⁴⁹ and functional analysis was proposed to target habitual avoidance and rumination by identifying antecedent cues, controlling exposure to these cues, and practicing alternative responses to them.⁵⁰ Within the cognitive model, thought challenging and concreteness training³⁸ were hypothesized to reduce negative thinking and overgeneralization cognitive bias, respectively, which are both characteristic of depression.^48,51 Within the emotional regulation model, relaxation was hypothesized to target physiological arousal and tension, while self-compassion training was hypothesized to activate the soothing and safeness emotional system and, thereby, reduce negative mood.^52-55 eAppendix 2 in Supplement 2 gives further details.

The primary outcome was depression symptoms (PHQ-9 score⁴⁶) measured at 3 months postrandomization and 6 months posttreatment follow-up. Secondary trial outcomes were anxiety severity (Generalized Anxiety Disorder–7 score⁵⁶) and social, home, and work functioning (Work and Social Adjustment Scale [WSAS] score⁵⁷).

Adherence for each component was defined a priori as completing the relevant online module. Questionnaires to examine potential mediators are described in eAppendix 3 in Supplement 2. Ethnicity was self-classified by participants using drop-down menus (options were African/African American/African Caribbean, Asian/Asian American/Asian British, Hispanic/Latinx, White, and mixed), with the opportunity to self-identify ethnicity in open text. Because the numbers in each of the categories other than White were small, these were collapsed into an “other” category (eAppendix 4 in Supplement 2). Ethnicity was included so we could examine whether the samples recruited from online recruitment and within the Improving Access to Psychological Therapies service were similar or differed. This was done for all demographic and background variables (eAppendix 5 in Supplement 2).

We assumed the smallest meaningful clinical important difference would be a small effect size (Cohen d or standardized mean difference of 0.2⁴³) for the main effect of a component or interaction between components on the before treatment to after treatment change in depression. To detect a Cohen d of 0.20 with 80% power at α = 0.10 (recommended to decrease the relative risk of type II to type I error and of prematurely ruling out potentially active components^11,12,18,), a sample size of 632 was required. Estimating 40% dropout attrition posttreatment, we required a sample size of 1056 for analysis of variance; with multiple measures on the primary outcome, a growth curve model required 30% to 50% fewer participants for the same power,⁵⁸ giving a conservative estimate of 736 participants.

Data were analyzed from July 2018 to April 2023. Participants were analyzed according to their randomization group, including all participants randomized regardless of intervention received or study withdrawal (intention to treat). The 7 factors were effect coded (−1 or +1) and modeled together to study the main effects and interactions independently.¹² Relevant analyses were adjusted for the stratification variables. Main effects and interactions were estimated based on aggregates across experimental conditions (Table 1). For primary and secondary outcomes, we used (1) baseline score–adjusted analysis of covariance (ANCOVA), modeling outcomes posttreatment (3 months postrandomization) and 6 months posttreatment as dependent variables and (2) analyses of maximum likelihood (ML)–based mixed-effects growth curve models including 6 measurements (baseline, end of introductory module, end of module 1, end of module 2, 3 months postrandomization, and 6 months posttreatment). Cohen d effect size was calculated for the adjusted estimates using the samples available for the regression models. Pooled SD was calculated from the change scores of baseline and follow-up measures using the same samples.⁵⁹ Complier average causal effect (CACE) analyses^60,61 were carried out using an instrumental variable method implemented via structural equation modeling to estimate intervention effects accounting for good adherence with the interventions while retaining the benefits of randomization. Results are presented with 90% CIs as we powered at α = 0.10; thus, 2-sided P < .10 was considered statistically significant. Analyses were conducted with Stata, version 17 (StataCorp LLC).⁶²

Between July 7, 2015, and March 29, 2017, 6940 individuals visited the online screener; 1289 fulfilled the inclusion criteria, as did 275 directly referred from clinical service. A total of 987 individuals completed the subsequent telephone screening, and 924 were eligible for the trial. After exclusion of individuals not willing or able to participate, 767 adults were randomized (Figure 1). The mean age (SD) age of participants was 38.5 (11.62) years (range, 18-76 years); 132 (17.2%) were men, 635 (82.8%) were women, 655 (85.4%) were White, and 36 (4.7%) were from other ethnic groups. For each component, half of participants were allocated (382-407 participants), and half were not allocated to it (360-384 participants). eAppendices 4 and 5 in the Supplement 2 show baseline characteristics of participants by factor and recruitment source.

The number of participants not completing follow-up assessment at 3 and 6 months was 286 (37%) and 261 (34%), respectively. Missing rates did not differ between the primary and secondary outcomes or intervention conditions; participants with missing data ranged from 298 to 314 at 3 months and from 267 to 273 at 6-month follow-up. A total of 506 participants (66%) completed the 6-month posttreatment follow-up.

Between 170 of 407 (42%) and 190 of 385 (49%) participants allocated to a treatment module completed it (eAppendix 6 in the Supplement). Fidelity to each component was 100%.

On average, participants receiving internet-delivered CBT had reduced depression. The change in PHQ-9 score from pretreatment to posttreatment was −7.79 (90% CI, −8.21 to −7.37; P < .001), with a recovery rate (PHQ-9 score of <10) of 61.7%; change in PHQ-9 score from pretreatment to 6-month follow-up was −8.63 (90% CI, −9.04 to −8.22; P = .001), with a recovery rate of 71.5%. ANCOVA and ML-based growth curve models gave comparable effect estimates; ANCOVA results are reported herein (eAppendix 7 in Supplement 2 gives ML-based growth curve models). Only absorption training had a significant main effect on depressive symptoms at 6-month follow-up; that is, its presence reduced depression more than its absence (posttreatment difference in PHQ-9 score, 0.21 [90% CI, −0.27 to 0.68; P = .47]; difference in change in PHQ-9 score from pretreatment to 6-month follow-up, −0.54 [90% CI, −0.97 to −0.11; P = .04]; Cohen d = 0.18). No other components had a significant main effect on depression at either posttreatment or 6-month follow-up (posttreatment: largest difference in PHQ-9 score [functional analysis], −0.09 [90% CI, −0.56 to 0.39]; 6-month follow-up: largest difference in PHQ-9 score [relaxation], −0.18 [90% CI, −0.61 to 0.25]) (Table 2, Table 3, and Figure 2). Of a priori 2 × 2 interactions tested, only 2 were significant; at 6-month follow-up, self-compassion × activity scheduling had a positive synergistic interaction (difference in PHQ-9 score, −0.83; 90% CI, −1.28 to −0.38; P < .001), but self-compassion × concreteness training had an antagonistic interaction (difference in PHQ-9 score, 0.45; 90% CI, 0.01-0.90; P = .09). Relative to the average symptom reduction from baseline to 6-month follow-up, absorption contributed 6.4% (0.5/8.79).

Results for secondary outcomes (anxiety and work, home, and social functioning) were similar to those for depression (Table 3 and eAppendix 8 in Supplement 2). There was no main effect of any component for anxiety, and only absorption training had a significant effect on work, home, and social functioning at 6-month follow-up (posttreatment difference in WSAS score, −0.26 [90% CI, −0.98 to 0.45]; P = .10; 6-month follow-up difference in WSAS score, −0.61 [90%CI, −1.23 to −0.01]; P = .54).

eAppendix 9 in Supplement 2 shows the results of effect modification analyses by referral source, depression severity, and prescription of antidepressant medication. The results of the CACE analyses were consistent with those of the intention-to-treat analyses (eAppendix 10 in Supplement 2).

Five participants were hospitalized for suicide attempt or serious self-harm, and none died during the 6-month follow-up period. These events were judged as unrelated to interventions.

This randomized optimization trial in 767 adults with depression symptoms found no significant main effect of 6 components within internet-delivered CBT for depression, anxiety, or work, home, and social functioning posttreatment and at 6-month follow-up. There was a significant but small main effect of absorption training on reducing depression at 6-month follow-up.

To our knowledge, this is the first randomized factorial trial to test the direct effects of components within internet-delivered CBT in a large-scale sample of adults with depression. Internet-delivered CBT had equivalent efficacy with the wider evidence base for CBT for depression, exceeding the mean recovery rates of 51% observed in Improving Access to Psychological Therapies services and matching effects reported in a recent meta-analysis,⁶³ indicating that it was likely efficacious. There was no significant main effect of 6 of 7 treatment components on depression symptoms. These preliminary results suggest that the active ingredients of internet-delivered CBT are more likely to be common treatment factors than specific CBT strategies, consistent with recent arguments.⁴²

The findings of our study suggest that the observed reduction in depressive symptoms is likely due to some combination of spontaneous remission, regression to the mean, the specific components, and the constant component present in all conditions. The constant component includes the introductory module, asynchronous written support from a therapist, monitoring of symptoms, and factors common across all the CBT components, including cognitive behavioral psychoeducation, learning and practicing a new behavioral or cognitive coping strategy, planning, and review of homework. The constant component may engage both pan-therapy common factors (eg, therapeutic alliance, hope, and healthy actions) and factors only common to all CBT components, although we could not distinguish their effects on outcome.

The main effect of absorption training at 6-month follow-up provides, to our knowledge, the first direct causal evidence for a specific treatment component influencing depression outcome, albeit a small, preliminary finding that needs replication. Relative to the average symptom reduction from baseline to 6-month follow-up, absorption contributed 6.4% (0.56/8.79). Absorption focused on increasing contact with positive reinforcers by using principles from flow theory⁴⁹ and encouraging patients to become immersed in their activities through changing their mindset, environment, attentional focus, and selection of activities. These positive effects are consistent with emergent evidence for the specific benefits of behavioral activation and exposure to reward.^64,65 If this observed effect is robust, enhancing absorption within CBT may increase treatment efficacy.

This study has strengths. These include the randomized factorial design, inclusion of participants recruited from both health services and the community, the large sample size, high levels of treatment fidelity, the 6-month follow-up period, measurement of multiple outcomes, and adherence monitoring.

This study also has limitations. First, as defined by our research question, we only examined components within CBT and, thus, could not disentangle pan-therapy common factors from factors common to all CBT components. Second, the constant component may have been too strong, making it hard to detect additional effects of specific factors. Third, the dose of each component may have been insufficient since module completion was under 50% and their sequencing meant that patients only practiced each component for a few weeks. Nonetheless, we had comparable treatment effects to those of other CBT interventions, and rates of treatment completion paralleled those typically found for internet-delivered CBT.⁶⁶ Fourth, we do not know how generalizable internet-delivered CBT is to face-to-face therapy. Fifth, because of the fractional design, the significant main effect of absorption training could be attributable to 1 or more of the interactions with which this main effect was aliased. Sixth, beyond the effects averaged across all patients, there may have been individual differences in response to each component; individuals may have only responded to certain components, and a component may have been positive for some individuals but inactive or iatrogenic for others, producing an overall null effect. Developing reliable treatment rules to predict who responds optimally to which component may enable personalization within internet-delivered CBT to improve outcomes.^67,68

In this randomized optimization trial of adults with depression, of 7 specific components, only absorption may have uniquely contributed to reduced depression at 6-month follow-up. Given this trial’s novelty, limitations, and nonsignificant findings, we cautiously suggest that internet-delivered CBT may reduce depression through an as-yet-undetermined combination of spontaneous remission, pan-therapy common factors, and factors common to all CBT components, although further replication is needed (similar findings have been reported⁶⁹). These findings highlight the potential value of factorial designs in determining how therapies work.

Accepted for Publication: April 22, 2023.

Published Online: June 28, 2023. doi:10.1001/jamapsychiatry.2023.1937

Correction: This article was corrected on September 6, 2023, to fix an error in the Additional Contributions.

Corresponding Author: Edward Watkins, PhD, Sir Henry Wellcome Building for Mood Disorders Research, College of Life and Environmental Sciences, University of Exeter, Perry Road, Exeter EX4 4QG, United Kingdom (e.r.watkins@exeter.ac.uk).

Author Contributions: Dr Watkins and Mr Mostazir had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Watkins, Collins.

Acquisition, analysis, or interpretation of data: Newbold, Tester-Jones, Mostazir.

Drafting of the manuscript: All authors.

Critical revision of the manuscript for important intellectual content: Watkins, Newbold, Collins, Mostazir.

Statistical analysis: Watkins, Collins, Mostazir.

Obtained funding: Watkins.

Administrative, technical, or material support: Newbold, Tester-Jones, Collins, Mostazir.

Supervision: Watkins.

Conflict of Interest Disclosures: Dr Watkins reported receiving royalties from Guilford Press and personal fees from the British Association for Behavioural and Cognitive Psychotherapies and Bespoke outside the submitted work. No other disclosures were reported.

Funding/Support: Funding for this trial was provided by grants from the Cornwall Partnership NHS Foundation Trust and South West Academic Health Science Network (Dr Watkins).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 3.

Additional Contributions: We thank Jenny Cadman, DClinPsy (Great Yarmouth and Waveney Youth Team, Norfolk and Suffolk NHS Foundation Trust), and Carly Morriss (formerly Thornton), DClinPsy (now at the Adult Eating Disorder Service), for their role as trial therapists. Both were formerly employed at the College of Life and Environmental Sciences, University of Exeter, and were paid as part of their employment for involvement in the trial. We also thank all the BeMe staff for their support in the delivery of the trial.