Efficacy of Self-guided Internet-Based Cognitive Behavioral Therapy in the Treatment of Depressive Symptoms: A Meta-analysis of Individual Participant Data

Key Points

Questions  Is self-guided internet-based cognitive behavioral therapy effective in treating depressive symptoms and which variables moderate treatment outcome?

Findings  In this meta-analysis of individual participant data from 3876 adults, internet-based cognitive behavioral therapy was more effective compared with controls. Adherence predicted better treatment outcomes within the experimental condition.

Meaning  Self-guided internet-based cognitive behavioral therapy may be a viable alternative to current first-step treatment approaches for symptoms of depression, particularly in those individuals who are not willing to have any therapeutic contact.

Importance  Self-guided internet-based cognitive behavioral therapy (iCBT) has the potential to increase access and availability of evidence-based therapy and reduce the cost of depression treatment.

Objectives  To estimate the effect of self-guided iCBT in treating adults with depressive symptoms compared with controls and evaluate the moderating effects of treatment outcome and response.

Data Sources  A total of 13 384 abstracts were retrieved through a systematic literature search in PubMed, Embase, PsycINFO, and Cochrane Library from database inception to January 1, 2016.

Study Selection  Randomized clinical trials in which self-guided iCBT was compared with a control (usual care, waiting list, or attention control) in individuals with symptoms of depression.

Data Extraction and Synthesis  Primary authors provided individual participant data from 3876 participants from 13 of 16 eligible studies. Missing data were handled using multiple imputations. Mixed-effects models with participants nested within studies were used to examine treatment outcomes and moderators.

Main Outcomes and Measures  Outcomes included the Beck Depression Inventory, Center for Epidemiological Studies–Depression Scale, and 9-item Patient Health Questionnaire scores. Scales were standardized across the pool of the included studies.

Results  Of the 3876 study participants, the mean (SD) age was 42.0 (11.7) years, 2531 (66.0%) of 3832 were female, 1368 (53.1%) of 2574 completed secondary education, and 2262 (71.9%) of 3146 were employed. Self-guided iCBT was significantly more effective than controls on depressive symptoms severity (β = −0.21; Hedges g  = 0.27) and treatment response (β = 0.53; odds ratio, 1.95; 95% CI, 1.52-2.50; number needed to treat, 8). Adherence to treatment was associated with lower depressive symptoms (β = −0.19; P = .001) and greater response to treatment (β = 0.90; P < .001). None of the examined participant and study-level variables moderated treatment outcomes.

Conclusions and Relevance  Self-guided iCBT is effective in treating depressive symptoms. The use of meta-analyses of individual participant data provides substantial evidence for clinical and policy decision making because self-guided iCBT can be considered as an evidence-based first-step approach in treating symptoms of depression. Several limitations of the iCBT should be addressed before it can be disseminated into routine care.

Many studies^1-4 have found that depressive symptoms can be effectively treated with psychotherapy, pharmacotherapy, or both. Nevertheless, many people with depressive symptoms do not seek help, and even well-resourced health care systems find it difficult to marshal enough qualified therapists to offer psychological interventions. Access barriers to psychotherapy include limited availability of trained clinicians, high cost of treatment, and fear of stigmatization.^5-8 As a consequence, a significant number of individuals with depressive symptoms remain untreated.^9,10

Self-guided internet-based cognitive behavioral therapy (iCBT) without therapist support can allow physicians, such as general practitioners, to provide easy and affordable access to psychological treatments and reduce the cost of such treatments. A meta-analysis¹¹ found a small but significant effect size of self-guided iCBT compared with control conditions. However, recent large trials found a range of effects, varying from small to moderate effect sizes^12,13 to no effect.^14,15 These contradicting findings drew much attention and raised concerns about the benefits of these interventions.

Randomized clinical trials (RCTs) and study-level systematic reviews often lack adequate power and precision in their estimates. Statistically underpowered samples also preclude identification of clinically useful moderators or predictors of treatment outcome.¹⁶ Meta-analyses using individual participant data (IPD) estimate aggregate effect sizes using IPD from RCTs. The IPD maximize power to detect a true effect while allowing the exploration of study variability (eg, level of support, treatment adherence, setting) and participant characteristics as moderators of treatment outcome. The present study reports the results of an IPD meta-analysis of trials on self-guided iCBT for adult depressive symptoms compared with control conditions. The term self-guided iCBT is defined as CBT delivered via the internet, which may involve automated feedback but does not provide support related to the therapeutic content.

Studies were included if the participants were adults (aged >18 years) with elevated symptoms of depression based on any diagnosis or any self-report scale of depression. Only those RCTs in which self-guided iCBT was compared with a control condition (usual care, waiting list, or attention control) were included. No language or publication status exclusions were applied.

The analysis was completed in compliance with the Preferred Reporting Items for Systematic Review and Meta-analyses (PRISMA) IPD Statement.¹⁷ We used an existing database on psychological treatments for depression¹⁸ that is updated annually by a systematic literature search in the bibliographic databases of PubMed, Embase, PsycINFO, and Cochrane Library (from inception to January 1, 2016). In these searches, various index and free terms of psychotherapy and depression are used in different combinations (full search strings for PubMed are provided in the eMethods in the Supplement). Two researchers (P.C. and E.K.) independently examined titles and abstracts of 13 384 articles. The full text of studies that possibly met the inclusion criteria according to 1 of the 2 reviewers was retrieved. In case of disagreement regarding inclusion, consensus was sought through discussion. We also asked key researchers in the field whether they knew of unpublished trials.

Authors of eligible articles were contacted for permission to use their data sets. Reminders were sent after 2 weeks and if necessary after 1 month. If no response was received, we excluded the trial. Authors were asked to provide data on sociodemographic, clinical, and intervention characteristics, including information regarding randomized group, baseline and follow-up total scores of depressive symptoms, treatment adherence information (total number of sessions completed divided by total number of treatment sessions), age, sex, educational level (primary, secondary, and tertiary education), employment status (employed or unemployed), relationship status (in a relationship or not), and comorbid anxiety symptoms at baseline (yes or no; based on a clinical interview or elevated anxiety symptoms ratings on self-report measures). Finally, we combined all individual data sets into a merged data set, using a generic standardized protocol for integrating IPD sets.¹ We also used study-level variables, which were available from the full reports (type of comparator condition, recruitment, level of support). The selection of moderator variables has been based on previous literature related to moderators of face-to-face CBT or iCBT.^16,19

We examined the risk of bias in the included studies using the criteria of the Cochrane Collaboration risk of bias assessment tool.²⁰ Two independent reviewers (E.K., P.C.) evaluated the included studies to determine whether there was a risk for bias related to selection, performance, detection, attrition, and outcome reporting. In case of unclear risk of bias for 1 or more key domains, we contacted the first authors of the included studies for clarifications.

We conducted a traditional meta-analysis to examine differences among the 13 studies that provided the IPD and the 3 studies that did not. We used data reported in the articles to calculate the effect sizes (Hedges g).²¹ The reader is referred to the eMethods in the Supplement for details regarding the methods of the traditional meta-analysis.

Studies included in this IPD meta-analysis used measures such as the Center of Epidemiologic Studies–Depression Scale,²² the Beck Depression Inventory I²³ or II,²⁴ (hereafter referred to as Beck Depression Inventory) or the 9-item Patient Health Questionnaire²⁵ to monitor change in depressive symptoms severity. These depression measures were standardized by transformation into z scores across the pool of the studies before conducting the main analysis.

Missing outcome data at the posttreatment assessment were estimated using multiple imputation under the missing-at-random assumption (mi impute mvn in STATA software, version 13.1; StataCorp). This method generated 100 imputed data sets using data on baseline depressive symptoms scores, age, sex, and group. These new imputed data sets included the observed and the imputed standardized depressive symptoms scores for the missing values. They were analyzed separately using the selected model, and the results were averaged according to Rubin’s rules.²⁶ We also conducted sensitivity analyses using only participants with complete data after treatment to examine whether there was a difference between those who dropped out of the RCTs and those who provided posttreatment data.

In a 1-stage IPD meta-analysis, we merged all IPD from all studies with participants nested within studies. One-stage IPD meta-analysis yields more precise and less biased estimates of effect, maximizes the power, and accounts for parameter correlation.^27,28 We calculated the standardized β coefficient for the examined comparisons. This estimate indicates how many SDs the dependent variable (depressive symptoms severity or the log odds ratio [OR] of treatment response) changes per SD increase in the predictor variable. Thus, the higher the β is the greater the effect of the predictor variable on the dependent variable, although there is no association among the variables if the β is 0. All analyses were conducted with STATA statistical software, version 131. The primary analysis was 2-fold. First, we analyzed the effects of the interventions on depressive symptom severity at the end of treatment using a multilevel mixed-effects linear regression (using a random intercepts model with a random effect for each trial and fixed effects for the intervention and the symptoms severity, using STATA’s mixed command). The posttreatment depression scores were used as the dependent variable and trial arm condition (treatment vs control) as the independent variable, while controlling for baseline depressive symptom severity.

Second, we analyzed the effects of the interventions on treatment response (defined as a 50% reduction in baseline depressive symptoms scores) at the posttreatment assessment using a multilevel mixed-effects logistic regression (using a random intercepts model with a random effect for each trial and fixed effects for the intervention and the depressive symptoms severity, using STATA’s melogit command). The response (yes or no) was the dependent variable, and condition was the independent variable.

Third, we ran a 2-stage IPD meta-analysis analyzing the IPD separately in each study and then combining the estimates to calculate the pooled effect sizes (Hedges g) for depressive symptoms severity. Two-stage IPD meta-analysis facilitates analysis standardization across the included studies and estimation of outcomes that are not available in the published reports, such as treatment response.²⁹ Similarly, we calculated the OR of treatment response and numbers needed to treat (NNTs), which allowed us to compare the results of the present meta-analysis with those reported in earlier meta-analyses. In addition, 2-stage IPD meta-analysis also allowed us to examine the moderation effect of study-level variables. Thus, subgroup-moderator analyses were conducted using a mixed-effects model in which the random-effects model was used to pool studies within subgroups, whereas between-subgroup differences were tested as fixed effects. We also ran meta-regression analyses to examine the association between treatment duration and treatment outcomes (severity of depressive symptoms and treatment response).

We tested whether available demographic and clinical characteristics moderated the effect of self-guided iCBT on depression outcomes (depressive symptoms severity and treatment response). Not all included studies reported data on the examined moderators (for precise numbers regarding the missing data, see Table 1 and Table 2). To examine moderators, we added the interaction between each potential moderator and treatment outcome on depression severity into the multilevel mixed-effects linear regression model. We similarly added the interaction between each potential moderator and treatment response into the multilevel mixed-effects logistic regression model. Each potential moderator was included in a separate model as a main effect.

We examined whether adherence to treatment predicted within treatment group effect size for the experimental condition only, using a linear mixed model, which regressed posttreatment depressive symptoms severity on treatment adherence and baseline depressive symptoms severity (fixed effects) and using random intercepts for the studies. Treatment adherence was defined as the total number of sessions that each participant completed divided by the total number of treatment sessions.

The systematic search resulted in 16 eligible articles of 1885 full-text articles screened. We were able to obtain IPD from 13 of the 16 eligible trials (81%), yielding a total of 3876 participants.^{12,14,15,30-38} Three eligible data sets^39-41 were unavailable and thus could not be included in the IPD meta-analyses. Figure 1 shows the study selection process.

Seven of the included studies^{30,31,34,35,37,38} recruited participants through the community. The included RCTs examined iCBT, with interventions comprising 5 to 11 online sessions. Four of the included trials provided support related to the technical aspects of the online platforms,^15,31,33,36 whereas 9 trials were purely self-guided.^{12-14,30,32,34,35,37,38} The control conditions used were attention placebo, no treatment, treatment as usual, or waiting list. The included studies were conducted in 6 countries: Australia, Germany, Spain, Switzerland, the Netherlands, and the United Kingdom (eTable 1 in the Supplement presents a summary of study characteristics).

Of the 3876 study participants, the mean (SD) age was 42.0 (11.7) years, 2531 (66.0%) of 3832 were female, 1368 (53.1%) of 2574 completed secondary education, and 2262 (71.9%) of 3146 were employed. The mean baseline depressive symptoms scores were 25.7 on the Center of Epidemiologic Studies–Depression Scale, 28.3 on the Beck Depression Inventory, and 14.1 on the 9-item Patient Health Questionnaire in their respective studies. Finally, 71 (1.8%) of 3876 randomized participants did not start the treatment or did not provide baseline and posttreatment data, and 1048 (27.0%) of 3876 dropped out of the RCT and did not provide posttreatment depressive symptoms scores. (eTable 2 in the Supplement provides a summary of participants’ characteristics.)

All included studies scored low on all examined items of the Cochrane risk of bias tool. Random allocation sequences were adequately generated, and the allocation was sufficiently concealed in all included RCTs. Participants were not masked because this is difficult to achieve in psychotherapy research. All studies used self-report outcome measures. Missing data were imputed as part of the present IPD to minimize study attrition bias. Finally, studies were assessed as being free of outcome reporting bias and other sources of bias (eTable 3 in the Supplement).

Sixteen studies examined the comparison between self-guided iCBT and control groups. The results of the traditional meta-analysis revealed that self-guided iCBT outperformed the control conditions at posttreatment assessment (g = 0.33; 95% CI, 0.19-0.46; P < .001). Heterogeneity was moderate to high and significant (I² = 71%; 95% CI, 51%-82%; P < .001). There was no significant difference between the outcome findings of studies included in the present IPD meta-analysis and studies with unavailable data (P = .95) (Figure 2). There was some indication of publication bias. With the use of the Duval and Tweedie trim and fill method, values for 5 studies were imputed and the point estimate reduced to g = 0.21 (95% CI, 0.07-0.34), and the Egger test result was significant (P < .001) (eFigure 1 in the Supplement).

Table 1 presents the main findings of the 1-stage IPD meta-analysis on depressive symptoms severity after testing (ranging from 6 to 16 weeks after randomization). There was a significant effect of self-guided iCBT over control conditions on depressive symptoms (β = −0.21; P < .001). Complete cases yielded similar outcomes (β = −0.19; P < .001). None of the participant-level variables (sociodemographic and clinical characteristics) significantly moderated outcome after treatment (Table 1). However, adherence to treatment predicted significantly better outcomes within the self-guided iCBT group (β = −0.19; P = .001).

The 2-stage IPD meta-analysis resulted in a pooled effect size of g = 0.27 (95% CI, 0.17-0.37; P < .001) in favor of self-guided iCBT (eTable 4 in the Supplement). Similar outcomes were obtained in complete cases analyses (g = 0.32; 95% CI, 0.17-0.46; P < .001). None of the examined study-level variables (type of comparator condition, recruitment, level of support, and treatment duration) were significantly associated with treatment outcome (eTable 4, eFigure 2, and eFigure 3 in the Supplement).

A significant effect in favor of self-guided iCBT over controls was found for treatment response (β = 0.53; P < .001) (Table 2). Complete cases analyses resulted in similar outcomes (β = 0.50; P < .001). None of the sociodemographic and clinical characteristics of participants were significantly associated with treatment response (Table 2). Treatment adherence significantly predicted treatment response (β = 0.90; P < .001).

The OR was 1.95 (95% CI, 1.52-2.50; P < .001) in favor of the self-guided iCBT group, which corresponds to a NNT of 8 (95% CI, 6-12) (eTable 5 in the Supplement). Similar outcomes were found when we conducted complete case analysis (OR, 1.88; 95% CI, 1.34-2.64; P < .001; NNT, 9; 95% CI, 6-17). None of the examined study-level variables was significantly associated with treatment response (eTable 5, eFigure 4, and eFigure 5 in the Supplement).

In this study, we examined the effects of self-guided iCBT on severity and treatment response. We aimed to identify moderators of treatment outcome. We found that self-guided iCBT had lower depressive symptom severity and greater treatment response compared with control conditions after testing. These findings were robust in complete case analyses. Treatment adherence was significantly related to treatment outcomes within the self-guided iCBT group. None of the examined participant- and study-level variables significantly moderated the treatment effect.

The finding that self-guided iCBT results in a significant effect on depression outcomes is consistent with previous literature.⁴² However, the present IPD meta-analysis provides stronger evidence and improves the precision of the estimates because of the novel methodologic approach used. Moreover, previous literature⁴² did not examine NNTs. The current findings indicate that we need to treat 8 individuals with depressive symptoms with self-guided iCBT to expect a 50% symptom reduction. Although this NNT is relatively large and its clinical relevance could be doubted, it can still have a considerable effect when large groups of patients use the treatment, especially considering the low costs of self-guided iCBT.

The role of treatment adherence in outcomes has been identified by a previous review in the field conducted by Donkin and colleagues.⁴³ The authors concluded that the number of sessions correlated with outcomes in the interventions that targeted at depressive symptoms.⁴³ In other words, participants did better when they adhered to the intervention. However, treatment adherence follows the course of the intervention and may be influenced by response to treatment as much as vice versa. As previous research findings⁴⁴ have suggested, there may be different preexisting factors (eg, age and sex) that influence the association between treatment adherence and treatment outcomes.

It is also interesting that baseline depressive symptoms scores did not moderate treatment outcomes. This finding contrasts with the findings of the IPD meta-analysis of low-intensity interventions by Bower et al,¹⁶ who found that higher levels of depressive symptoms at baseline were associated with better depressive outcomes (greater decrease in depressive symptoms) after the completion of low-intensity interventions. However, this effect was relatively small. The authors concluded that it might not be clinically relevant and that it is safer to assume that low-intensity interventions work equally across a range of severities.

Among the strengths of the present study was its high power to detect small statistically significant differences between intervention and controls and to yield more precise and robust evidence compared with traditional meta-analyses. Moreover, the included RCTs had high methodologic quality, which allows us to be confident that the present analysis is relatively free of critical biases. However, many internet-delivered interventions incorporate repeated use of symptom inventories with each online session. This repeated administration of symptom inventories might yield lower mean scores with each wave of measurement (completer biases related to self-report ratings).⁴⁵ Moreover, the included studies did not report on recruitment issues related to large-scale, fully unguided internet-administered interventions, including factors such as repeated registration attempts by individuals who did not meet inclusion criteria or who were dissatisfied with their intervention allocation. These matters constitute a potential threat to validity and should be addressed by future research in this field.

Several limitations of our IPD meta-analysis should be mentioned. We observed moderate to high heterogeneity. Unfortunately, the subgroup analyses did not provide any indication of which study-level variables are associated with the observed heterogeneity. Moreover, our findings are at risk (albeit low) of availability bias because we could not access data from 3 eligible studies of the 16. However, the results of the traditional meta-analysis indicated that the findings of these 3 unavailable trials did not differ from the findings of the included RCTs. Another limitation is that we could not examine duration of symptoms as a potential moderator of treatment outcome. Duration of symptoms is important because individuals with chronic depressive symptoms may not always respond rapidly to treatment. Furthermore, most of the included trials recruited their self-referred participants through the community, thereby limiting our ability to generalize the present results to clinical samples. Finally, there was some indication of publication bias, suggesting that unpublished trials with negative findings might be missing from the present sample of studies.

Self-guided iCBT produces results that are encouraging. The absence of a significant difference in treatment outcomes associated with clinical and sociodemographic characteristics implies that self-guided iCBT can be used by most individuals with depressive symptoms regardless of the severity of their symptoms or their sociodemographic background. Currently, antidepressant medications are widely used in the treatment of depressive symptoms, whereas psychotherapeutic interventions are provided to a lesser degree, despite many individuals with depressive symptoms preferring psychotherapy to antidepressants.⁴⁶ However, the high treatment costs and the limited number of trained clinicians hamper the implementation of psychotherapy in practice.

The findings of the present IPD meta-analysis suggest that self-guided iCBT may be a viable alternative to current first-step treatment approaches for symptoms of depression, particularly in those individuals who are not willing to have any therapeutic contact. This form of intervention seems to be valuable for patients with primary depressive problems and those with depressive symptoms in the context of a primary somatic problem.^47,48 This self-help form of CBT can provide treatment access at low cost to large numbers of individuals worldwide who have depressive symptoms. Although it is beyond the scope of this study, unguided iCBT has several limitations that should be addressed before it is disseminated as part of routine care (eg, high dropout rates, small effects compared with face-to-face and guided internet interventions, and possible participant selection bias).

Given the effects found for treatment adherence, future research should focus on improving retention of participants in self-guided iCBT programs with the aim of maximizing positive therapeutic outcomes. Further research is also needed to examine additional moderators (eg, sleep quality, cognitive performance, duration of symptoms), long-term outcomes, and the value of adding therapist or coach support to these treatments. Finally, future studies should focus on the pragmatic effectiveness of iCBT in routine care settings.

Corresponding Author: Eirini Karyotaki, MSc, Department of Clinical Psychology and EMGO Institute for Health and Care Research, VU Amsterdam, Van der Boechorststraat 1, 1081 BT Amsterdam, the Netherlands (e.karyotaki@vu.nl).

Accepted for Publication: January 5, 2017.

Published Online: February 22, 2017. doi:10.1001/jamapsychiatry.2017.0044

Author Contributions: Ms Karyotaki had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Karyotaki, Kleiboer, Mira, Botella Arbona, Bretón-López, Gilbody, Moritz, Cuijpers.

Acquisition, analysis, or interpretation of data: Karyotaki, Riper, Twisk, Hoogendoorn, Kleiboer, Mackinnon, Meyer, Littlewood, Andersson, Christensen, Klein, Schröder, Scheider, Griffiths, Farrer, Huibers, Phillips, Gilbody, Moritz, Berger, Pop, Spek, Cuijpers.

Drafting of the manuscript: Karyotaki, Mira, Botella Arbona, Bretón-López, Moritz.

Critical revision of the manuscript for important intellectual content: Riper, Twisk, Hoogendoorn, Kleiboer, Mackinnon, Meyer, Littlewood, Andersson, Christensen, Klein, Schröder, Scheider, Griffiths, Farrer, Huibers, Phillips, Gilbody, Berger, Pop, Spek, Cuijpers.

Statistical analysis: Karyotaki, Twisk, Hoogendoorn, Mackinnon, Pop.

Obtained funding: Riper, Scheider, Gilbody, Cuijpers.

Administrative, technical, or material support: Mira, Botella Arbona, Littlewood, Andersson, Christensen, Klein, Schröder, Bretón-López, Farrer, Gilbody, Moritz, Spek, Cuijpers.

Study supervision: Riper, Kleiboer, Moritz, Cuijpers.

Conflict of Interest Disclosures: Dr Klein reported receiving funding for clinical trials (German Federal Ministry of Health, Servier), payments for presentations on internet interventions (Servier), and payments for workshops and books (Beltz, Elsevier and Hogrefe) on psychotherapy for chronic depression and psychiatric emergencies. No other disclosures were reported.

Funding/Support: This work was performed under grant 603098 for the European Comparative Effectiveness Research on Internet-based Depression Treatment (E-COMPARED) project funded by the European Commission’s Seventh Framework Program. At the time of the study, Dr Griffiths was supported by National Health & Medical Research Council Fellowship 1059620.

Role of the Funder/Sponsor: The European Commission had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The decision to submit the article for publication was a condition of the funding and was made before any results were available.

Additional Contributions: Carmen Domnica Cotet, PhD, helped with data extraction and did not receive additional compensation in association with her work on this article.