Coached Mobile App Platform for the Treatment of Depression and Anxiety Among Primary Care Patients: A Randomized Clinical Trial | Anxiety Disorders | JAMA Psychiatry | JAMA Network
[Skip to Navigation]
Figure 1.  CONSORT Flow Diagram
CONSORT Flow Diagram
Figure 2.  Change in Least Square Means Scores
Change in Least Square Means Scores

A, Change in depression least square mean scores by intervention condition over the intervention period (with imputed data). B, Change in anxiety least square mean scores by intervention condition over the intervention period (with imputed data).

Table 1.  Baseline Demographics of the Complete Sample and by Intervention Condition
Baseline Demographics of the Complete Sample and by Intervention Condition
Table 2.  Means and Standard Errors for Intervention Conditions at Each Time (Observed Data)a
Means and Standard Errors for Intervention Conditions at Each Time (Observed Data)a
Table 3.  App Use Metrics for the Full IntelliCare Platform and Each Individual App
App Use Metrics for the Full IntelliCare Platform and Each Individual App
1.
Ruscio  AM, Hallion  LS, Lim  CCW,  et al.  Cross-sectional comparison of the epidemiology of DSM-5 generalized anxiety disorder across the globe.   JAMA Psychiatry. 2017;74(5):465-475. doi:10.1001/jamapsychiatry.2017.0056PubMedGoogle ScholarCrossref
2.
Hasin  DS, Sarvet  AL, Meyers  JL,  et al.  Epidemiology of adult DSM-5 major depressive disorder and its specifiers in the United States.   JAMA Psychiatry. 2018;75(4):336-346. doi:10.1001/jamapsychiatry.2017.4602PubMedGoogle ScholarCrossref
3.
Olfson  M, Blanco  C, Marcus  SC.  Treatment of adult depression in the United States.   JAMA Intern Med. 2016;176(10):1482-1491. doi:10.1001/jamainternmed.2016.5057PubMedGoogle ScholarCrossref
4.
Siu  AL, Bibbins-Domingo  K, Grossman  DC,  et al; US Preventive Services Task Force (USPSTF).  Screening for depression in adults: US Preventive Services Task Force Recommendation Statement.   JAMA. 2016;315(4):380-387. doi:10.1001/jama.2015.18392PubMedGoogle ScholarCrossref
5.
O’Connor  EA, Whitlock  EP, Beil  TL, Gaynes  BN.  Screening for depression in adult patients in primary care settings: a systematic evidence review.   Ann Intern Med. 2009;151(11):793-803. doi:10.7326/0003-4819-151-11-200912010-00007PubMedGoogle ScholarCrossref
6.
McCrone  P, Knapp  M, Proudfoot  J,  et al.  Cost-effectiveness of computerised cognitive-behavioural therapy for anxiety and depression in primary care: randomised controlled trial.   Br J Psychiatry. 2004;185:55-62. doi:10.1192/bjp.185.1.55PubMedGoogle ScholarCrossref
7.
Gilbody  S, Littlewood  E, Hewitt  C,  et al; REEACT Team.  Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial.   BMJ. 2015;351:h5627. doi:10.1136/bmj.h5627PubMedGoogle ScholarCrossref
8.
Andrews  G, Basu  A, Cuijpers  P,  et al.  Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis.   J Anxiety Disord. 2018;55:70-78. doi:10.1016/j.janxdis.2018.01.001PubMedGoogle ScholarCrossref
9.
Gilbody  S, Brabyn  S, Lovell  K,  et al; REEACT collaborative.  Telephone-supported computerised cognitive-behavioural therapy: REEACT-2 large-scale pragmatic randomised controlled trial.   Br J Psychiatry. 2017;210(5):362-367. doi:10.1192/bjp.bp.116.192435PubMedGoogle ScholarCrossref
10.
Christensen  H, Griffiths  KM, Farrer  L.  Adherence in internet interventions for anxiety and depression.   J Med Internet Res. 2009;11(2):e13. doi:10.2196/jmir.1194PubMedGoogle Scholar
11.
Vaish  R, Wyngarden  K, Chen  J, Cheung  B, Bernstein  MS. Twitch crowdsourcing: crowd contributions in short bursts of time. Presented at: SIGCHI Conference on Human Factors in Computing Systems; 2014; Toronto, ON, Canada.
12.
Andrews  S, Ellis  DA, Shaw  H, Piwek  L.  Beyond self-report: tools to compare estimated and real-world smartphone use.   PLoS One. 2015;10(10):e0139004. doi:10.1371/journal.pone.0139004PubMedGoogle Scholar
13.
Baumel  A, Muench  F, Edan  S, Kane  JM.  Objective user engagement with mental health apps: systematic search and panel-based usage analysis.   J Med Internet Res. 2019;21(9):e14567. doi:10.2196/14567PubMedGoogle Scholar
14.
Firth  J, Torous  J, Nicholas  J,  et al.  The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials.   World Psychiatry. 2017;16(3):287-298. doi:10.1002/wps.20472PubMedGoogle ScholarCrossref
15.
Firth  J, Torous  J, Nicholas  J, Carney  R, Rosenbaum  S, Sarris  J.  Can smartphone mental health interventions reduce symptoms of anxiety? a meta-analysis of randomized controlled trials.   J Affect Disord. 2017;218:15-22. doi:10.1016/j.jad.2017.04.046PubMedGoogle ScholarCrossref
16.
Kwasny  MJ, Schueller  SM, Lattie  E, Gray  EL, Mohr  DC.  Exploring the use of multiple mental health apps within a platform: secondary analysis of the IntelliCare Field Trial.   JMIR Ment Health. 2019;6(3):e11572. doi:10.2196/11572PubMedGoogle Scholar
17.
Chiauzzi  E, Newell  A.  Mental health apps in psychiatric treatment: a patient perspective on real world technology usage.   JMIR Ment Health. 2019;6(4):e12292. doi:10.2196/12292PubMedGoogle Scholar
18.
Barlow  DH, Farchione  TJ, Bullis  JR,  et al.  The unified protocol for transdiagnostic treatment of emotional disorders compared with diagnosis-specific protocols for anxiety disorders: a randomized clinical trial.   JAMA Psychiatry. 2017;74(9):875-884. doi:10.1001/jamapsychiatry.2017.2164PubMedGoogle ScholarCrossref
19.
Mohr  DC, Tomasino  KN, Lattie  EG,  et al.  IntelliCare: an eclectic, skills-based app suite for the treatment of depression and anxiety.   J Med Internet Res. 2017;19(1):e10. doi:10.2196/jmir.6645PubMedGoogle Scholar
20.
Mohr  DC, Schueller  SM, Tomasino  KN,  et al.  Comparison of the effects of coaching and receipt of app recommendations on depression, anxiety, and engagement in the IntelliCare platform: factorial randomized controlled trial.   J Med Internet Res. 2019;21(8):e13609. doi:10.2196/13609PubMedGoogle Scholar
21.
Zhang  R, Nicholas  J, Knapp  AA,  et al.  Clinically meaningful use of mental health apps and its effects on depression: mixed methods study.   J Med Internet Res. 2019;21(12):e15644. doi:10.2196/15644PubMedGoogle Scholar
22.
Kroenke  K, Strine  TW, Spitzer  RL, Williams  JB, Berry  JT, Mokdad  AH.  The PHQ-8 as a measure of current depression in the general population.   J Affect Disord. 2009;114(1-3):163-173. doi:10.1016/j.jad.2008.06.026PubMedGoogle ScholarCrossref
23.
Spitzer  RL, Kroenke  K, Williams  JB, Löwe  B.  A brief measure for assessing generalized anxiety disorder: the GAD-7.   Arch Intern Med. 2006;166(10):1092-1097. doi:10.1001/archinte.166.10.1092PubMedGoogle ScholarCrossref
24.
Posner  K, Brown  GK, Stanley  B,  et al.  The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults.   Am J Psychiatry. 2011;168(12):1266-1277. doi:10.1176/appi.ajp.2011.10111704PubMedGoogle ScholarCrossref
25.
Graham  AK, Greene  CJ, Powell  T,  et al.  Lessons learned from service design of a trial of a digital mental health service: informing implementation in primary care clinics.   Transl Behav Med. In press.Google Scholar
26.
Tomasino  KN, Noth  F, Bardsley  LR, Lattie  EG, Mohr  DC. IntelliCare Study Coaching Manual. Northwestern University: DigitalHub. Published 2016. Accessed April 13, 2020. https://digitalhub.northwestern.edu/files/00fa4294-5b9f-4afc-897a-7fffceae8f3f
27.
Kroenke  K, Spitzer  RL, Williams  JB.  The PHQ-9: validity of a brief depression severity measure.   J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.xPubMedGoogle ScholarCrossref
28.
Pham  Q, Graham  G, Carrion  C,  et al.  A library of analytic indicators to evaluate effective engagement with consumer mhealth apps for chronic conditions: scoping review.   JMIR Mhealth Uhealth. 2019;7(1):e11941. doi:10.2196/11941PubMedGoogle Scholar
29.
Kraemer  HC.  Is it time to ban the P value?   JAMA Psychiatry. 2019. doi:10.1001/jamapsychiatry.2019.1965PubMedGoogle Scholar
30.
Cuijpers  P, Van Straten  A, Warmerdam  L, Smits  N.  Characteristics of effective psychological treatments of depression: a metaregression analysis.   Psychother Res. 2008;18(2):225-236. doi:10.1080/10503300701442027PubMedGoogle ScholarCrossref
31.
Cuijpers  P, Sijbrandij  M, Koole  S, Huibers  M, Berking  M, Andersson  G.  Psychological treatment of generalized anxiety disorder: a meta-analysis.   Clin Psychol Rev. 2014;34(2):130-140. doi:10.1016/j.cpr.2014.01.002PubMedGoogle ScholarCrossref
32.
Rollman  BL, Herbeck Belnap  B, Abebe  KZ,  et al.  Effectiveness of online collaborative care for treating mood and anxiety disorders in primary care: a randomized clinical trial.   JAMA Psychiatry. 2018;75(1):56-64. doi:10.1001/jamapsychiatry.2017.3379PubMedGoogle ScholarCrossref
33.
Mohr  DC, Weingardt  KR, Reddy  M, Schueller  SM.  Three problems with current digital mental health research . . . and three things we can do about them.   Psychiatr Serv. 2017;68(5):427-429. doi:10.1176/appi.ps.201600541PubMedGoogle ScholarCrossref
34.
Mohr  DC, Ho  J, Duffecy  J,  et al.  Perceived barriers to psychological treatments and their relationship to depression.   J Clin Psychol. 2010;66(4):394-409. doi:10.1002/jclp.20659PubMedGoogle Scholar
35.
Mohr  DC, Hart  SL, Howard  I,  et al.  Barriers to psychotherapy among depressed and nondepressed primary care patients.   Ann Behav Med. 2006;32(3):254-258. doi:10.1207/s15324796abm3203_12PubMedGoogle ScholarCrossref
36.
Graham  AK, Lattie  EG, Mohr  DC.  Experimental therapeutics for digital mental health.   JAMA Psychiatry. 2019. doi:10.1001/jamapsychiatry.2019.2075PubMedGoogle Scholar
37.
Mohr  DC, Riper  H, Schueller  SM.  A solution-focused research approach to achieve an implementable revolution in digital mental health.   JAMA Psychiatry. 2018;75(2):113-114. doi:10.1001/jamapsychiatry.2017.3838PubMedGoogle ScholarCrossref
38.
Mohr  DC, Lyon  AR, Lattie  EG, Reddy  M, Schueller  SM.  Accelerating digital mental health research from early design and creation to successful implementation and sustainment.   J Med Internet Res. 2017;19(5):e153. doi:10.2196/jmir.7725PubMedGoogle Scholar
39.
Donkin  L, Christensen  H, Naismith  SL, Neal  B, Hickie  IB, Glozier  N.  A systematic review of the impact of adherence on the effectiveness of e-therapies.   J Med Internet Res. 2011;13(3):e52. doi:10.2196/jmir.1772PubMedGoogle Scholar
40.
Fuhr  K, Schröder  J, Berger  T,  et al.  The association between adherence and outcome in an Internet intervention for depression.   J Affect Disord. 2018;229:443-449. doi:10.1016/j.jad.2017.12.028PubMedGoogle ScholarCrossref
41.
Mohr  DC, Duffecy  J, Jin  L,  et al.  Multimodal e-mental health treatment for depression: a feasibility trial.   J Med Internet Res. 2010;12(5):e48. doi:10.2196/jmir.1370PubMedGoogle Scholar
42.
Gilbody  S, Bower  P, Fletcher  J, Richards  D, Sutton  AJ.  Collaborative care for depression: a cumulative meta-analysis and review of longer-term outcomes.   Arch Intern Med. 2006;166(21):2314-2321. doi:10.1001/archinte.166.21.2314PubMedGoogle ScholarCrossref
43.
Muntingh  AD, van der Feltz-Cornelis  CM, van Marwijk  HW, Spinhoven  P, van Balkom  AJ.  Collaborative care for anxiety disorders in primary care: a systematic review and meta-analysis.   BMC Fam Pract. 2016;17:62. doi:10.1186/s12875-016-0466-3PubMedGoogle ScholarCrossref
44.
The MacArthur Foundation’s Initiative on Depression and Primary Care. The MacArthur Initiative on Depression and Primary Care at Dartmouth and Duke: Depression Management Toolkit. Hanover, NH: Dartmouth;2004.
45.
Whitney  DG, Peterson  MD.  US national and state-level prevalence of mental health disorders and disparities of mental health care use in children.   JAMA Pediatr. 2019;173(4):389-391. doi:10.1001/jamapediatrics.2018.5399PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    May 20, 2020

    Coached Mobile App Platform for the Treatment of Depression and Anxiety Among Primary Care Patients: A Randomized Clinical Trial

    Author Affiliations
    • 1Center for Behavioral Intervention Technologies, Northwestern University, Chicago, Illinois
    • 2Department of Medical Social Sciences, Northwestern University, Chicago, Illinois
    • 3Psychiatric Research Institute, University of Arkansas for Medical Sciences, Little Rock
    • 4Department of Preventive Medicine, Northwestern University, Chicago, Illinois
    • 5Actualize Therapy, Inc, Chicago, Illinois
    • 6Department of Biomedical Informatics, University of Arkansas for Medical Sciences, Little Rock
    JAMA Psychiatry. 2020;77(9):906-914. doi:10.1001/jamapsychiatry.2020.1011
    Key Points

    Question  Is a mobile intervention platform composed of a suite of simple-to-use apps, supported by brief coaching, efficacious for treating depression and anxiety among primary care patients?

    Findings  In this randomized clinical trial of 146 patients with depression and anxiety, a mobile platform achieved greater reductions in depression and anxiety and higher odds of recovery compared with treatment-as-usual wait list control individuals, and effects were sustained at follow-up. Engagement with apps was high throughout the intervention period.

    Meaning  Results support the efficacy of a platform approach to mobile intervention using apps designed to fit into the fabric of users’ lives for treating patients with depression and anxiety in primary care.

    Abstract

    Importance  Depression and anxiety are common and disabling. Primary care is the de facto site for treating these mental health problems but is typically underresourced to meet the burden of these demands.

    Objective  To evaluate the efficacy of a mobile intervention platform, IntelliCare, for addressing depression and anxiety among primary care patients.

    Design, Setting, and Participants  Two-arm randomized clinical trial at internal medicine clinics at the University of Arkansas for Medical Sciences. Adult primary care patients (N = 146) who screened positive for depression on the Patient Health Questionnaire-8 (PHQ; score  ≥ 10) or anxiety on the Generalized Anxiety Disorder-7 (GAD-7; score ≥ 8) were recruited between July 17, 2018, and December 14, 2018.

    Interventions  The coach-supported platform composed of a suite of apps, was delivered over 8 weeks. Wait list control participants received treatment as usual for 8 weeks, then the mobile platform.

    Main Outcomes and Measures  Primary outcomes were changes in depression (PHQ-9) and anxiety (GAD-7) during the intervention period. Secondary outcomes were differences in the proportion of patients who achieved recovery (PHQ-9/GAD-7 <5 or 50% improvement from baseline), sustainment of intervention effects during 2-month follow-up, and app use during the intervention period.

    Results  One hundred forty-six patients were included (119 of 146 were women [81.5%]; mean [SD] age, 42.3 [13.8] years). Of the 146 patients, 122 (83.6%) were diagnosed as having depression and 131 (89.7%) were diagnosed as having anxiety. A greater proportion of intervention vs wait list control participants achieved recovery from depression (n = 38 of 64 [59%] vs n = 18 of 58 [31%]; odds ratio, 3.25; 95% CI, 1.54-6.86) and anxiety (n = 37 of 65 [57%] vs n = 25 of 66 [38%]; odds ratio, 2.17; 95% CI, 1.08-4.36). Sustained effects were observed for depression (slope, 0.01; 95% CI, –0.09 to 0.10; P = .92) and anxiety scores (slope, 0.02; 95% CI, –0.08 to 0.12; P = .67) during follow-up. App use was high, with a median of 93 and 98 sessions among participants with depression and anxiety, respectively.

    Conclusions and Relevance  In this trial, a mobile intervention app was effective for depression and anxiety among primary care patients. Findings also support designing digital mental health interventions as platforms containing simple, brief apps that can be bundled by users to meet their needs.

    Trial Registration  ClinicalTrials.gov Identifier: NCT03500536.

    Introduction

    Depression and anxiety are prevalent, impairing, and costly health problems, most often treated in primary care.1-3 The US Preventive Services Task Force recommends routine depression screening in primary care4; however, without additional assistance to support treatment, increased identification provides no benefit.5 Unfortunately, demand for mental health services outweighs workforce capacity and resources to meet these needs. Digital mental health interventions (DMHIs), using Internet-based and mobile tools, can be cost-effective and have potential to improve capacity of health care settings to address mental health problems.6 However, implementation in primary care has often failed, primarily because patients do not use the tools.7

    Computer-delivered DMHIs are primarily based on evidence-based therapies, rely heavily on psychoeducation, and typically require use for 30 to 45 minutes per week for 6 or more weeks. Despite good efficacy in randomized clinical trials (RCTs),8 real-world engagement is challenging.7,9,10 In contrast to computers, people tend to use smartphone apps for a single purpose, in short bursts of time, sometimes frequently throughout the day.11,12 Thus, app-based DMHIs often focus on 1 to 2 psychological strategies and more on actions than psychoeducation.13 App-based DMHIs have demonstrated efficacy in RCTs,14,15 are increasingly used by the public, and produce reasonable early engagement, but adherence drops dramatically over the first 2 weeks.13

    App-based DMHIs may have greater success if they accommodate app use conventions and overcome 2 major challenges for delivering mental health interventions. First, the many effective behavioral strategies may appeal differentially to different people. Second, people do not tend to use mental health apps for long periods. This may be because people seek the novelty of new apps16 or have acquired the app’s benefit and no longer find it useful. Thus, a single app approach for a mental health condition may be less effective.17

    The IntelliCare platform was designed to address these challenges. Rather than a single app, the platform contains multiple apps, each of which is brief, simple, and supports a single clinical target. For example, the Daily Feats app targets goal-setting; the sole activity is setting and monitoring completion of daily goals, with scaffolding to increase complexity. Because depression and anxiety are highly comorbid1,2 and respond to similar psychological strategies,18 the platform was designed to address both conditions. This mobile platform has been evaluated in field trials, the first of which assessed symptom changes over 8 weeks and the second of which tested 2 engagement strategies.19,20 In both trials, the platform produced significant reductions in depression and anxiety19,20 and consistent engagement.16,21

    Quiz Ref IDThe purpose of this RCT was to evaluate the efficacy of the platform for the treatment of depression and anxiety in primary care patients. We hypothesized that individuals randomized to receive the mobile platform would report greater reductions in depression or anxiety symptoms compared with treatment-as-usual wait list control individuals (WLC) during the 2-month intervention period (primary outcome) and have a higher proportion of patients achieve recovery from depression or anxiety (secondary outcome). Secondary outcomes also included evaluating sustainment of effects over follow-up among individuals in the condition and engagement metrics. To our knowledge, this is the first evaluation of an app platform for the treatment of a mental health condition.

    Methods
    Design

    This RCT compared IntelliCare with treatment as usual WLC for depression and anxiety. The University of Arkansas for Medical Sciences (UAMS) institutional review board approved the study. A data safety monitoring board provided oversight. Supplement 1 provides the trial protocol.

    Participants

    The trial protocol in Supplement 1 details entry criteria. In brief, adult UAMS primary care patients were eligible if they had a compatible smartphone (Android or Apple) and elevated symptoms of depression or anxiety at screening (score ≥10 on the Patient Health Questionnaire-8 [PHQ-8]22 or ≥8 on the Generalized Anxiety Disorder-7 scale [GAD-7]23). Individuals were excluded, among others, if they were acutely suicidal,24 had a diagnosis for which the IntelliCare study might not be appropriate, and were currently in or scheduled to initiate psychotherapy to avoid treatment interference. Psychotropic medication was permitted if dose was stable over the past 2 weeks to eliminate medication placebo effects.

    Procedure

    Participants were recruited between July 17, 2018, and December 14, 2018, from UAMS internal medicine clinics on the main medical center campus and surrounding communities. These clinics serve a diverse, often low-income population. All clinics provided routine depression screening; some had on-site integrated behavioral health resources. Participants were referred by a clinician or responded to outreach methods. Recruitment strategies have been described25 and are in the protocol in Supplement 1.

    Interested individuals completed an online screener; those whose results indicated potential eligibility were automatically invited to provide online written informed consent. After medical record review by the research team, individuals who remained potentially eligible were invited to complete an online baseline assessment.

    Following randomization, participants in the intervention condition began IntelliCare via an onboarding telephone call with their assigned coach. Wait list control participants were informed they would have access to IntelliCare after an 8-week delay. All participants were prompted to complete assessments every 4 weeks for 16 weeks, administered online and independent of the intervention to mask outcomes to coaches. Participants could receive up to $100 for completing the research assessments. Data collection ended on April 17, 2019.

    Randomization

    An independent statistician created a computer-generated randomization sequence with a 1-to-1 ratio in randomly permutated blocks of 4 and 6. Randomization assignment was concealed until after entry criteria were confirmed.

    Intervention Conditions
    IntelliCare

    The intervention was delivered over 8 weeks using the coach-supported IntelliCare platform. IntelliCare is a suite of mobile apps available on iPhone and Android operating systems. The Hub app organized the user experience by supporting access to clinically focused IntelliCare apps, providing a library of psychoeducational material, and administering a weekly symptom assessment.19 In this study, participants received access to 5 clinically focused apps (described in the protocol in Supplement 1).19 Each week, a coach recommended a new app to download and try, based on the participant’s preferences and a recommendation protocol. Participants were encouraged to try the newly recommended app but could download any app at any time and use or discontinue apps as preferred.

    Coaches were 2 bachelor’s degree–level individuals who received training in the coaching manual26 and weekly supervision with a clinical psychologist. Coaching was delivered primarily via short message service text messaging; coaches typically initiated 2 contacts per week and responded to participants’ messages. Participants also received a welcome packet via email (developed with health literacy experts), 1 onboarding telephone call (approximately 30-45 minutes), and the offer for an optional midtreatment call (approximately 10-15 minutes). Further details are described in the protocol in Supplement 1.

    Coaches managed participants using an online dashboard, which provided information on each participant’s app use and weekly symptom assessment score and enabled sending and receiving text messages. Apps remained available to participants after treatment ended.

    Wait List Control

    The WLC allowed participants to continue treatment-as-usual through primary care. Following the 8-week delay, WLC individuals received the coached intervention.

    Outcome Assessments

    Quiz Ref IDPrimary outcomes (PHQ-927 and GAD-723) were measured at baseline and each follow-up assessment. App use metrics were captured automatically and stored on a secure server. App use was defined using 3 commonly used metrics20,28: number of app sessions, number of days the intervention was used, and time to last use (ie, number of days between the first and last launch) during the treatment period. An app session was defined as a user-initiated action or event after 5 minutes of no activity. Engagement with coaching was defined as number of messages sent by coaches and participants.

    Analyses

    Assuming 80% power and α = .05, a medium effect size of 0.5 between IntelliCare and WLC could be expected to be detected with 128 participants (64 per arm). Eighteen additional participants enrolled before recruitment procedures closed. All analyses were performed at a 2-sided type 1 error rate of .05 in SAS, version 9.4 (SAS Institute Inc).

    Analyses followed intention to treat. Five participants did not have any follow-up data during the first phase of the study, evenly split between study arms (IntelliCare, 3; WLC, 2). No evidence indicated these participants were other than missing at random; hence, multiple imputation methods were used to impute follow-up data for these participants.

    Primary outcomes were analyzed separately for participants who screened positive for depression and for anxiety to assess change in depression and anxiety, respectively. General linear mixed models were fit for PHQ-9 scores among those with depression and GAD-7 scores among those with anxiety for each imputed data set. Results were combined for those analyses to present valid inferences for the effect of the intervention over time. Primary analyses were unadjusted; secondary analyses adjusted for age, sex, and race.

    Secondary outcome analyses were conducted. Univariable logistic regression models were fit to determine whether there was an association between study condition and recovery at the postintervention point, defined as PHQ-9/GAD-7 scores less than 5 or 50% reduction from baseline; additional models were fit adjusting for age, sex, and race. To assess sustained response between posttreatment and 16-week follow-up (among only those in the IntelliCare condition), general linear mixed models were fit for PHQ-9 among those with depression and GAD-7 among those with anxiety; models did not use imputed data. Correlations were used to assess the association between app use metrics and outcomes and engagement with coaching and outcomes at the postintervention point. Because these were not planned comparisons, only confidence intervals are reported.29

    Results
    Participants

    Figure 1 presents the CONSORT diagram. Of 435 individuals screened, 146 participants (34%) were enrolled and randomized. At screening, 122 participants presented with depression (64 IntelliCare; 58 WLC) and 131 presented with anxiety (65 IntelliCare; 66 WLC); most (n = 107; 73%) had comorbid depression and anxiety. Table 1 shows pretreatment characteristics of the sample and 2 study conditions among the subsets with depression or anxiety. The lost-to-follow-up rate was low, with 141 of 146 participants (96.6%) completing at least 1 follow-up assessment.

    Primary Outcome: Changes in Depression or Anxiety

    Quiz Ref IDFigure 2A and B depict the least square means (LSM) of outcomes over time for depression and anxiety, with imputed data from 5 multiply imputed data sets. eTable 1 in Supplement 2 provides model parameter estimates. There was a significant group-by-time effect over 8 weeks (F2,33 316 = 15.9; P < .001), meaning those assigned to the mobile platform experienced a faster rate of improvement than WLC. The LSM difference in depression scores between the mobile platform and WLC at week 4 was 2.88 (SE = 0.82; Cohen d effect size [ES] = 0.52) and at week 8 was 4.33 (SE = 0.83; d ES = 0.78). The LSM difference in anxiety scores at week 4 was 2.38 (SE = 0.77; d ES = 0.47) and at week 8 was 3.19 (SE = 0.76; d ES = 0.64).

    After adjusting for age, sex, and race, the effect of group by treatment interaction remained significant for depression (F2,34 315 = 15.3; P < .001). The LSM difference in depression scores at week 4 was 2.91 (SE = 0.83; ESd = 0.43) and at week 8 was 4.37 (SE = 0.83; d ES = 0.64). Similarly, the group by treatment interaction remained significant for anxiety (F2,3228.4 = 11.9; P < .001). The LSM difference in anxiety scores at week 4 was 2.51 (SE = 0.78; d ES = 0.41) and at week 8 was 3.33 (SE = 0.76; d ES = 0.55).

    Secondary Outcomes
    Recovery at Postintervention Point

    The rate of recovery from depression was 59.4% in IntelliCare and 31.0% in WLC. The rate of recovery from anxiety was 56.9% in IntelliCare and 37.9% in WLC. The odds of recovery for depression were 3.25 (95% CI, 1.54-6.86) times greater and for anxiety were 2.17 (95% CI, 1.08-4.36) times greater for IntelliCare compared with WLC. After adjusting for age, sex, and race, odds of recovery increased slightly to 3.42 (95% CI, 1.57-7.48) for depression and 2.24 (95% CI, 1.07-4.68) for anxiety.

    Sustained Response at Follow-up

    Among those randomized to IntelliCare, effects were sustained from posttreatment to 16-week follow-up. There was no evidence of a linear change in slope for change in depression scores (slope, 0.01; 95% CI, –0.09 to 0.10; P = .92), nor were the means different over time (F2,119= 0.02; P = .98). Similarly, there was no evidence of a linear change in slope for change in anxiety scores (slope, 0.02; 95% CI, –0.08 to 0.12; P = .67), nor were the means different over time (F2,118= 0.09; P = .92). Means and standard errors for the study conditions by depression and anxiety scores at each point are presented in Table 2, using observed data for ease of interpretation.

    App Use

    Quiz Ref IDTable 3 presents app use metrics for the full suite and each app. Neither app sessions (r, −0.03; 95% CI, −0.22 to 0.15), time to last use (r, −0.14; 95% CI, −0.31 to 0.05), nor days used (r, −0.05; 95% CI, −0.23 to 0.14) were strongly associated with changes in depression. Neither app sessions (r, 0.01; 95% CI, −0.17 to 0.19), time to last use (r, −0.04; 95% CI, −0.22 to 0.13)], nor days used (r, 0.02; 95% CI, −0.15 to 0.20) were strongly associated with changes in anxiety. eTable 2 in Supplement 2 shows correlations with outcomes for each app.

    At 8 weeks following treatment, 119 participants (81.5%) had some app use. For all participants, postintervention median time to last app use was 28 days (range, 0-212 days) and median days used was 7 (range, 0-102 days).

    Engagement With Coaching

    Among those with depression, coaches sent a mean of 32 messages (SD, 9; range, 7-61); participants sent a mean of 19 messages (SD, 12; range, 1-68). There was a negligible correlation between change in PHQ-9 and number of messages from coaches (r, −0.07; 95% CI, −0.25 to 0.13) or participants (r, −0.03; −0.22 to 0.16). Among those with anxiety, coaches sent a mean of 32 messages (SD, 10; range, 7-61); participants sent a mean of 19 messages (SD, 12; range, 1-68). The correlation between change in GAD-7 and number of messages from coaches (r, −0.01; 95% CI, −0.19 to 0.17) or participants (r, −0.02; 95% CI, −0.20 to 0.17) was not meaningful.

    Adverse Events

    There were no study-related adverse events.

    Discussion

    This is, to our knowledge, the first RCT of an app-based platform for the treatment of depression and anxiety and of any app-based mental health treatment among primary care patients. Individuals who received the mobile platform had a greater reduction in depression and anxiety symptoms compared with treatment-as-usual WLC individuals, and changes were sustained over 2-month follow-up. Although analysis of the treatment period for the WLC was not planned or performed, once these participants received treatment, their depression and anxiety scores dropped to levels similar to those in the treatment group, further strengthening the conclusion that IntelliCare is effective for depression and anxiety among primary care patients.

    Effect sizes of 0.78 and 0.64 for depression and anxiety in this trial are in the range of effect sizes in meta-analyses of face-to-face psychotherapy (0.69 and 0.84, respectively)30,31 and compare favorably with effect sizes for app-based treatments (0.38 and 0.33, respectively)14,15 and web-based interventions evaluated in primary care (0.31 and 0.26, respectively).32,33

    Digital mental health interventions have been championed as a solution to access barriers and high dropout rates for psychotherapy among primary care patients.34,35 However, engagement has proven challenging. An effectiveness trial of a web-based intervention found participants logged in for approximately half of the 8 sessions32; in 2 implementation trials of web-based interventions in English primary care settings, median uses were 1-2, with only 10% to 19% completing their intervention.7,9 While we are unaware of mental health app adherence data in primary care, the median number of mental health app sessions is 3 to 9 among active public users.13 In contrast, we found high app use, with a median of 93 to 98 app sessions across all apps over 8 weeks. Our findings may be because the platform approach matches how people use apps: participants could use or ignore apps as preferred, and the novelty of receiving new apps is likely engaging.16,21,36 The sample also comprised a group rarely represented in technology research: in contrast to past research, which has tended to attract those drawn to technology,33 participants were from a diverse, traditionally underserved community seeking help for depression and anxiety. Thus, findings suggest this service appealed to patients more generally and underscore the importance of designing DMHIs to match patients’ app use preferences.33,37,38 App use was not strongly associated with outcomes, consistent with much of the DMHI literature.16,39,40 The association between use and outcomes is complex. Clinically meaningful app engagement may be more important than raw quantity.21 Furthermore, people tend to use DMHIs less as their depression improves.41 Taken together, the relative high engagement suggests that platform approaches, such as IntelliCare, may be more acceptable to patients and likely produce better outcomes than disorder-focused single apps; however, this remains to be tested in a trial.

    Results also inform potential for scale. Although routine screening has increased the identification of patients with mental health problems in primary care, this has not necessarily led to improved outcomes.5 Further, despite the success of collaborative care, which embeds behavioral health specialists in primary care,42,43 demand for such services frequently exceeds health care organizations’ capacity. Digital mental health intervention app platforms have potential to enhance behavioral health treatment effectiveness by improving clinicians’ efficiency while maintaining cost-effectiveness. Indeed, in this trial, coaches spent less than 1 hour on the telephone with participants, communicating primarily through text messages. Text messaging is simple, unobtrusive, acceptable to patients, and overcomes the often high rates of disengagement frequently seen in phone-based coaching.7,9

    Limitations

    Quiz Ref IDStudy limitations should be considered in interpreting results. Although previous studies20 of the mobile platform have shown maintenance of gains over 6 months,20 this trial did not have a follow-up period sufficient to replicate those findings. By using self-report depression and anxiety measures to align with screening/monitoring methods in primary care,44 findings may not generalize to mental health clinics where thorough clinical assessments are performed. Because 56% of the sample was receiving antidepressant medication, it is possible some of the within-treatment effects were owing to medication; however, these effects should be equivalent across treatment arms and do not affect the validity of the between-treatment effects. We also did not capture data on patients’ use of other apps during the intervention period. Finally, although the research was conducted in partnership with UAMS clinics, study team members delivered the intervention or assessments (performed by different individuals to prevent coaches from knowing outcomes). Future work will be required to design and evaluate implementation strategies that integrate DMHI app platforms into clinic workflows. Further, design work to support translation to adolescent populations and care settings may be beneficial given the prevalence of mental health problems and unmet treatment need among youths.45

    Conclusions

    The mobile app is effective for depression and anxiety among primary care patients, which represents an important innovation for alleviating the burden of these mental health problems in this population. Findings support the movement of DMHIs from single apps for depression or anxiety to platform approaches containing multiple brief apps that patients can bundle to meet their needs and fit into the fabric of their lives.

    Back to top
    Article Information

    Corresponding Author: David C. Mohr, PhD, Center for Behavioral Intervention Technologies, Department of Preventive Medicine, Northwestern University, 750 N Lake Shore Dr, 10th Floor, Chicago, IL 60611 (d-mohr@northwestern.edu).

    Accepted for Publication: March 9, 2020.

    Published Online: May 20, 2020. doi:10.1001/jamapsychiatry.2020.1011

    Author Contributions: Drs Mohr and Kwasny had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Greene, Lieponis, Mohr.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Graham, Greene, Kaiser, Mohr.

    Critical revision of the manuscript for important intellectual content: Graham, Greene, Kwasny, Lieponis, Powell, Mohr.

    Statistical analysis: Kwasny, Powell, Mohr.

    Obtained funding: Greene, Mohr.

    Administrative, technical, or material support: Graham, Greene, Kaiser, Lieponis, Mohr.

    Supervision: Graham, Greene, Mohr.

    Conflict of Interest Disclosures: Dr Mohr has an ownership interest in Adaptive Health Inc, which has a license from Northwestern University to commercialize IntelliCare. Drs Graham and Kwasny and Ms Kaiser have received consulting fees from Actualize Therapy LLC. Dr Graham reported grants from the National Institute of Diabetes and Digestive and Kidney Diseases during the conduct of the study and from the National Institute of Mental Health outside the submitted work. Dr Greene reported grants from Actualize Therapy during the conduct of the study. Dr Powell reported grants from the National Institutes of Health Small Business Innovation Research during the conduct of the study. Dr Mohr reported grants from the National Institute of Mental Health during the conduct of the study; personal fees from Apple Inc; and other support from Actualize Therapy Inc and Otsuka Pharmaceuticals outside the submitted work; in addition, Dr Mohr had a patent to US Patent 15/654,245, 2018 pending. No other disclosures were reported.

    Funding/Support: This work was supported by grants from the National Institutes of Health (R44 MH114725 and K01 DK116925) as well as by the Translational Research Institute (U54 TR001629) through the National Center for Advancing Translational Sciences of the National Institutes of Health.

    Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Data Sharing Statement: See Supplement 3.

    References
    1.
    Ruscio  AM, Hallion  LS, Lim  CCW,  et al.  Cross-sectional comparison of the epidemiology of DSM-5 generalized anxiety disorder across the globe.   JAMA Psychiatry. 2017;74(5):465-475. doi:10.1001/jamapsychiatry.2017.0056PubMedGoogle ScholarCrossref
    2.
    Hasin  DS, Sarvet  AL, Meyers  JL,  et al.  Epidemiology of adult DSM-5 major depressive disorder and its specifiers in the United States.   JAMA Psychiatry. 2018;75(4):336-346. doi:10.1001/jamapsychiatry.2017.4602PubMedGoogle ScholarCrossref
    3.
    Olfson  M, Blanco  C, Marcus  SC.  Treatment of adult depression in the United States.   JAMA Intern Med. 2016;176(10):1482-1491. doi:10.1001/jamainternmed.2016.5057PubMedGoogle ScholarCrossref
    4.
    Siu  AL, Bibbins-Domingo  K, Grossman  DC,  et al; US Preventive Services Task Force (USPSTF).  Screening for depression in adults: US Preventive Services Task Force Recommendation Statement.   JAMA. 2016;315(4):380-387. doi:10.1001/jama.2015.18392PubMedGoogle ScholarCrossref
    5.
    O’Connor  EA, Whitlock  EP, Beil  TL, Gaynes  BN.  Screening for depression in adult patients in primary care settings: a systematic evidence review.   Ann Intern Med. 2009;151(11):793-803. doi:10.7326/0003-4819-151-11-200912010-00007PubMedGoogle ScholarCrossref
    6.
    McCrone  P, Knapp  M, Proudfoot  J,  et al.  Cost-effectiveness of computerised cognitive-behavioural therapy for anxiety and depression in primary care: randomised controlled trial.   Br J Psychiatry. 2004;185:55-62. doi:10.1192/bjp.185.1.55PubMedGoogle ScholarCrossref
    7.
    Gilbody  S, Littlewood  E, Hewitt  C,  et al; REEACT Team.  Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial.   BMJ. 2015;351:h5627. doi:10.1136/bmj.h5627PubMedGoogle ScholarCrossref
    8.
    Andrews  G, Basu  A, Cuijpers  P,  et al.  Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis.   J Anxiety Disord. 2018;55:70-78. doi:10.1016/j.janxdis.2018.01.001PubMedGoogle ScholarCrossref
    9.
    Gilbody  S, Brabyn  S, Lovell  K,  et al; REEACT collaborative.  Telephone-supported computerised cognitive-behavioural therapy: REEACT-2 large-scale pragmatic randomised controlled trial.   Br J Psychiatry. 2017;210(5):362-367. doi:10.1192/bjp.bp.116.192435PubMedGoogle ScholarCrossref
    10.
    Christensen  H, Griffiths  KM, Farrer  L.  Adherence in internet interventions for anxiety and depression.   J Med Internet Res. 2009;11(2):e13. doi:10.2196/jmir.1194PubMedGoogle Scholar
    11.
    Vaish  R, Wyngarden  K, Chen  J, Cheung  B, Bernstein  MS. Twitch crowdsourcing: crowd contributions in short bursts of time. Presented at: SIGCHI Conference on Human Factors in Computing Systems; 2014; Toronto, ON, Canada.
    12.
    Andrews  S, Ellis  DA, Shaw  H, Piwek  L.  Beyond self-report: tools to compare estimated and real-world smartphone use.   PLoS One. 2015;10(10):e0139004. doi:10.1371/journal.pone.0139004PubMedGoogle Scholar
    13.
    Baumel  A, Muench  F, Edan  S, Kane  JM.  Objective user engagement with mental health apps: systematic search and panel-based usage analysis.   J Med Internet Res. 2019;21(9):e14567. doi:10.2196/14567PubMedGoogle Scholar
    14.
    Firth  J, Torous  J, Nicholas  J,  et al.  The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials.   World Psychiatry. 2017;16(3):287-298. doi:10.1002/wps.20472PubMedGoogle ScholarCrossref
    15.
    Firth  J, Torous  J, Nicholas  J, Carney  R, Rosenbaum  S, Sarris  J.  Can smartphone mental health interventions reduce symptoms of anxiety? a meta-analysis of randomized controlled trials.   J Affect Disord. 2017;218:15-22. doi:10.1016/j.jad.2017.04.046PubMedGoogle ScholarCrossref
    16.
    Kwasny  MJ, Schueller  SM, Lattie  E, Gray  EL, Mohr  DC.  Exploring the use of multiple mental health apps within a platform: secondary analysis of the IntelliCare Field Trial.   JMIR Ment Health. 2019;6(3):e11572. doi:10.2196/11572PubMedGoogle Scholar
    17.
    Chiauzzi  E, Newell  A.  Mental health apps in psychiatric treatment: a patient perspective on real world technology usage.   JMIR Ment Health. 2019;6(4):e12292. doi:10.2196/12292PubMedGoogle Scholar
    18.
    Barlow  DH, Farchione  TJ, Bullis  JR,  et al.  The unified protocol for transdiagnostic treatment of emotional disorders compared with diagnosis-specific protocols for anxiety disorders: a randomized clinical trial.   JAMA Psychiatry. 2017;74(9):875-884. doi:10.1001/jamapsychiatry.2017.2164PubMedGoogle ScholarCrossref
    19.
    Mohr  DC, Tomasino  KN, Lattie  EG,  et al.  IntelliCare: an eclectic, skills-based app suite for the treatment of depression and anxiety.   J Med Internet Res. 2017;19(1):e10. doi:10.2196/jmir.6645PubMedGoogle Scholar
    20.
    Mohr  DC, Schueller  SM, Tomasino  KN,  et al.  Comparison of the effects of coaching and receipt of app recommendations on depression, anxiety, and engagement in the IntelliCare platform: factorial randomized controlled trial.   J Med Internet Res. 2019;21(8):e13609. doi:10.2196/13609PubMedGoogle Scholar
    21.
    Zhang  R, Nicholas  J, Knapp  AA,  et al.  Clinically meaningful use of mental health apps and its effects on depression: mixed methods study.   J Med Internet Res. 2019;21(12):e15644. doi:10.2196/15644PubMedGoogle Scholar
    22.
    Kroenke  K, Strine  TW, Spitzer  RL, Williams  JB, Berry  JT, Mokdad  AH.  The PHQ-8 as a measure of current depression in the general population.   J Affect Disord. 2009;114(1-3):163-173. doi:10.1016/j.jad.2008.06.026PubMedGoogle ScholarCrossref
    23.
    Spitzer  RL, Kroenke  K, Williams  JB, Löwe  B.  A brief measure for assessing generalized anxiety disorder: the GAD-7.   Arch Intern Med. 2006;166(10):1092-1097. doi:10.1001/archinte.166.10.1092PubMedGoogle ScholarCrossref
    24.
    Posner  K, Brown  GK, Stanley  B,  et al.  The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults.   Am J Psychiatry. 2011;168(12):1266-1277. doi:10.1176/appi.ajp.2011.10111704PubMedGoogle ScholarCrossref
    25.
    Graham  AK, Greene  CJ, Powell  T,  et al.  Lessons learned from service design of a trial of a digital mental health service: informing implementation in primary care clinics.   Transl Behav Med. In press.Google Scholar
    26.
    Tomasino  KN, Noth  F, Bardsley  LR, Lattie  EG, Mohr  DC. IntelliCare Study Coaching Manual. Northwestern University: DigitalHub. Published 2016. Accessed April 13, 2020. https://digitalhub.northwestern.edu/files/00fa4294-5b9f-4afc-897a-7fffceae8f3f
    27.
    Kroenke  K, Spitzer  RL, Williams  JB.  The PHQ-9: validity of a brief depression severity measure.   J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.xPubMedGoogle ScholarCrossref
    28.
    Pham  Q, Graham  G, Carrion  C,  et al.  A library of analytic indicators to evaluate effective engagement with consumer mhealth apps for chronic conditions: scoping review.   JMIR Mhealth Uhealth. 2019;7(1):e11941. doi:10.2196/11941PubMedGoogle Scholar
    29.
    Kraemer  HC.  Is it time to ban the P value?   JAMA Psychiatry. 2019. doi:10.1001/jamapsychiatry.2019.1965PubMedGoogle Scholar
    30.
    Cuijpers  P, Van Straten  A, Warmerdam  L, Smits  N.  Characteristics of effective psychological treatments of depression: a metaregression analysis.   Psychother Res. 2008;18(2):225-236. doi:10.1080/10503300701442027PubMedGoogle ScholarCrossref
    31.
    Cuijpers  P, Sijbrandij  M, Koole  S, Huibers  M, Berking  M, Andersson  G.  Psychological treatment of generalized anxiety disorder: a meta-analysis.   Clin Psychol Rev. 2014;34(2):130-140. doi:10.1016/j.cpr.2014.01.002PubMedGoogle ScholarCrossref
    32.
    Rollman  BL, Herbeck Belnap  B, Abebe  KZ,  et al.  Effectiveness of online collaborative care for treating mood and anxiety disorders in primary care: a randomized clinical trial.   JAMA Psychiatry. 2018;75(1):56-64. doi:10.1001/jamapsychiatry.2017.3379PubMedGoogle ScholarCrossref
    33.
    Mohr  DC, Weingardt  KR, Reddy  M, Schueller  SM.  Three problems with current digital mental health research . . . and three things we can do about them.   Psychiatr Serv. 2017;68(5):427-429. doi:10.1176/appi.ps.201600541PubMedGoogle ScholarCrossref
    34.
    Mohr  DC, Ho  J, Duffecy  J,  et al.  Perceived barriers to psychological treatments and their relationship to depression.   J Clin Psychol. 2010;66(4):394-409. doi:10.1002/jclp.20659PubMedGoogle Scholar
    35.
    Mohr  DC, Hart  SL, Howard  I,  et al.  Barriers to psychotherapy among depressed and nondepressed primary care patients.   Ann Behav Med. 2006;32(3):254-258. doi:10.1207/s15324796abm3203_12PubMedGoogle ScholarCrossref
    36.
    Graham  AK, Lattie  EG, Mohr  DC.  Experimental therapeutics for digital mental health.   JAMA Psychiatry. 2019. doi:10.1001/jamapsychiatry.2019.2075PubMedGoogle Scholar
    37.
    Mohr  DC, Riper  H, Schueller  SM.  A solution-focused research approach to achieve an implementable revolution in digital mental health.   JAMA Psychiatry. 2018;75(2):113-114. doi:10.1001/jamapsychiatry.2017.3838PubMedGoogle ScholarCrossref
    38.
    Mohr  DC, Lyon  AR, Lattie  EG, Reddy  M, Schueller  SM.  Accelerating digital mental health research from early design and creation to successful implementation and sustainment.   J Med Internet Res. 2017;19(5):e153. doi:10.2196/jmir.7725PubMedGoogle Scholar
    39.
    Donkin  L, Christensen  H, Naismith  SL, Neal  B, Hickie  IB, Glozier  N.  A systematic review of the impact of adherence on the effectiveness of e-therapies.   J Med Internet Res. 2011;13(3):e52. doi:10.2196/jmir.1772PubMedGoogle Scholar
    40.
    Fuhr  K, Schröder  J, Berger  T,  et al.  The association between adherence and outcome in an Internet intervention for depression.   J Affect Disord. 2018;229:443-449. doi:10.1016/j.jad.2017.12.028PubMedGoogle ScholarCrossref
    41.
    Mohr  DC, Duffecy  J, Jin  L,  et al.  Multimodal e-mental health treatment for depression: a feasibility trial.   J Med Internet Res. 2010;12(5):e48. doi:10.2196/jmir.1370PubMedGoogle Scholar
    42.
    Gilbody  S, Bower  P, Fletcher  J, Richards  D, Sutton  AJ.  Collaborative care for depression: a cumulative meta-analysis and review of longer-term outcomes.   Arch Intern Med. 2006;166(21):2314-2321. doi:10.1001/archinte.166.21.2314PubMedGoogle ScholarCrossref
    43.
    Muntingh  AD, van der Feltz-Cornelis  CM, van Marwijk  HW, Spinhoven  P, van Balkom  AJ.  Collaborative care for anxiety disorders in primary care: a systematic review and meta-analysis.   BMC Fam Pract. 2016;17:62. doi:10.1186/s12875-016-0466-3PubMedGoogle ScholarCrossref
    44.
    The MacArthur Foundation’s Initiative on Depression and Primary Care. The MacArthur Initiative on Depression and Primary Care at Dartmouth and Duke: Depression Management Toolkit. Hanover, NH: Dartmouth;2004.
    45.
    Whitney  DG, Peterson  MD.  US national and state-level prevalence of mental health disorders and disparities of mental health care use in children.   JAMA Pediatr. 2019;173(4):389-391. doi:10.1001/jamapediatrics.2018.5399PubMedGoogle ScholarCrossref
    ×