Effect of a Digital Intervention on Depressive Symptoms in Patients With Comorbid Hypertension or Diabetes in Brazil and Peru: Two Randomized Clinical Trials | Depressive Disorders | JAMA | JAMA Network
[Skip to Navigation]
Sign In
Visual Abstract. Digital Intervention for Depressive Symptoms in Patients With Comorbid Hypertension or Diabetes
Digital Intervention for Depressive Symptoms in Patients With Comorbid Hypertension or Diabetes
Figure 1.  Flow of Participant Recruitment, Randomization, and Follow-up for the São Paulo, Brazil, Trial
Flow of Participant Recruitment, Randomization, and Follow-up for the São Paulo, Brazil, Trial

aSome participants met more than 1 ineligibility criterion.

bIndicates 44 participants in each unit.

cNot all participants who remained enrolled in the trials attended follow-up assessments or completed all Patient Health Questionnaire-9 (PHQ-9) items. Numbers are included here to allow matching of the number of participants with tables presenting results. Eight participants in the intervention group and 11 participants in the enhanced usual care group who did not complete 3-month follow-up completed 6-month follow-up.

dThe number of participants excluding those who withdrew consent or died (denominators) was 392 for the digital intervention group and 399 for the enhanced usual care group.

Figure 2.  Flow of Participant Recruitment, Randomization, and Follow-up for the Trial in Lima, Peru
Flow of Participant Recruitment, Randomization, and Follow-up for the Trial in Lima, Peru

aSome participants met more than 1 ineligibility criterion.

bNot all participants who remained enrolled in the trials attended follow-up assessments or completed all Patient Health Questionnaire-9 (PHQ-9) items. Numbers are included here to allow matching of the number of participants with tables presenting results. Three participants in the intervention group and 6 participants in the enhanced usual care group who did not attend 3-month follow-up attended 6-month follow-up.

cThe number of participants excluding those who withdrew consent or died (denominators) was 209 for the digital intervention group and 208 for the enhanced usual care group.

Figure 3.  PHQ-9 Scores at Baseline vs 3 Months and Violin Plots of PHQ-9 Scores at 3 Months in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
PHQ-9 Scores at Baseline vs 3 Months and Violin Plots of PHQ-9 Scores at 3 Months in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru

In both scatterplots (left panels), the diagonal blue line divides participants who improved in Patient Health Questionnaire-9 (PHQ-9) scores at 3 months (below the line) from those who got worse (above the line). The diagonal brown line divides participants who achieved at least a 50% reduction from baseline PHQ-9 scores at 3-month follow-up (below the line) from those who did not achieve it (above the line), to illustrate the primary outcome. In the violin plots, the horizontal line in the box indicates the median, upper and lower ends of the boxes indicate the 25th and 75th quartiles, and the whiskers indicate the highest and lowest values within 1.5 times the interquartile range (IQR) of PHQ-9 scores. The wider gray shaded areas in the violin plots indicate a higher probability that members of the population will take on the given value; thinner areas indicate a lower probability. In São Paulo, the median score was 9 (IQR, 6-14) for the digital intervention group and 11 (IQR, 7-15) for the enhanced usual care group. In Lima, the median score was 6 (IQR, 2-12) in the digital intervention group and 10 (IQR, 6-14) in the enhanced usual care group.

Table 1.  Baseline Characteristics in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
Baseline Characteristics in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
Table 2.  PHQ-9 Primary and Secondary Outcomes at 3- and 6-Month Follow-up Assessments Among Participants in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
PHQ-9 Primary and Secondary Outcomes at 3- and 6-Month Follow-up Assessments Among Participants in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
Table 3.  Secondary Outcomes, Quality of Life (EQ5D-3L), Functioning (WHODAS-II), and Behavioral Activation (BADS-SF) at 3- and 6-Month Follow-up Among Participants in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
Secondary Outcomes, Quality of Life (EQ5D-3L), Functioning (WHODAS-II), and Behavioral Activation (BADS-SF) at 3- and 6-Month Follow-up Among Participants in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
Table 4.  Adverse and Unexpected Events in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
Adverse and Unexpected Events in the Digital Intervention and Enhanced Usual Care Groups in São Paulo, Brazil, and Lima, Peru
1.
World Health Organization.  Depression and Other Common Mental Disorders: Global Health Estimates. World Health Organization; 2017.
2.
Pan American Health Organization.  The Burden of Mental Disorders in the Region of the Americas, 2018. Pan American Health Organization; 2018.
3.
Gold  SM, Köhler-Forsberg  O, Moss-Morris  R,  et al.  Comorbid depression in medical diseases.   Nat Rev Dis Primers. 2020;6(1):69. doi:10.1038/s41572-020-0200-2PubMedGoogle ScholarCrossref
4.
Macinko  J, Andrade  FCD, Nunes  BP, Guanais  FC.  Primary care and multimorbidity in six Latin American and Caribbean countries.   Rev Panam Salud Publica. 2019;43:e8-e8. doi:10.26633/RPSP.2019.8PubMedGoogle ScholarCrossref
5.
Chibanda  D, Weiss  HA, Verhey  R,  et al.  Effect of a primary care-based psychological intervention on symptoms of common mental disorders in Zimbabwe: a randomized clinical trial.   JAMA. 2016;316(24):2618-2626. doi:10.1001/jama.2016.19102PubMedGoogle ScholarCrossref
6.
Patel  V, Weobong  B, Weiss  HA,  et al.  The Healthy Activity Program (HAP), a lay counsellor-delivered brief psychological treatment for severe depression, in primary care in India: a randomised controlled trial.   Lancet. 2017;389(10065):176-185. doi:10.1016/S0140-6736(16)31589-6PubMedGoogle ScholarCrossref
7.
Araya  R, Rojas  G, Fritsch  R,  et al.  Treating depression in primary care in low-income women in Santiago, Chile: a randomised controlled trial.   Lancet. 2003;361(9362):995-1000. doi:10.1016/S0140-6736(03)12825-5PubMedGoogle ScholarCrossref
8.
Torous  J, Cerrato  P, Halamka  J.  Targeting depressive symptoms with technology.   Mhealth. 2019;5:19. doi:10.21037/mhealth.2019.06.04PubMedGoogle ScholarCrossref
9.
Firth  J, Torous  J, Nicholas  J,  et al.  The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials.   World Psychiatry. 2017;16(3):287-298. doi:10.1002/wps.20472PubMedGoogle ScholarCrossref
10.
Kuehn  BM.  Global mental health services are collapsing as demand grows.   JAMA. 2020;324(19):1933-1933. doi:10.1001/jama.2020.22085PubMedGoogle Scholar
11.
Shariful Islam  SM, Chow  CK, Redfern  J,  et al.  Effect of text messaging on depression in patients with coronary heart disease: a substudy analysis from the TEXT ME randomised controlled trial.   BMJ Open. 2019;9(2):e022637. doi:10.1136/bmjopen-2018-022637PubMedGoogle Scholar
12.
Prabhakaran  D, Jha  D, Prieto-Merino  D,  et al.  Effectiveness of an mHealth-based electronic decision support system for integrated management of chronic conditions in primary care: the mWellcare cluster-randomized controlled trial.   Circulation. 2019;139(3):380-391. doi:10.1161/CIRCULATIONAHA.118.038192PubMedGoogle ScholarCrossref
13.
Mohr  DC, Lyon  AR, Lattie  EG, Reddy  M, Schueller  SM.  Accelerating digital mental health research from early design and creation to successful implementation and sustainment.   J Med Internet Res. 2017;19(5):e153. doi:10.2196/jmir.7725PubMedGoogle Scholar
14.
Brandt  LR, Hidalgo  L, Diez-Canseco  F,  et al.  Addressing depression comorbid with diabetes or hypertension in resource-poor settings: a qualitative study about user perception of a nurse-supported smartphone app in Peru.   JMIR Ment Health. 2019;6(6):e11701. doi:10.2196/11701PubMedGoogle Scholar
15.
Menezes  P, Quayle  J, Garcia Claro  H,  et al.  Use of a mobile phone app to treat depression comorbid with hypertension or diabetes: a pilot study in Brazil and Peru.   JMIR Ment Health. 2019;6(4):e11698. doi:10.2196/11698PubMedGoogle Scholar
16.
Kroenke  K, Spitzer  RL, Williams  JB.  The PHQ-9: validity of a brief depression severity measure.   J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.xPubMedGoogle ScholarCrossref
17.
Sealed Envelope.  Randomisation and online databases for clinical trials. Access verified April 22, 2021. https://www.sealedenvelope.com/
18.
Ekers  D, Webster  L, Van Straten  A, Cuijpers  P, Richards  D, Gilbody  S.  Behavioural activation for depression; an update of meta-analysis of effectiveness and sub group analysis.   PLoS One. 2014;9(6):e100100. doi:10.1371/journal.pone.0100100PubMedGoogle Scholar
19.
Mohr  DC, Cuijpers  P, Lehman  K.  Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions.   J Med Internet Res. 2011;13(1):e30. doi:10.2196/jmir.1602PubMedGoogle Scholar
20.
Coley  RY, Boggs  JM, Beck  A, Hartzler  AL, Simon  GE.  Defining success in measurement-based care for depression: a comparison of common metrics.   Psychiatr Serv. 2020;71(4):312-318. doi:10.1176/appi.ps.201900295PubMedGoogle ScholarCrossref
21.
Cuijpers  P, Stringaris  A, Wolpert  M.  Treatment outcomes for depression: challenges and opportunities.   Lancet Psychiatry. 2020;7(11):925-927. doi:10.1016/S2215-0366(20)30036-5PubMedGoogle ScholarCrossref
22.
McMillan  D, Gilbody  S, Richards  D.  Defining successful treatment outcome in depression using the PHQ-9: a comparison of methods.   J Affect Disord. 2010;127(1-3):122-129. doi:10.1016/j.jad.2010.04.030PubMedGoogle ScholarCrossref
23.
Rabin  R, de Charro  F.  EQ-5D: a measure of health status from the EuroQol Group.   Ann Med. 2001;33(5):337-343. doi:10.3109/07853890109002087PubMedGoogle ScholarCrossref
24.
World Health Organization.  World Health Organization Disability Assessment Schedule: WHODAS II. World Health Organization; 2000.
25.
Rojas  G, Fritsch  R, Solis  J,  et al.  Treatment of postnatal depression in low-income mothers in primary-care clinics in Santiago, Chile: a randomised controlled trial.   Lancet. 2007;370(9599):1629-1637. doi:10.1016/S0140-6736(07)61685-7PubMedGoogle ScholarCrossref
26.
Patel  V, Weiss  HA, Chowdhary  N,  et al.  Effectiveness of an intervention led by lay health counsellors for depressive and anxiety disorders in primary care in Goa, India (MANAS): a cluster randomised controlled trial.   Lancet. 2010;376(9758):2086-2095. doi:10.1016/S0140-6736(10)61508-5PubMedGoogle ScholarCrossref
27.
Singla  DR, Waqas  A, Hamdani  SU,  et al.  Implementation and effectiveness of adolescent life skills programs on low- and middle-income countries: a critical review and meta-analysis.   Behav Res Ther. 2020;130:103402. doi:10.1016/j.brat.2019.04.010PubMedGoogle Scholar
28.
Kounali  D, Button  KS, Lewis  G,  et al.  How much change is enough? evidence from a longitudinal study on depression in UK primary care.   Psychol Med. 2020;1-8. doi:10.1017/S0033291720003700PubMedGoogle Scholar
29.
White  IR, Royston  P, Wood  AM.  Multiple imputation using chained equations: issues and guidance for practice.   Stat Med. 2011;30(4):377-399. doi:10.1002/sim.4067PubMedGoogle ScholarCrossref
30.
Sawyer  A, Kaim  A, Le  H-N,  et al.  The effectiveness of an app-based nurse-moderated program for new mothers with depression and parenting problems (eMums Plus): pragmatic randomized controlled trial.   J Med Internet Res. 2019;21(6):e13689. doi:10.2196/13689PubMedGoogle Scholar
31.
Arjadi  R, Nauta  MH, Scholte  WF,  et al.  Internet-based behavioural activation with lay counsellor support versus online minimal psychoeducation without support for treatment of depression: a randomised controlled trial in Indonesia.   Lancet Psychiatry. 2018;5(9):707-716. doi:10.1016/S2215-0366(18)30223-2PubMedGoogle ScholarCrossref
32.
Graham  AK, Lattie  EG, Mohr  DC.  Experimental therapeutics for digital mental health.   JAMA Psychiatry. 2019;76(12):1223-1224. doi:10.1001/jamapsychiatry.2019.2075PubMedGoogle ScholarCrossref
33.
Naslund  JA, Aschbrenner  KA, Araya  R,  et al.  Digital technology for treating and preventing mental disorders in low-income and middle-income countries: a narrative review of the literature.   Lancet Psychiatry. 2017;4(6):486-500. doi:10.1016/S2215-0366(17)30096-2PubMedGoogle ScholarCrossref
34.
Jiménez-Molina  Á, Franco  P, Martínez  V, Martínez  P, Rojas  G, Araya  R.  Internet-based interventions for the prevention and treatment of mental disorders in Latin America: a scoping review.   Front Psychiatry. 2019;10:664. doi:10.3389/fpsyt.2019.00664PubMedGoogle ScholarCrossref
35.
Carter  H, Araya  R, Anjur  K, Deng  D, Naslund  JA.  The emergence of digital mental health in low-income and middle-income countries: a review of recent advances and implications for the treatment and prevention of mental disorders.   J Psychiatr Res. 2021;133:223-246. doi:10.1016/j.jpsychires.2020.12.016PubMedGoogle ScholarCrossref
Original Investigation
May 11, 2021

Effect of a Digital Intervention on Depressive Symptoms in Patients With Comorbid Hypertension or Diabetes in Brazil and Peru: Two Randomized Clinical Trials

Author Affiliations
  • 1Centre for Global Mental Health, Institute of Psychiatry, Psychology, and Neuroscience, King’s College London, London, United Kingdom
  • 2Population Mental Health Research Centre, Universidade de São Paulo, São Paulo, Brazil
  • 3Department of Preventive Medicine, Faculdade de Medicina, Universidade de São Paulo, São Paulo, Brazil
  • 4School of Nursing, Universidade Estadual de Campinas, Campinas, Brazil
  • 5CRONICAS Center of Excellence in Chronic Diseases, Universidad Peruana Cayetano Heredia, Lima, Peru
  • 6Bristol Medical School, Faculty of Health Sciences, University of Bristol, Bristol, United Kingdom
  • 7Federal University of ABC, Engineering, Modeling and Applied Social Sciences Center (CECS), Santo André, Brazil
  • 8Center for Behavioral Intervention Technologies, Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
  • 9Department of Medicine, School of Medicine, Universidad Peruana Cayetano Heredia, Lima, Peru
JAMA. 2021;325(18):1852-1862. doi:10.1001/jama.2021.4348
Visual Abstract. Digital Intervention for Depressive Symptoms in Patients With Comorbid Hypertension or Diabetes
Digital Intervention for Depressive Symptoms in Patients With Comorbid Hypertension or Diabetes
Key Points

Question  What is the effect of a behavioral activation digital intervention delivered over a 6-week period on depressive symptoms among patients with comorbid hypertension or diabetes in Brazil and Peru?

Findings  In 2 randomized clinical trials conducted separately in São Paulo, Brazil (880 participants), and Lima, Peru (432 participants), a significantly greater proportion of participants who received the digital intervention, compared with enhanced usual care, experienced at least a 50% reduction in depressive symptoms at 3 months (40.7% vs 28.6% in Brazil; [odds ratio, 1.6]; 52.7% vs 34.1% in Peru [odds ratio, 2.1]), although the differences were no longer statistically significant at 6 months.

Meaning  A digital intervention for patients with depressive symptoms and comorbid hypertension or diabetes significantly improved depressive symptoms at 3 months compared with enhanced usual care in Peru and Brazil, but the effects were not sustained at 6 months in either of the 2 trials.

Abstract

Importance  Depression is a leading contributor to disease burden globally. Digital mental health interventions can address the treatment gap in low- and middle-income countries, but the effectiveness in these countries is unknown.

Objective  To investigate the effectiveness of a digital intervention in reducing depressive symptoms among people with diabetes and/or hypertension.

Design, Setting, and Participants  Participants with clinically significant depressive symptoms (Patient Health Questionnaire-9 [PHQ-9] score ≥10) who were being treated for hypertension and/or diabetes were enrolled in a cluster randomized clinical trial (RCT) at 20 sites in São Paulo, Brazil (N=880; from September 2016 to September 2017; final follow-up, April 2018), and in an individual-level RCT at 7 sites in Lima, Peru (N=432; from January 2017 to September 2017; final follow-up, March 2018).

Interventions  An 18-session, low-intensity, digital intervention was delivered over 6 weeks via a provided smartphone, based on behavioral activation principles, and supported by nurse assistants (n = 440 participants in 10 clusters in São Paulo; n = 217 participants in Lima) vs enhanced usual care (n = 440 participants in 10 clusters in São Paulo; n = 215 participants in Lima).

Main Outcomes and Measures  The primary outcome was a reduction of at least 50% from baseline in PHQ-9 scores (range, 0-27; higher score indicates more severe depression) at 3 months. Secondary outcomes included a reduction of at least 50% from baseline PHQ-9 scores at 6 months.

Results  Among 880 patients cluster randomized in Brazil (mean age, 56.0 years; 761 [86.5%] women) and 432 patients individually randomized in Peru (mean age, 59.7 years; 352 [81.5%] women), 807 (91.7%) in Brazil and 426 (98.6%) in Peru completed at least 1 follow-up assessment. The proportion of participants in São Paulo with a reduction in PHQ-9 score of at least 50% at 3-month follow-up was 40.7% (159/391 participants) in the digital intervention group vs 28.6% (114/399 participants) in the enhanced usual care group (difference, 12.1 percentage points [95% CI, 5.5 to 18.7]; adjusted odds ratio [OR], 1.6 [95% CI, 1.2 to 2.2]; P = .001). In Lima, the proportion of participants with a reduction in PHQ-9 score of at least 50% at 3-month follow-up was 52.7% (108/205 participants) in the digital intervention group vs 34.1% (70/205 participants) in the enhanced usual care group (difference, 18.6 percentage points [95% CI, 9.1 to 28.0]; adjusted OR, 2.1 [95% CI, 1.4 to 3.2]; P < .001). At 6-month follow-up, differences across groups were no longer statistically significant.

Conclusions and Relevance  In 2 RCTs of patients with hypertension or diabetes and depressive symptoms in Brazil and Peru, a digital intervention delivered over a 6-week period significantly improved depressive symptoms at 3 months when compared with enhanced usual care. However, the magnitude of the effect was small in the trial from Brazil and the effects were not sustained at 6 months.

Trial Registration  ClinicalTrials.gov: NCT02846662 (São Paulo) and NCT03026426 (Lima)

Introduction

Depression has been the largest contributor to total years lived with disability in the Americas, with Brazil and Peru among the highest contributors in 2015.1,2 Depression has been commonly associated with diabetes and cardiovascular diseases, a comorbidity associated with poorer treatment adherence and outcomes,3 posing a major burden in Latin America.4

The lack of trained personnel to deliver effective mental health care has contributed greatly to a large treatment gap in Brazil and Peru.2 Task shifting or transferring tasks from specialists to other health workers has been used to reduce this gap in low- and middle-income countries.5-7 Brazil and Peru have targeted programs within their health care systems to treat hypertension and/or diabetes, but not depression, with a prominent task-shifted role given to nonspecialists.

The use of digital interventions is increasing,8,9 a trend that has accelerated since the advent of the COVID-19 pandemic.10 Digital interventions have demonstrated efficacy, particularly when including human support,9 and offer potentially scalable and affordable solutions to reduce the depression treatment gap in low- and middle-income countries, but evidence from these countries and among patients with chronic physical health comorbidities is lacking. Studies of the effectiveness of digital interventions in populations with chronic comorbid conditions, primarily in high-income countries, have shown mixed results, in part due to adherence problems.8,11,12 To address this, a user-centered approach was utilized to design the intervention, with participation of patients and nurses.13

A low-intensity, behavioral activation, digital intervention named CONEMO (English translation, emotional control) for depression among individuals with hypertension and/or diabetes was developed. The digital intervention was minimally supported by nurse assistants, and feasibility studies showed good acceptability among users.14,15

Two randomized clinical trials (RCTs) were conducted to assess the effect of the intervention on depressive symptoms among individuals with hypertension and/or diabetes attending public health care facilities in Sao Paulo, Brazil, and Lima, Peru.

Methods

Trials were approved by the data and safety monitoring board of the US National Institute of Mental Health and local ethics committees in São Paulo and Lima. All participants gave written informed consent prior to entering the trials. Protocols and the statistical analysis plan are provided in Supplement 1.

The trial designs and the type of human support provided were different, including the health care systems, yet both trials used the same digital intervention for populations with similar comorbid conditions.

Settings and Participants

Quiz Ref IDEligible participants were adults (≥21 years) who reported receiving treatment for hypertension and/or diabetes at primary care units in São Paulo or attending ambulatory treatment clinics for hypertension or diabetes in Lima. Patients who had a Patient Health Questionnaire-9 (PHQ-9) score of 10 or greater (range, 0-27; higher score indicates more severe depression) and ability to read a text on a smartphone screen were invited. Individuals assessed as having high suicide risk and pregnant women with gestational diabetes and/or hypertension at screening were excluded.

Procedures

Quiz Ref IDA cluster RCT in São Paulo, Brazil, and an individual-level RCT in Lima, Peru, were conducted. In São Paulo, 35 primary care units (clusters) located in the eastern part of the city were invited to participate, but 9 of them were too small (with less than 4 family health teams), and 6 declined to participate. All the remaining 20 units, 10 teaching (internship sites) and 10 nonteaching units, agreed to participate. In Lima, 3 outpatient clinics and 4 primary care centers agreed to participate. Fieldwork took place between September 19, 2016, and April 2, 2018, in Brazil and between January 24, 2017, and March 30, 2018, in Peru. The PHQ-916 was used to assess eligibility, severity of depressive symptoms, and main outcomes.

Randomization

In São Paulo, the sample was stratified according to teaching status, with a single block in each stratum. A statistician, who was not involved in recruitment, undertook the cluster, stratified randomization of clinics blind to their identities, with 10 clinics randomized to each group, 5 from each stratum. In Lima, individual randomization was undertaken using a 1:1 allocation ratio, with balance attained with respect for any of the countries health center and baseline severity of depressive symptoms (PHQ-9 score <15 or ≥15) through stochastic minimization with a 30% chance of simple random allocation, using an online randomization system.17

Research assistants collecting baseline and outcome data from participants were blind to treatment allocation. Concealing allocation from clinical staff delivering the digital intervention or managing the safety net was not feasible.

Intervention and Control Conditions
Intervention Group: Digital Intervention

Quiz Ref IDThis was a low-intensity intervention aimed primarily at reducing depressive symptoms, delivered by a smartphone app in Portuguese and Spanish, and minimally supported by nurse assistants. The app consisted of 18 brief automated mini-sessions, delivered over a 6-week period at a rate of 3 mini sessions per week, each requiring less than 10 minutes to complete. The app content was based on behavioral activation, an evidence-based psychological approach to treat depression18 that focused on increasing participation in activities pleasant or meaningful to the participant that could be easily adapted for self-treatment. Although this digital intervention did not aim to improve the management of hypertension or diabetes, many suggested activities aimed to improve comorbid physical conditions (eg, healthy eating or physical activity). App use data were reviewed by nurse assistants through a dashboard installed on tablet computers (see outline in eFigure 1 in Supplement 2).

Nurse assistants met with intervention participants for an initial face-to-face meeting, and participants received a smartphone with the preinstalled app and completed a tutorial on its use. Nurse assistants provided support to the app using the supportive accountability-coaching model.19 At the beginning of the study, nurse assistants placed 2 mandatory phone calls to all intervention participants to assist with any difficulties and to enhance motivation for using the digital intervention. Additional calls were prompted through notifications sent to nurse assistants when the automated system detected nonadherence. If participants requested help regarding clinical issues, they were referred to their usual health care services. Patients could request technical assistance by using the help button in the app, which leads to a live call from the nurse assistant. There was no additional clinical contact as part of this digital intervention. Nurse assistants received training and were supervised weekly by clinical psychologists.

All other health services considered in the enhanced usual care group were also available in the digital intervention group and used at the discretion of local health teams, including treatment for depression, diabetes, or hypertension.

Control Group: Enhanced Usual Care

The treatment of depression, hypertension, or diabetes in this group was left to the discretion of local clinicians. Participants in the intervention and control groups were assessed for depressive symptoms up to 4 times during the first month and again during research follow-up assessments; if considered at risk, they were referred to specialist services as per the safety protocol (Supplement 1). None of these procedures were part of the preexisting usual care; thus, this was considered an enhanced usual care approach. Participants in the control group did not receive a smartphone.

Measures

Sociodemographics, clinical history of chronic conditions, and health care utilization were assessed by self-report. Outcome assessments occurred at 3- and 6-month follow-up after completing the intervention.

Primary Outcome

The primary outcome was improvement in depressive symptoms, defined as the proportion of participants with at least a 50% reduction from baseline PHQ-916 scores at 3-month follow-up assessments. This indicator has been used in many depression treatment trials and is considered a robust way of ascertaining depression treatment improvements.20-22

Secondary Outcomes

Secondary outcomes included the proportion of participants with a reduction of at least 50% from baseline PHQ-9 scores at 6-month follow-up assessments; quality of life measured by the 3-level version of the Euroqol Group Quality of life assessment instrument (EQ-5D-3L [score range, 0-1 with the greater score indicating highest quality of life])23; disability assessed with the World Health Organization Disability Assessment Schedule-II (WHODAS-II [score range, 0-100 with greater score indicating more disability])24; behavioral activation assessed with the Behavioral Activation for Depression Scale-Short Form (BADS-SF [score range, 0-54 with greater scores indicating higher levels of activation])21; and health care service utilization, as per number of health service consultations, hospital admissions, and home visits by primary care teams. All secondary outcomes were analyzed at 3 and 6 months.

Sample Size and Statistical Analyses

For the individual-level trial (in Lima) to detect a 15% difference (35% vs 50%) in the primary outcome across groups with 80% power and 2-sided 5% significance level, 183 participants per group were needed. With 15% attrition, 432 individuals were required in total. The same target was used in São Paulo, with inflation of sample size due to cluster randomization using an intracluster correlation coefficient of 0.025. All of the 20 clusters available in São Paulo recruited participants, leading to an inflated total sample size of 732 participants. Expecting a 15% loss to follow-up, the sample size needed was 842; this was increased to 880 to aim for 44 patients in each cluster in São Paulo.

A 15 percentage-point difference was a meaningful and realistic effect size when compared with other trials for depression in primary care conducted in in Chile,7,25 and elsewhere.6,26,27 A minimal clinically important difference for the main outcome variable (PHQ-9) has never been established for treatment of depressive symptoms in either of the countries involved or other low- and middle-income countries. Studies from high-income countries suggest that a difference between 17% and 20%28 can be considered as a reasonable minimal clinically important difference, but in view of the low intensity of the intervention, this study was powered for a 15% difference as a meaningful target from a clinical and public health perspective.

Data analyses were conducted separately for each trial using a similar approach. Descriptive baseline data were compared across groups to examine potential imbalances. Primary and secondary outcomes were analyzed according to their randomization group, with missing data handled through multiple imputation using chained equations methods,29 conducting sensitivity analysis with complete case analyses excluding those with missing data. All regression analyses were adjusted for the relevant stratification variables and random-effects parameters to account for clustering, including the clinic as the unit of randomization in São Paulo. The primary outcome analysis used logistic regression. In secondary analyses and according to a prespecified statistical analysis plan, results were adjusted for sex, education, income, age, chronic diseases, and marital status as well as baseline values for continuous outcome variables. Similar methods were used for the following secondary outcomes without adjustment for test multiplicity:

(1) a reduction of at least 50% in PHQ-9 score from baseline to 6 months, using random-effects logistic regression;

(2), EQ-5D-3L, WHODAS-II, and BADS-SF total scores at 3 and 6 months using random-effects linear regression models and repeated measures analyses; and

(3) use of health care services during 2 time periods (0-3 months and 3-6 months) using random-effects Poisson models in São Paulo and Poisson regression models in Lima.

Prespecified subgroup analyses using likelihood ratio tests of interactions across groups and educational levels (<9 vs ≥9 years) and baseline severity of depressive symptoms (PHQ-9 <15 vs ≥15) were performed using the continuous measures of these 2 potential effect modifiers without adjustment for confounders.

All statistical tests were 2-sided, and while specific P values are presented, those less than .05 were considered statistically significant. Because of the potential for type I error due to multiple comparisons, findings for analyses of secondary end points should be interpreted as exploratory. All analyses were conducted using STATA 15 (StataCorp).

Results
Sample and Participants’ Characteristics

Participant recruitment and retention (per CONSORT guidelines) is shown in Figure 1 for São Paulo and in Figure 2 for Lima. Of the 10 688 patients screened in São Paulo, 880 (8.2%) were enrolled, while in Lima, 432 (7.5%) were enrolled of 5785 patients screened. Enrolled participants were not notably different from those who declined (eTable 1 in Supplement 2). A total of 807 (91.7%) participants in São Paulo and 426 (98.6%) in Lima completed one or both follow-ups.

In both trials, individuals randomized to the digital intervention group had similar characteristics with those randomized to the enhanced usual care group at baseline. At baseline, 507 (57.6%) participants in São Paulo and 157 (36.3%) in Lima had PHQ-9 of 16 points or greater, suggesting at least moderately severe depression (Table 1).

In São Paulo, 70 participants (18%) did not complete any of the active intervention sessions, 286 (65%) completed at least 9 sessions, and 199 (45%) completed all 18 sessions. In Lima, where 4 (2%) did not complete any of the active intervention sessions, 200 (92%) completed at least 9 sessions, and 169 (78%) completed all 18 sessions.

Primary Outcome

Quiz Ref IDIn both trials, statistically significant differences in favor of the digital intervention groups at 3-month follow-up were found. In São Paulo, 159 of 391 (40.7%) participants in the digital intervention group who completed the 3-month outcomes had a reduction of at least 50% in the PHQ-9 score compared with 114 of 399 (28.6%) in the enhanced usual care group (between-group absolute difference, 12.1 percentage points [95% CI, 5.5-18.7]). In Lima, 108/205 (52.7%) participants improved in the digital intervention group compared with 70/205 (34.1%) in the control group (absolute difference, 18.6 percentage points [95% CI, 9.1-28.0]).

The odds ratio (OR) for this difference across groups, estimated after imputing missing values, was 1.6 (95% CI, 1.2-2.2) in São Paulo and 2.1 (95% CI, 1.4-3.2) in Lima (Table 2). These results persisted after adjustment for baseline covariates and in complete case analyses (eTable 2 in Supplement 2). The distributions of PHQ-9 scores for the primary outcome are shown in Figure 3.

Secondary Outcomes

Quiz Ref IDAt 6 months in São Paulo, 173 of 375 participants (46.1%) in the digital intervention group who completed the 6-month follow-up showed improvements in PHQ-9 scores, compared with 153 of 379 (40.4%) in the control group (between-group absolute difference, 5.7% [95% CI, −2.2% to 13.9%]; OR, 1.2 (95% CI, 0.9 to 1.7)]), and at 6 months in Lima, 112 of 203 (55.2%) participants improved in the digital intervention group and 101 of 197 (51.3%) in the control group (between-group absolute difference, 3.9% [95% CI, −5.3% to 13.9%]; OR, 1.2 [95% CI, 0.8 to 1.7]). None of these comparisons were statistically significant (Table 2).

At 3 months, in both trials, there were statistically significant differences between groups in favor of the digital intervention group for quality of life (EQ5D-3L) and disability (WHODAS-II), while BADS-SF total scores were significantly increased among digital intervention participants in Lima only (Table 3). Coefficients represent adjusted differences in means between groups. For instance, in Lima, the coefficient for WHODAS-II of −6.3 points (95% CI, −9.2 to −3.3) indicates an estimated difference in means between groups of 6.3 points on the WHODAS-II at 3 months, adjusting for baseline WHODAS-II scores and the randomization strata used in Lima.

At 6 months, there were no statistically significant differences between groups in the EQ-5D-3L or WHODAS-II total scores in either trial. There was a statistically significant difference in favor of the digital intervention group in the BADS-SF total score in both trials (Table 3). The analyses of repeated measures for all outcomes are provided in eFigure 2 in Supplement 2.

The mean number of medical consultations from baseline to the 3-month follow-up was significantly lower in the digital intervention group compared with the control group in São Paulo (regression coefficient, −0.2 [95% CI, −0.3 to −0.1]), while an effect in the opposite direction was seen in Lima (regression coefficient, 0.1 [95% CI, 0.0 to 0.2]). There were no significant differences in the number of consultations at 6 months in either trial. Also, there were no significant differences in the number of hospitalizations or home visits between groups at 3 and 6 months in either trial (eTable 3 in Supplement 2). All of these results remained similar after adjustment for baseline covariates and in complete case analyses (eTable 2 in Supplement 2).

Interactions

There was a statistically significant interaction between treatment group and time for the PHQ-9 scores in both trials (in São Paulo, P = .02; in Lima, P < .001), with differences across groups at 3 months decreasing by 6 months. In subgroup analyses, no evidence of interactions between treatment groups and baseline severity of depressive symptoms (São Paulo, P = .57; Lima, P = .45) or educational level (São Paulo, P = .59; Lima, P = .16) on the primary outcome were found at 3 months (eTable 4 in Supplement 2).

Adverse Events

In the intervention group, worsening of depressive symptoms occurred in 10% of patients and worsening of suicide ideation occurred in 6% of patients vs worsening by 12% and 7% in the control group (Table 4). Three deaths occurred in the intervention group and 8 deaths occurred in the control group; all were judged to be unrelated to study participation (Table 4).

Discussion

In 2 RCTs conducted in São Paulo, Brazil and Lima, Peru, patients with hypertension or diabetes and depressive symptoms who received the digital intervention were significantly more likely to have a 50% or greater reduction of depressive symptoms than those in the enhanced usual care group 3 months after completing the intervention, but differences between groups were no longer statistically significant after 6 months in either of the 2 trials.

One possible reason for the fading of these clinical benefits is that the app was only available during the first 6 weeks of the trial. While the digital intervention group maintained improvement at 6 months, the enhanced usual care group showed further improvement over that time. This may reflect the natural progression of a depressive episode but may also be the result of additional assessments and triggered referrals.30,31

Adherence, which has been a major challenge with digital health interventions,32 was considerably higher than rates seen in trials of other digital interventions in low- and middle-income countries,33 with 65% in São Paulo and 92% in Lima completing more than half of the sessions. The overall good adherence may be the result of user-centered methods employed in the design, resulting in an app that was easy to use and fit into the fabric of people’s lives. The adherence differences between sites may be partly explained by the fact that dedicated study nurses supported the intervention in Lima, while in São Paulo, the intervention was supported by nurse assistants who were employed by existing services and who had many other competing duties. Dedicated nurses might have paid more attention to contacting patients and providing more assistance and motivation when app use decreased. It is also possible that the higher educational level of Peruvian participants, compared with those from Brazil, might have played a role in adherence to the intervention.

The samples in the 2 countries were different with better-educated participants reporting milder depressive symptoms in Lima. However, while acknowledging the low power of these analyses, no interactions with baseline severity or educational levels were found in either of the countries. This suggests that benefits can be accrued even among those with lower levels of education. Two-thirds of the sample were adults older than 40 years of age, suggesting that digital mental health interventions may also be appropriate across a wide age range.

Studies involving digital mental health interventions, especially apps, are increasing,8,9,33-35 but methodologically rigorous clinical trials from low- and middle-income countries are scarce. The results of this study provide further evidence to support the use of digital interventions within the health care sector in some low- and middle-income countries. An 18.6% increase in response rates, compared with the control condition as seen in Lima, would result in a sizable reduction in depressive symptoms if this digital intervention were implemented at a larger scale. Even in Brazil, with a smaller increase of 12.1% in response rates, this brief and low-intensity digital intervention might have a clear public health effect in a country with a notable treatment gap for depression. Studies in the United Kingdom have found that differences for the PHQ-9 around a threshold between 17% and 20% are probably worth pursuing.28 However, caution must be observed when applying minimal clinically important difference values arising from studies in different contexts and with diverse populations, instruments, and interventions. In addition, effective remote digital interventions are sorely needed with the current COVID-19 epidemic.10

Limitations

This study has several limitations. First, enhanced usual care included a safety net for high-risk participants for ethical reasons, which likely improved outcomes in the control group, diluted differences across groups, and potentially rendered more conservative results. Second, these findings should not be generalized to fully automated deployment, as this digital intervention used nurses to support patient app use. However, the nurse support goes in line with task-shifting efforts to minimize the reliance on mental health specialists. Third, this study cannot distinguish the relative contribution of the main components (ie, the app and nurses) to improve adherence or outcomes.

Conclusions

In 2 RCTs of patients with hypertension or diabetes and depressive symptoms in Brazil and Peru, a digital intervention delivered over a 6-week period significantly improved depressive symptoms at 3 months when compared with enhanced usual care. However, the magnitude of the effect was small in the trial from Brazil and the effects were not sustained at 6 months.

Back to top
Article Information

Corresponding Authors: Ricardo Araya, MD, PhD, Centre for Global Mental Health, King’s College London, Five De Crespigny Park, London SE5 8AF, United Kingdom (ricardo.araya@kcl.ac.uk); Jaime Miranda, MD, PhD, Centro de Excelencia en Enfermedades Crónicas, Universidad Peruana Cayetano Heredia, Av Armendáriz 445, Miraflores, Lima 18, Perú (jaime.miranda@upch.pe).

Accepted for Publication: March 8, 2021.

Author Contributions: Dr Araya had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Joint first authors: Drs Araya and Menezes. Joint senior authors: Drs Mohr and Miranda.

Concept and design: Araya, Menezes, Garcia Claro, Diez-Canseco, Peters, Martins Castro, Begale, Mohr, Miranda.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Araya, Menezes, Garcia Claro, Daley, Quayle, Diez-Canseco, Santos, Rocha, Scotton, Fernandes, Begale, Mohr, Miranda.

Critical revision of the manuscript for important intellectual content: Araya, Menezes, Garcia Claro, Brandt, Diez-Canseco, Peters, Toyama, Aschar, Hidalgo-Padilla, Martins Castro, Cavero, Fernandes, Mohr, Miranda.

Statistical analysis: Araya, Menezes, Garcia Claro, Peters, Scotton, Fernandes.

Obtained funding: Araya, Menezes, Mohr, Miranda.

Administrative, technical, or material support: Araya, Menezes, Garcia Claro, Brandt, Quayle, Diez-Canseco, Santos, Aschar, Hidalgo-Padilla, Martins Castro, Cavero, Rocha, Scotton, Begale, Mohr, Miranda.

Supervision: Araya, Menezes, Garcia Claro, Brandt, Quayle, Diez-Canseco, Cavero, Miranda.

Other - quantitative aspects of the trial design: Peters.

Conflict of Interest Disclosures: Dr Araya reported a pending patent for CONEMO (the digital intervention used in both studies). Dr Menezes reported receipt of grants from NIMH (U19MH098780) during the conduct of the study. Dr Brandt reported receipt of grants from NIMH (U19MH098780) during the conduct of the study. Dr Diez-Canseco reported receipt of grants from NIMH (U19MH098780) during the conduct of the study. Dr Hidalgo-Padilla reported receipt of grants from NIMH (U19MH098780) during the conduct of the study. Dr Cavero reported receipt of grants from NIMH (U19MH098780) during the conduct of the study. Dr Rocha reported receipt of grants from NIMH during the conduct of the study. Dr Scotton reported receipt of grants from NIMH during the conduct of the study. Dr Mohr reported receipt of grants from NIMH during the conduct of the study; personal fees from Otsuka Pharmaceuticals, from Apple Inc, Pear Therapeutics, and One Mind Foundation; and other from Adaptive Health Inc Equity Interest outside the submitted work. Dr Miranda reported receipt of grants from NIMH during the conduct of the study. No other disclosures were reported.

Funding/Support: This project was funded by NIMH (U19MH098780).

Role of the Funder/Sponsor: NIMH researchers composed the data and safety monitoring board (DSMB) and executive committee and took part in reviewing and approving research protocols and DSMB reports. The funder did not take part in the data collection or analysis. Funders did not participate in the preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 3.

Additional Contributions: We thank all of the staff, especially the nurses, and the study participants.

References
1.
World Health Organization.  Depression and Other Common Mental Disorders: Global Health Estimates. World Health Organization; 2017.
2.
Pan American Health Organization.  The Burden of Mental Disorders in the Region of the Americas, 2018. Pan American Health Organization; 2018.
3.
Gold  SM, Köhler-Forsberg  O, Moss-Morris  R,  et al.  Comorbid depression in medical diseases.   Nat Rev Dis Primers. 2020;6(1):69. doi:10.1038/s41572-020-0200-2PubMedGoogle ScholarCrossref
4.
Macinko  J, Andrade  FCD, Nunes  BP, Guanais  FC.  Primary care and multimorbidity in six Latin American and Caribbean countries.   Rev Panam Salud Publica. 2019;43:e8-e8. doi:10.26633/RPSP.2019.8PubMedGoogle ScholarCrossref
5.
Chibanda  D, Weiss  HA, Verhey  R,  et al.  Effect of a primary care-based psychological intervention on symptoms of common mental disorders in Zimbabwe: a randomized clinical trial.   JAMA. 2016;316(24):2618-2626. doi:10.1001/jama.2016.19102PubMedGoogle ScholarCrossref
6.
Patel  V, Weobong  B, Weiss  HA,  et al.  The Healthy Activity Program (HAP), a lay counsellor-delivered brief psychological treatment for severe depression, in primary care in India: a randomised controlled trial.   Lancet. 2017;389(10065):176-185. doi:10.1016/S0140-6736(16)31589-6PubMedGoogle ScholarCrossref
7.
Araya  R, Rojas  G, Fritsch  R,  et al.  Treating depression in primary care in low-income women in Santiago, Chile: a randomised controlled trial.   Lancet. 2003;361(9362):995-1000. doi:10.1016/S0140-6736(03)12825-5PubMedGoogle ScholarCrossref
8.
Torous  J, Cerrato  P, Halamka  J.  Targeting depressive symptoms with technology.   Mhealth. 2019;5:19. doi:10.21037/mhealth.2019.06.04PubMedGoogle ScholarCrossref
9.
Firth  J, Torous  J, Nicholas  J,  et al.  The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials.   World Psychiatry. 2017;16(3):287-298. doi:10.1002/wps.20472PubMedGoogle ScholarCrossref
10.
Kuehn  BM.  Global mental health services are collapsing as demand grows.   JAMA. 2020;324(19):1933-1933. doi:10.1001/jama.2020.22085PubMedGoogle Scholar
11.
Shariful Islam  SM, Chow  CK, Redfern  J,  et al.  Effect of text messaging on depression in patients with coronary heart disease: a substudy analysis from the TEXT ME randomised controlled trial.   BMJ Open. 2019;9(2):e022637. doi:10.1136/bmjopen-2018-022637PubMedGoogle Scholar
12.
Prabhakaran  D, Jha  D, Prieto-Merino  D,  et al.  Effectiveness of an mHealth-based electronic decision support system for integrated management of chronic conditions in primary care: the mWellcare cluster-randomized controlled trial.   Circulation. 2019;139(3):380-391. doi:10.1161/CIRCULATIONAHA.118.038192PubMedGoogle ScholarCrossref
13.
Mohr  DC, Lyon  AR, Lattie  EG, Reddy  M, Schueller  SM.  Accelerating digital mental health research from early design and creation to successful implementation and sustainment.   J Med Internet Res. 2017;19(5):e153. doi:10.2196/jmir.7725PubMedGoogle Scholar
14.
Brandt  LR, Hidalgo  L, Diez-Canseco  F,  et al.  Addressing depression comorbid with diabetes or hypertension in resource-poor settings: a qualitative study about user perception of a nurse-supported smartphone app in Peru.   JMIR Ment Health. 2019;6(6):e11701. doi:10.2196/11701PubMedGoogle Scholar
15.
Menezes  P, Quayle  J, Garcia Claro  H,  et al.  Use of a mobile phone app to treat depression comorbid with hypertension or diabetes: a pilot study in Brazil and Peru.   JMIR Ment Health. 2019;6(4):e11698. doi:10.2196/11698PubMedGoogle Scholar
16.
Kroenke  K, Spitzer  RL, Williams  JB.  The PHQ-9: validity of a brief depression severity measure.   J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.xPubMedGoogle ScholarCrossref
17.
Sealed Envelope.  Randomisation and online databases for clinical trials. Access verified April 22, 2021. https://www.sealedenvelope.com/
18.
Ekers  D, Webster  L, Van Straten  A, Cuijpers  P, Richards  D, Gilbody  S.  Behavioural activation for depression; an update of meta-analysis of effectiveness and sub group analysis.   PLoS One. 2014;9(6):e100100. doi:10.1371/journal.pone.0100100PubMedGoogle Scholar
19.
Mohr  DC, Cuijpers  P, Lehman  K.  Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions.   J Med Internet Res. 2011;13(1):e30. doi:10.2196/jmir.1602PubMedGoogle Scholar
20.
Coley  RY, Boggs  JM, Beck  A, Hartzler  AL, Simon  GE.  Defining success in measurement-based care for depression: a comparison of common metrics.   Psychiatr Serv. 2020;71(4):312-318. doi:10.1176/appi.ps.201900295PubMedGoogle ScholarCrossref
21.
Cuijpers  P, Stringaris  A, Wolpert  M.  Treatment outcomes for depression: challenges and opportunities.   Lancet Psychiatry. 2020;7(11):925-927. doi:10.1016/S2215-0366(20)30036-5PubMedGoogle ScholarCrossref
22.
McMillan  D, Gilbody  S, Richards  D.  Defining successful treatment outcome in depression using the PHQ-9: a comparison of methods.   J Affect Disord. 2010;127(1-3):122-129. doi:10.1016/j.jad.2010.04.030PubMedGoogle ScholarCrossref
23.
Rabin  R, de Charro  F.  EQ-5D: a measure of health status from the EuroQol Group.   Ann Med. 2001;33(5):337-343. doi:10.3109/07853890109002087PubMedGoogle ScholarCrossref
24.
World Health Organization.  World Health Organization Disability Assessment Schedule: WHODAS II. World Health Organization; 2000.
25.
Rojas  G, Fritsch  R, Solis  J,  et al.  Treatment of postnatal depression in low-income mothers in primary-care clinics in Santiago, Chile: a randomised controlled trial.   Lancet. 2007;370(9599):1629-1637. doi:10.1016/S0140-6736(07)61685-7PubMedGoogle ScholarCrossref
26.
Patel  V, Weiss  HA, Chowdhary  N,  et al.  Effectiveness of an intervention led by lay health counsellors for depressive and anxiety disorders in primary care in Goa, India (MANAS): a cluster randomised controlled trial.   Lancet. 2010;376(9758):2086-2095. doi:10.1016/S0140-6736(10)61508-5PubMedGoogle ScholarCrossref
27.
Singla  DR, Waqas  A, Hamdani  SU,  et al.  Implementation and effectiveness of adolescent life skills programs on low- and middle-income countries: a critical review and meta-analysis.   Behav Res Ther. 2020;130:103402. doi:10.1016/j.brat.2019.04.010PubMedGoogle Scholar
28.
Kounali  D, Button  KS, Lewis  G,  et al.  How much change is enough? evidence from a longitudinal study on depression in UK primary care.   Psychol Med. 2020;1-8. doi:10.1017/S0033291720003700PubMedGoogle Scholar
29.
White  IR, Royston  P, Wood  AM.  Multiple imputation using chained equations: issues and guidance for practice.   Stat Med. 2011;30(4):377-399. doi:10.1002/sim.4067PubMedGoogle ScholarCrossref
30.
Sawyer  A, Kaim  A, Le  H-N,  et al.  The effectiveness of an app-based nurse-moderated program for new mothers with depression and parenting problems (eMums Plus): pragmatic randomized controlled trial.   J Med Internet Res. 2019;21(6):e13689. doi:10.2196/13689PubMedGoogle Scholar
31.
Arjadi  R, Nauta  MH, Scholte  WF,  et al.  Internet-based behavioural activation with lay counsellor support versus online minimal psychoeducation without support for treatment of depression: a randomised controlled trial in Indonesia.   Lancet Psychiatry. 2018;5(9):707-716. doi:10.1016/S2215-0366(18)30223-2PubMedGoogle ScholarCrossref
32.
Graham  AK, Lattie  EG, Mohr  DC.  Experimental therapeutics for digital mental health.   JAMA Psychiatry. 2019;76(12):1223-1224. doi:10.1001/jamapsychiatry.2019.2075PubMedGoogle ScholarCrossref
33.
Naslund  JA, Aschbrenner  KA, Araya  R,  et al.  Digital technology for treating and preventing mental disorders in low-income and middle-income countries: a narrative review of the literature.   Lancet Psychiatry. 2017;4(6):486-500. doi:10.1016/S2215-0366(17)30096-2PubMedGoogle ScholarCrossref
34.
Jiménez-Molina  Á, Franco  P, Martínez  V, Martínez  P, Rojas  G, Araya  R.  Internet-based interventions for the prevention and treatment of mental disorders in Latin America: a scoping review.   Front Psychiatry. 2019;10:664. doi:10.3389/fpsyt.2019.00664PubMedGoogle ScholarCrossref
35.
Carter  H, Araya  R, Anjur  K, Deng  D, Naslund  JA.  The emergence of digital mental health in low-income and middle-income countries: a review of recent advances and implications for the treatment and prevention of mental disorders.   J Psychiatr Res. 2021;133:223-246. doi:10.1016/j.jpsychires.2020.12.016PubMedGoogle ScholarCrossref
×