[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.
Flow of Participants
Flow of Participants

AQ indicates Acrophobia Questionnaire; PHQ, Patient Health Questionnaire.

Figure 2.
Heterogeneous Treatment Effects (n = 193) With Respect to Initial Acrophobia Score and Self-reported Experience Mechanisms (n = 52-55)
Heterogeneous Treatment Effects (n = 193) With Respect to Initial Acrophobia Score and Self-reported Experience Mechanisms (n = 52-55)
Table 1.  
Demographic and Baseline Characteristics
Demographic and Baseline Characteristics
Table 2.  
Inferential Statistics of Treatment Outcome Measures, Intention to Treat (n = 193)
Inferential Statistics of Treatment Outcome Measures, Intention to Treat (n = 193)
Table 3.  
Virtual Reality Cognitive Behavioral Therapy App Treatment Effect Estimates
Virtual Reality Cognitive Behavioral Therapy App Treatment Effect Estimates
1.
Kessler  RC, Berglund  PA, Bruce  ML,  et al.  The prevalence and correlates of untreated serious mental illness.  Health Serv Res. 2001;36(6 pt 1):987-1007.PubMedGoogle Scholar
2.
Fairburn  CG, Patel  V.  The impact of digital technology on psychological treatments and their dissemination.  Behav Res Ther. 2017;88:19-25. doi:10.1016/j.brat.2016.08.012PubMedGoogle ScholarCrossref
3.
de Graaf  R, ten Have  M, van Gool  C, van Dorsselaer  S.  Prevalence of mental disorders and trends from 1996 to 2009: results from the Netherlands Mental Health Survey and Incidence Study-2.  Soc Psychiatry Psychiatr Epidemiol. 2012;47(2):203-213. doi:10.1007/s00127-010-0334-8PubMedGoogle ScholarCrossref
4.
Kessler  RC, Petukhova  M, Sampson  NA, Zaslavsky  AM, Wittchen  HU.  Twelve-month and lifetime prevalence and lifetime morbid risk of anxiety and mood disorders in the United States.  Int J Methods Psychiatr Res. 2012;21(3):169-184. doi:10.1002/mpr.1359PubMedGoogle ScholarCrossref
5.
Depla  MF, ten Have  ML, van Balkom  AJ, de Graaf  R.  Specific fears and phobias in the general population: results from the Netherlands Mental Health Survey and Incidence Study (NEMESIS).  Soc Psychiatry Psychiatr Epidemiol. 2008;43(3):200-208. doi:10.1007/s00127-007-0291-zPubMedGoogle ScholarCrossref
6.
Stinson  FS, Dawson  DA, Patricia Chou  S,  et al.  The epidemiology of DSM-IV specific phobia in the USA: results from the National Epidemiologic Survey on Alcohol and Related Conditions.  Psychol Med. 2007;37(7):1047-1059. doi:10.1017/S0033291707000086PubMedGoogle ScholarCrossref
7.
Trumpf  J, Margraf  J, Vriends  N, Meyer  AH, Becker  ES.  Specific phobia predicts psychopathology in young women.  Soc Psychiatry Psychiatr Epidemiol. 2010;45(12):1161-1166. doi:10.1007/s00127-009-0159-5PubMedGoogle ScholarCrossref
8.
Choy  Y, Fyer  AJ, Goodwin  RD.  Specific phobia and comorbid depression: a closer look at the National Comorbidity Survey data.  Compr Psychiatry. 2007;48(2):132-136. doi:10.1016/j.comppsych.2006.10.010Google ScholarCrossref
9.
Abramowitz  JS.  The practice of exposure therapy: relevance of cognitive-behavioral theory and extinction theory.  Behav Ther. 2013;44(4):548-558. doi:10.1016/j.beth.2013.03.003PubMedGoogle ScholarCrossref
10.
Slater  M, Sanchez-Vives  MV.  Enhancing our lives with immersive virtual reality.  Frontiers Robotics AI.2016;3:76. doi:10.3389/frobt.2016.00074Google ScholarCrossref
11.
Fodor  LA, Coteț  CD, Cuijpers  P, Szamoskozi  Ș, David  D, Cristea  IA.  The effectiveness of virtual reality based interventions for symptoms of anxiety and depression: a meta-analysis.  Sci Rep. 2018;8(1):10323. doi:10.1038/s41598-018-28113-6PubMedGoogle ScholarCrossref
12.
Opriş  D, Pintea  S, García-Palacios  A, Botella  C, Szamosközi  Ş, David  D.  Virtual reality exposure therapy in anxiety disorders: a quantitative meta-analysis.  Depress Anxiety. 2012;29(2):85-93. doi:10.1002/da.20910PubMedGoogle ScholarCrossref
13.
Morina  N, Ijntema  H, Meyerbröker  K, Emmelkamp  PM.  Can virtual reality exposure therapy gains be generalized to real-life? a meta-analysis of studies applying behavioral assessments.  Behav Res Ther. 2015;74:18-24. doi:10.1016/j.brat.2015.08.010PubMedGoogle ScholarCrossref
14.
Carl  E, Stein  AT, Levihn-Coon  A,  et al Virtual reality exposure therapy for anxiety and related disorders: a meta-analysis of randomized controlled trials.  J Anxiety Disord. 2019;61:27-36. doi:10.1016/j.janxdis.2018.08.003Google ScholarCrossref
15.
Parsons  TD, Rizzo  AA.  Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis.  J Behav Ther Exp Psychiatry. 2008;39(3):250-261. doi:10.1016/j.jbtep.2007.07.007PubMedGoogle ScholarCrossref
16.
Benbow  AA, Anderson  PL.  A meta-analytic examination of attrition in virtual reality exposure therapy for anxiety disorders.  J Anxiety Disord. 2019;61:18-26. doi:10.1016/j.janxdis.2018.06.006Google ScholarCrossref
17.
Fernández-Álvarez  J, Rozental  A, Carlbring  P,  et al Deterioration rates in virtual reality therapy: an individual patient data level meta-analysis.  J Anxiety Disord. 2019;61:3-17. doi:10.1016/j.janxdis.2018.06.005Google ScholarCrossref
18.
Lindner  P, Miloff  A, Hamilton  W,  et al.  Creating state of the art, next-generation virtual reality exposure therapies for anxiety disorders using consumer hardware platforms: design considerations and future directions.  Cogn Behav Ther. 2017;46(5):404-420. doi:10.1080/16506073.2017.1280843PubMedGoogle ScholarCrossref
19.
Freeman  D, , Haselton  P, , Freeman  J, ,  et al Automated psychological therapy using immersive virtual reality for treatment of fear of heights: a single-blind, parallel-group, randomised controlled trial.  Lancet Psychiatry. 2018;5(8):625-632. doi:10.1016/S2215-0366(18)30226-8-8Google ScholarCrossref
20.
Donker  T, Van Esveld  S, Fischer  N, Van Straten  A.  0Phobia: towards a virtual cure for acrophobia: study protocol for a randomized controlled trial.  Trials. 2018;19(1):433. doi:10.1186/s13063-018-2704-6PubMedGoogle ScholarCrossref
21.
Cohen  DC.  Comparison of self-report and overt-behavioral procedures for assessing acrophobia.  Behav Ther. 1977;8(1):17-23. doi:10.1016/S0005-7894(77)80116-0Google ScholarCrossref
22.
Steinman  SA, Teachman  BA.  Cognitive processing and acrophobia: validating the Heights Interpretation Questionnaire.  J Anxiety Disord. 2011;25(7):896-902 .doi:10.1016/j.janxdis.2011.05.001Google ScholarCrossref
23.
Kroenke  K, Spitzer  RL, Williams  JB.  The PHQ-9: validity of a brief depression severity measure.  J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.xPubMedGoogle ScholarCrossref
24.
Donker  T, van Straten  A, Marks  I, Cuijpers  P.  A brief Web-based screening questionnaire for common mental disorders: development and validation.  J Med Internet Res. 2009;11(3):e19. doi:10.2196/jmir.1134PubMedGoogle ScholarCrossref
25.
Saghaei  M.  Random allocation software for parallel group randomized trials.  BMC Med Res Methodol. 2004;4:26. doi:10.1186/1471-2288-4-26PubMedGoogle ScholarCrossref
26.
Cuijpers  P, Huibers  M, Ebert  DD, Koole  SL, Andersson  G.  How much psychotherapy is needed to treat depression? a metaregression analysis.  J Affect Disord. 2013;149(1-3):1-13. doi:10.1016/j.jad.2013.02.030PubMedGoogle ScholarCrossref
27.
Emmelkamp  PM, Krijn  M, Hulsbosch  AM, de Vries  S, Schuemie  MJ, van der Mast  CA.  Virtual reality treatment versus exposure in vivo: a comparative evaluation in acrophobia.  Behav Res Ther. 2002;40(5):509-516. doi:10.1016/S0005-7967(01)00023-7PubMedGoogle ScholarCrossref
28.
Abelson  JL, Curtis  GC.  Cardiac and neuroendocrine responses to exposure therapy in height phobics: desynchrony within the ‘physiological response system’.  Behav Res Ther. 1989;27(5):561-567. doi:10.1016/0005-7967(89)90091-0PubMedGoogle ScholarCrossref
29.
Beck  AT, Epstein  N, Brown  G, Steer  RA.  An inventory for measuring clinical anxiety: psychometric properties.  J Consult Clin Psychol. 1988;56(6):893-897. doi:10.1037/0022-006X.56.6.893PubMedGoogle ScholarCrossref
30.
Bangor  A.  An empirical evaluation of the system usability scale.  Int J Hum Comput Interact. 2008;24(6):574-594. doi:10.1080/10447310802205776Google ScholarCrossref
31.
Schubert  T, Friedmann  F, Regenbrecht  H.  The experience of presence: factor analytic insights.  Presence Teleop Virt.2001;10(3):266-281. doi:10.1162/105474601300343603Google ScholarCrossref
32.
Pearlin  LI, Schooler  C.  The structure of coping.  J Health Soc Behav. 1978;19(1):2-21. doi:10.2307/2136319PubMedGoogle ScholarCrossref
33.
LaViola  JJ  Jr.  A discussion of cybersickness in virtual environments.  ACM SIGCHI Bullet. 2000;32(1):47-56. doi:10.1145/333329.333344Google ScholarCrossref
34.
Kennedy  RS, Lane  NE, Berbaum  KS, Lilienthal  MG.  Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness.  Int J Aviat Psychol. 1993;3(3):203-220. doi:10.1207/s15327108ijap0303_3Google ScholarCrossref
35.
Survalyzer Nederland  BV. Utrecht, the Netherlands. http://www.survalyzer.com. Accessed Augst 27, 2018.
36.
Donker  T, Bennett  K, Bennett  A,  et al.  Internet-delivered interpersonal psychotherapy versus internet-delivered cognitive behavioral therapy for adults with depressive symptoms: randomized controlled noninferiority trial.  J Med Internet Res. 2013;15(5):e82. doi:10.2196/jmir.2307.Google ScholarCrossref
37.
Issakidis  C, Andrews  G.  Pretreatment attrition and dropout in an outpatient clinic for anxiety disorders.  Acta Psychiatr Scand. 2004;109(6):426-433. doi:10.1111/j.1600-0047.2004.00264.xPubMedGoogle ScholarCrossref
38.
Lee  DS.  Training, wages, and sample selection: estimating sharp bounds on treatment effects.  Rev Econ Stud. 2009;76(3):1071-1102. doi:10.1111/j.1467-937X.2009.00536.xGoogle ScholarCrossref
39.
Karyotaki  E, Riper  H, Twisk  J,  et al.  Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data.  JAMA Psychiatry. 2017;74(4):351-359. doi:10.1001/jamapsychiatry.2017.0044PubMedGoogle ScholarCrossref
40.
Reliable and clinically significant change. Mounted by Chris Evans, 1998. http://www.psyctc.org/stats/rcsc.htm. Accessed August 18, 2018.
41.
Cohen  DC. Personality predictors and the outcome of varieties of desensitization. Unpublished doctoral dissertation, Harvard University, Cambridge, MA; 1972.
42.
Jacobson  NS, Truax  P.  Clinical significance: a statistical approach to defining meaningful change in psychotherapy research.  J Consult Clin Psychol. 1991;59(1):12-19. doi:10.1037/0022-006X.59.1.12PubMedGoogle ScholarCrossref
43.
Christensen  L, Mendoza  JL.  A method of assessing change in a single subject: an alteration of the RC index.  Behav Ther. 1986;17(3):305-308. doi:10.1016/S0005-7894(86)80060-0Google ScholarCrossref
44.
Furukawa  TA, Leucht  S.  How to obtain NNT from Cohen’s d: comparison of two methods.  PLoS One. 2011;6(4):e19070. doi:10.1371/journal.pone.0019070PubMedGoogle ScholarCrossref
45.
Pedersen  AB, Mikkelsen  EM, Cronin-Fenton  D,  et al.  Missing data and multiple imputation in clinical epidemiological research.  Clin Epidemiol. 2017;9:157-166. doi:10.2147/CLEP.S129785PubMedGoogle ScholarCrossref
46.
Rozental  A, Castonguay  L, Dimidjian  S,  et al.  Negative effects in psychotherapy: commentary and recommendations for future research and clinical practice.  BJPsych Open. 2018;4(4):307-312. doi:10.1192/bjo.2018.42PubMedGoogle ScholarCrossref
47.
Arroll  B, Wallace  HB, Mount  V, Humm  SP, Kingsford  DW.  A systematic review and meta-analysis of treatments for acrophobia.  Med J Aust. 2017;206(6):263-267. doi:10.5694/mja16.00540PubMedGoogle ScholarCrossref
48.
Wolitzky-Taylor  KB, Horowitz  JD, Powers  MB, Telch  MJ.  Psychological approaches in the treatment of specific phobias: a meta-analysis.  Clin Psychol Rev. 2008;28(6):1021-1037. doi:10.1016/j.cpr.2008.02.007PubMedGoogle ScholarCrossref
49.
Krijn  M, Emmelkamp  PMG, Biemond  R, de Wilde de Ligny  C, Schuemie  MJ, van der Mast  CA.  Treatment of acrophobia in virtual reality: the role of immersion and presence.  Behav Res Ther. 2004;42(2):229-239. doi:10.1016/S0005-7967(03)00139-6Google ScholarCrossref
50.
Andersson  G, Titov  N.  Advantages and limitations of Internet-based interventions for common mental disorders.  World Psychiatry. 2014;13(1):4-11. doi:10.1002/wps.20083PubMedGoogle ScholarCrossref
51.
Buntrock  C, Ebert  DD, Lehr  D,  et al.  Effect of a Web-based guided self-help intervention for prevention of major depression in adults with subthreshold depression: a randomized clinical trial.  JAMA. 2016;315(17):1854-1863. doi:10.1001/jama.2016.4326PubMedGoogle ScholarCrossref
52.
Statcounter GlobalStats. Mobile operating system market share Netherlands. Oct 2017- Oct 2018. http://gs.statcounter.com/os-market-share/mobile/netherlands. Accessed November 29, 2018.
Views 1,204
Citations 0
Original Investigation
March 20, 2019

Effectiveness of Self-guided App-Based Virtual Reality Cognitive Behavior Therapy for Acrophobia: A Randomized Clinical Trial

Author Affiliations
  • 1Department of Clinical, Neuro- and Developmental Psychology, Section Clinical Psychology, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands
  • 2Amsterdam Public Health Research Institute Amsterdam, Amsterdam, the Netherlands
  • 3Department of Education Sciences, Section Methods and Statistics, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands
  • 4Amsterdam Center for Learning Analytics, the Netherlands
  • 5Department of Psychology, University Stockholm, Stockholm, Sweden
  • 6Department of Psychology, University of Southern Denmark, Odense, Denmark
  • 7Department of Psychology of Conflict, Risk and Safety, University of Twente, Enschede, Amsterdam, the Netherlands
JAMA Psychiatry. Published online March 20, 2019. doi:10.1001/jamapsychiatry.2019.0219
Key Points

Question  Is fully self-guided app-based virtual reality cognitive behavior therapy using low-cost (cardboard) virtual reality goggles user friendly and effective in reducing acrophobia symptoms compared with a wait-list control group?

Findings  In a single-blind randomized clinical trial that included 193 participants with acrophobia symptoms, app-based therapy demonstrated a large and significant reduction in acrophobia symptoms compared with wait-list controls and was rated as user friendly.

Meaning  Acrophobia cognitive behavioral therapy can be effectively delivered without therapist intervention through standard smartphones and low-cost virtual reality goggles at a fraction of the cost of current face-to-face treatment or high-end virtual reality exposure therapy.

Abstract

Importance  Globally, access to evidence-based psychological treatment is limited. Innovative self-help methods using smartphone applications and low-cost virtual reality have the potential to significantly improve the accessibility and scalability of psychological treatments.

Objective  To examine the effectiveness of ZeroPhobia, a fully self-guided app-based virtual reality cognitive behavior therapy (VR CBT) using low-cost (cardboard) virtual reality goggles compared with a wait-list control group and to determine its user friendliness.

Design, Setting, and Participants  In a single-blind randomized clinical trial, participants were enrolled between March 24 and September 28, 2017, and randomly assigned (1:1) by an independent researcher to either VR CBT app or a wait-list control group. A total of 193 individuals aged 18 to 65 years from the Dutch general population with acrophobia symptoms and access to an Android smartphone participated. The 6 animated modules of the VR-CBT app and gamified virtual reality environments were delivered over a 3-week period in participants’ natural environment. Assessments were completed at baseline, immediately after treatment, and at 3-month follow-up. Analysis began April 6, 2018, and was intention to treat.

Intervention  Self-guided app-based VR CBT.

Main Outcomes and Measures  The primary outcome measure was the Acrophobia Questionnaire. The hypothesis was formulated prior to data collection.

Results  In total, 193 participants (129 women [66.84%]; mean [SD] age, 41.33 [13.64] years) were randomly assigned to intervention (n = 96) or a wait-list control group (n = 97). An intent-to-treat analysis showed a significant reduction of acrophobia symptoms at posttest at 3 months for the VR-CBT app compared with the controls (b = –26.73 [95% CI, −32.12 to −21.34]; P < .001; d = 1.14 [95% CI, 0.84 to 1.44]). The number needed to treat was 1.7. Sensitivity and robustness analysis confirmed these findings. Pretreatment attrition was 22 of 96 (23%) because of smartphone incompatibility. Of the 74 participants who started using the VR-CBT app, 57 (77%) completed the intervention fully.

Conclusions and Relevance  A low-cost fully self-guided app-based virtual reality cognitive behavioral therapy with rudimentary virtual reality goggles can produce large acrophobia symptom reductions. To our knowledge, this study is the first to show that virtual reality acrophobia treatment can be done at home without the intervention of a therapist.

Trial Registration  Trialregister.nl identifier: NTR6442

Introduction

Access to evidence-based psychological treatment for mental health disorders is a global challenge because of high-treatment costs1 and the limited availability of mental health professionals.2 Treatment coverage is below 50% and usually substantially lower.2 Novel technologies may contribute to accessible and affordable treatment options in important ways.

Specific phobias rank among the most prevalent mental health disorders,3,4 of which acrophobia is the most prevalent of all subtypes.5,6 Worryingly, specific phobias may increase the risk of developing other anxiety disorders7 and major depression.8

Several evidence-based therapies exist, most of which use cognitive behavior therapy (CBT), involving exposing individuals to their feared object or situation.9 An emergent form of exposure is virtual reality exposure therapy (VRET), in which immersive virtual environments replace real-life exposure settings.10 Meta-analyses indicate VRET is as effective as conventional exposure therapy11-13 with large effect sizes11-15 and similar dropout and deterioration rates.11,12,16,17 However, VRET has thus far required therapist involvement and uses high-end virtual reality (VR) equipment.18

Recently, efforts toward reducing therapist involvement using VRET have been undertaken. A randomized clinical trial19 targeting acrophobia demonstrated a large effect size (d = 2.0) but still required expensive technology, and treatment was delivered in a clinic under the supervision of a therapist. Hence, the relative effect of the intervention itself remains unknown and costs are not reduced. A literature search (eAppendix 1 in Supplement 2) yielded no results on previous studies using a mobile app for acrophobia treatment (except for the present study’s protocol20).

The present study tested the effectiveness and user friendliness of ZeroPhobia, a fully self-guided VR CBT for acrophobia symptoms that is delivered through a smartphone. To ensure scalability, the VR-CBT app relies on participants’ own smartphone and basic ($10) cardboard VR goggles, while the program can be followed at home. We hypothesized that the app would be associated with greater overall response at posttest compared with a wait-list control group and that the treatment gains would be maintained at 3-month follow-up. For robustness, we included a second questionnaire assessing acrophobia symptoms. Depressive symptoms were examined to assess whether the VR-CBT app could also influence depression levels. We also tested whether variation in perceived user friendliness, general anxiety, and cyber sickness when using the app affected acrophobia symptoms at posttest.

Methods
Study Design and Procedure

In this single-blind randomized clinical trial, participants were recruited from the Dutch general population through websites, magazines, and local media. Ethical approval was received from the Medical Ethics Committee of the Vrije Universiteit Medical Center (the trial protocol has been previously published20 and is available in Supplement 1). Participants provided written informed consent by email or mail.

Participants

To be considered for inclusion, individuals had to score at least 45.45 on the Acrophobia Questionnaire (AQ)–Anxiety (1 SD below the mean of a previous acrophobic sample),21,22 have access to an Android smartphone (Android version 5.1 Lollipop or higher, 4.7- to 5.5-inch screen, and gyroscope), be aged 18 to 65 years, and have provided informed consent. Exclusion criteria were insufficient Dutch language skills, current phobia treatment or receiving psychotropic medication, or having severe depression (total score >19 on the Patient Health Questionnaire–9 item23) or suicidality (score ≥3 on the Web Screening Questionnaire24). Enrollment commenced March 24 and ceased September 28, 2017. Analysis began in April 2018.

Randomization and Masking

A randomization list was created with Random Allocation Software25 using block randomization of 6, 8, 10, and 12 blocks at an allocation ratio of 1:1. Participants were randomized into 2 groups: intervention or wait-list. The randomization list was kept by an independent researcher who revealed the next randomization outcome after every inclusion, thus ensuring that the research team was blind for treatment allocation. All materials were completed online without researcher intervention.

Intervention: VR-CBT App

Participants received 6 animated CBT-based modules using 2-dimensional animations and a voice-over provided by a virtual therapist. The modules took between 5 and 40 minutes to complete. Participants were asked to complete the entire intervention within 3 weeks. Aside from the psychoeducation and CBT techniques, the VR-CBT app included a gamified immersive VR environment and four 360° videos covering the entire exposure spectrum. The participants started using VR and 360° videos from module 3 onwards and navigated through the virtual environment using gaze control. Because previous research found better effects when the treatment was delivered twice a week compared with once a week,26 the 6 app modules were offered over a period of 3 weeks. However, participants could practice with the VR-CBT app as often as they wanted. During the 3-week period, participants received weekly standardized motivational emails with reminders to start or continue with the app (Donker et al20 includes intervention details; eFigure in Supplement 2 includes VR screen shots). The VR environment was created with the Unity game engine (version 2017.3.0f3; Unity Technologies).

Outcomes

All measures at baseline, posttest, and 3-month follow-up were completed online. The participants were asked for demographic information and at each assessment point, whether they received any other professional acrophobia treatment.

The primary outcome was the AQ.21 In a previous study, the mean (SD) AQ-Anxiety scores in a Dutch population were 59.06 (17.12).27 Secondary outcomes included the Attitudes Toward Heights Questionnaire,28 the Beck Anxiety Inventory,29 the System Usability Scale30 to assess user friendliness, the Igroup Presence Questionnaire31 to assess presence in VR, Mastery32 to assess subjective feelings of control, and the Patient Health Questionnaire–9 item23 to assess depressive symptoms. The use of professional treatment during the VR-CBT app, the 3-item Simulator Sickness Questionnaire (SSQ),33,34 and vision impairment (eg, wearing glasses) were included as control variables. The SSQ is commonly used to assess simulator sickness but not in the context of anxiety treatment; the items completely overlap with anxiety symptoms (eg, nausea, dizziness).19 We used a simple raw score of whether the symptom was present or not. All assessments were programmed with Survalyzer software.35

Study Power

The primary outcome measure, the AQ, was used for the power calculations. In previous meta-analyses in which VRET vs a wait-list control group was compared using the AQ, effect sizes ranged from 0.93 to 1.23.12,14,15 However, the unguided nature of the VR-CBT app, the use of low-end equipment, and the lack of comparable studies led us to opt for a conservative estimate of Cohen d = 0.50. With 80% power at a 5% significance level (2-sided), 64 participants in each condition were required (128 participants in total). Because of the high dropout rates often encountered in self-guided interventions,36 we estimated a 40% dropout rate, meaning 180 participants were required.20

Statistical Analyses

Demographic and clinical characteristics were compared between the intervention and the wait-list control groups, with χ2 or variance analysis as appropriate. To assess whether pretreatment attrition (participants who failed to commence treatment after randomization37) and dropout (participants who discontinued treatment after commencing treatment37) were nonrandom, a balancing table was constructed comparing background characteristics, prescores, and other covariates between participants with and without missing outcome observations. Missing outcome observations were accounted for by using a single best regression-imputed value. Missing outcome values for the pretreatment attrition sample were imputed using the wait-list control sample, whereas for the dropout sample initial treatment assignment was used. The empirical evaluation was performed on an intention-to-treat (ITT) basis, using ordinary least squares regression models with prescores and background characteristics included. Standardized effect sizes (Cohen d) and confidence intervals were calculated for each end point.

Two additional robustness analyses were performed. First, because nonrandom sample attrition may bias the estimated treatment effects, nonparametric treatment effect bounds were estimated.38 Second, multiple imputation using an iterative Markov chain Monte Carlo method based on initial treatment assignment was performed to infer the statistical significance of the reported differences between the intervention and control groups. Furthermore, potential heterogeneity and mechanisms of effective treatment were addressed by estimating how treatment effects vary with prescores, general anxiety, and experiences when using the app. We also analyzed the predicted reductions in the posttest for 10-point bins of prescores using regression analyses to examine whether severity at baseline AQ predicts outcome. Sensitivity analyses using participants who completed posttest or follow-up after the intervention examined whether there was a difference in the results compared with the ITT analysis.39 We exploratively assessed clinically meaningful change on the AQ using the clinically significant change formula (method C40) with an AQ score less than 31.67.41 The reliable change criterion was assessed based on the AQ pretest SD scores with a Cronbach α of .91. This yielded an SE of change of 7.98, with a corresponding reliable change criterion on the AQ score of 15.65 (7.98 × 1.96).42,43 We also assessed the number needed to treat using the formula by Furukawa et al44 based on the effect size of the AQ. Two-sided P value less than .05 indicated statistical significance. Stata version 14.2 (StataCorp) was used for the analyses except for number needed to treat, which was conducted in R version 3.4.3 (R Project for Statistical Computing). A data monitoring committee was not required by the ethics committee because of the expected low safety risk of the participants.

Deviations From the Original Protocol

We deviated on some issues from the ethics protocol (Supplement 1). For details, see eAppendix 2 in Supplement 2.

Results
Sample Characteristics

Of 663 individuals who signed up for participation, 291 were ineligible and excluded from participation. In total, 193 participants filled in the baseline assessment and were randomly assigned to the VR-CBT app (n = 96) or wait-list control (n = 97). The pretreatment attrition rate was 23% in the app condition because of illness (1 [1%]) or an incompatible smartphone (21 [22%]). Several participants with smartphones lacking a gyroscope (required for experiencing VR) were erroneously included. To ensure that this did not jeopardize randomization, these participants were included in the ITT analyses, and Lee bounds38 were estimated to address potential bias. Figure 1 shows the participant flowchart.

Baseline demographics are provided in Table 1. Data were balanced between the groups. None of the participants reported receiving professional acrophobia treatment at baseline, posttest, or follow-up.

Treatment Adherence and Attrition

There were no significant differences in the demographics between participants (n = 193) and those who were ineligible or chose not to participate (n = 346) (data not shown). In the intervention group, 59% (57 of 96) of the participants completed the posttest and 49% (47 of 96) completed follow-up, compared with 91% (87 of 97) posttest responses for the wait-list group. Attrition (22 of 96 [23%]) and dropout (17 of 96 [18%]) were not related to background characteristics, prescores, and other covariates (eTable 1 in Supplement 2), as indicated by the likelihood ratio test for predicting missing outcomes (χ2 (9) = 3.47; P = .94), supporting the assumption that outcomes are missing completely at random.45

Primary Outcome

The intervention condition showed a significant reduction in acrophobia symptoms compared with the control on the AQ in the ITT analysis at posttest (b = −26.73 [95% CI, −32.12 to −21.34]; t191 = −9.79; P < .001; adjusted R2 = 0.52) with an effect size of d = 1.14 (95% CI, 0.84-1.44). The number needed to treat was 1.7 (Table 2 and Table 3).

Secondary Outcomes

The results of the ITT analyses (Table 3) demonstrated a significant intervention effect compared with the controls on acrophobia symptoms, Attention to Height Questionnaire (b = −12.59; t182 = −8.92; P < .001; d = 1.091; 95% CI, 0.787-1.393), general anxiety symptoms, Beck Anxiety Inventory (b = −3.87; t182 = −2.62; P < .001; d = 0.37; 95% CI, 0.087-656), and a greater sense of mastery (b = 1.01; t182 = −2.00; P = .047; d = −0.11; 95% CI, −0.389 to 0.176). Depressive symptoms did not change significantly between both groups, Patient Health Questionnaire–9 item (b = −0.49; t182 = −1.58; P ≥ .99; d = 0.14; 95% CI, −0.143 to 0.422).

Robustness and Sensitivity Analyses

When estimating the Lee bounds,38 the resulting upper and lower bounds (model 2) indicated that the treatment effect on the primary outcome measure (AQ) remained statistically significant. The ITT estimate in model 1 (b = −26.73, SE = 2.73) was close to the conservative upper Lee bound (b = −22.34, SE = 4.55) after imputing outcomes for the attrition sample using wait-list control participants. Applying multiple imputation techniques for the attrition and dropout sample, based on initial treatment assignment yielded somewhat bigger SEs (model 3), but statistical significance of the ITT effect on AQ was maintained (d = 1.53; 95% CI, 1.15-1.91; Table 3).

Heterogeneous Treatment Effects and Mechanisms

Adding an interaction term to the ITT treatment estimation model of Table 3 revealed that treatment effectiveness related positively to baseline acrophobia symptoms (ie, as a modifier) (b = –0.306; SE = 0.15; P = .04), indicating higher potential benefits for participants with more severe acrophobia symptoms (eTable 3 in Supplement 2).

The experience-related results indicate that a reduction in acrophobia symptoms tends to be larger when the usability of the app was higher (System Usability Scale), when the feeling of being present in the virtual environment was higher (Igroup Presence Questionnaire), and when participants experienced more symptoms of cyber sickness (SSQ) (Figure 2). The robustness analysis further confirmed that the VR-CBT app had an impact on the anxiety for heights and that the general anxiety effect did not drive the results (eTable 4 in Supplement 2). eTable 5 in Supplement 2 provides regression-predicted reductions in posttest for 10-point bins of prescores suggesting that moderation is quantitative. Those with relatively low levels of baseline acrophobia did benefit significantly from the treatment.

Complete Cases

For individuals who returned the posttest, the between-group AQ effect size was d = 1.53 (95% CI, 1.15-1.91; number needed to treat: 1.4), and for the app individuals, the within-group effect size was d = 2.68 (95% CI, 2.09-3.22) between baseline and follow-up. Effect sizes on secondary outcomes for app individuals were also large (eTable 2 in Supplement 2). The VR-CBT app participants’ total mean (SD) Igroup Presence Questionnaire score was 42.69 (10.40) (range, 14-70).

Exploratory Analysis

For individuals who returned the AQ posttest (n = 56), all participants showed reliable change. Furthermore, 44 of 56 (79%) experienced clinically significant change, a change of 57.97 or more points on the AQ.

User Friendliness and Adverse Effects

The VR-CBT app was rated as user friendly (mean [SD], 75.35 [14.74], n = 55) and can be interpreted as a good and usable system.22 No deterioration or negative effects as defined by Rozental et al46 were identified, except for 24 participants who reported 1 or more symptoms of transient cyber sickness.

Discussion

To our knowledge, this randomized clinical trial tests the first fully self-guided treatment for acrophobia. We showed that an app-based VR-CBT program using low-cost VR goggles is both effective and user friendly. Large reductions in acrophobia symptoms on both the self-reported primary outcome measure AQ and secondary outcome measure Attention to Height Questionnaire were obtained. At 3-month follow-up, the results were maintained for the intervention group as demonstrated by the large within-group effect size. Importantly, these findings are in line with previous meta-analyses of guided VR acrophobia studies using high-end VR equipment12,14,15 and similar or better compared with therapist-guided exposure in vivo.47,48 In the current study, dropout rates were similar compared with dropout rates in previous VR studies.16 The app participants also experienced a greater reduction in general anxiety and an increased sense of mastery compared with the controls. As expected, because the baseline level of depression was low, the symptoms did not decrease; hence, there were no between-group differences. App participants perceived the VR environment as realistic although these results are slightly lower compared with previous research.49 This might be because of the quality of the cardboard VR goggles. However, the VR-CBT app was rated as user friendly and a good and usable system. Interestingly, the participants who experienced more cyber sickness also experienced a larger reduction in acrophobia symptoms. This might be an artifact of the SSQ because it overlaps with symptoms of anxiety. For exposure to be beneficial, experiencing anxiety is required. Therefore, the SSQ might not be a valid measure of cyber sickness for treating anxiety.

Strengths and Limitations

One of the key strengths of the present study was its ecological validity because the intervention was conducted in the participants’ natural environments instead of a research laboratory. Furthermore, participants received no guidance when using the VR-CBT app or filling in assessments, thereby ruling out any influence of human contact. Another strength was that the missing outcome observations for participants assigned to treatment to the VR-CBT app but unable to use the app because of incompatible smartphones were based on wait-list control group characteristics (ie, no treatment effect was imputed), yielding conservative overall treatment effect estimates. Furthermore, the ITT design has strong credibility in terms of aggregate effects when the intervention is scaled and adopted in real life.

The current study had several limitations. First, attrition was relatively high in the intervention condition because of incompatible smartphones. However, attrition was statistically unrelated to all observable characteristics at baseline. To retain a balanced experimental sample, the results were imputed for missing participants using regression-based imputation. Robustness and sensitivity analyses confirmed that potential bias concerns and precision concerns did not compromise the statistical significance of the results. Second, the data rely exclusively on self-reported measurements. However, the behavioral measurement effect sizes were similar to those calculated from other self-reported measures.13 Because the AQ has no validated cutoff scores, we were unable to provide results on the clinical threshold for acrophobia. We did not include a diagnostic interview because we wanted to isolate the true effect of VR; even in the assessment phase, human involvement may already have influenced the results.50,51 Third, because our follow-up period was only 3 months and because of dropout, the results are exploratory, and longer-term effects remain unknown. Fourth, the generalizability of the results is limited to Dutch Android smartphone users. However, most of the Dutch population (57%) uses Android.52

Future research is needed to replicate the VR-CBT app to draw firm conclusions about its effectiveness, compare the app directly with high-end VRET and exposure in vivo treatment, examine its long-term effects, determine cost effectiveness compared with treatment as usual, and examine whether the app can also be effective in the treatment of other phobias and mental health disorders.

Conclusions

In sum, our findings support the hypothesis that a fully self-guided app-based VR-CBT, which can be done at home at a fraction of the cost of existing evidence-based treatment options, strongly reduces acrophobia symptoms. The current study adds to the development of innovative and scalable delivery methods of evidence-based treatments and underlines that new technologies have the potential to transform mental health care worldwide.

Back to top
Article Information

Corresponding Author: Tara Donker, PhD, Department of Clinical, Neuro- and Developmental Psychology, Vrije Universiteit Amsterdam, Van der Boechorststraat 1, 1081 BT Amsterdam, the Netherlands, (t.donker@vu.nl).

Accepted for Publication: January 7, 2019.

Published Online: March 20, 2019. doi:10.1001/jamapsychiatry.2019.0219

Author Contributions: Drs Cornelisz and van Klaveren had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Donker and van Gelder.

Acquisition, analysis, or interpretation of data: Donker, Cornelisz, van Klaveren, van Straten, Carlbring.

Drafting of the manuscript: Donker, Cornelisz, van Klaveren.

Critical revision of the manuscript for important intellectual content: van Klaveren, van Straten, Carlbring, Cuijpers, van Gelder.

Statistical analysis: Cornelisz, van Klaveren.

Obtained funding: Donker.

Administrative, technical, or material support: van Gelder.

Supervision: Donker, van Straten, Cuijpers.

Conflict of Interest Disclosures: Dr Donker reports grants from NWO Toegepaste en Technische Wetenschappen and NWO Creative Industrie-KIEM during the conduct of the study. Drs Donker and van Gelder have developed the virtual reality application ZeroPhobia, which is used in the present study in collaboration with Vrije Universiteit. ZeroPhobia is intended for commercial release. Hence, Drs Donker and van Gelder have not been involved in data analysis or any decisions related to the publication of findings. No other disclosures were reported.

Funding/Support: This study has been funded by NWO Toegepaste en Technische Wetenschappen (2016/STW/00099738) and NWO Creative Industrie-KIEM (314-98-076).

Role of the Funder/Sponsor: The funders of the study had no role in study design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, writing of the report, review or approval of the manuscript; and the decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 3.

Additional Contributions: We extend our gratitude to all the participants of the study; Bruno de Vos, BA (Studio Barbaar), for designing ZeroPhobia; Wouter Houwen (Fontys Hogescholen Eindhoven), Ruud op den Kelder, MSc (Robo Ruud), Bas van Loon (Fontys Hogescholen Eindhoven), and Doruk Eker, MSc (Orb Amsterdam), for programming ZeroPhobia; Rufus van Baardwijk, MSc, for ZeroPhobia sound; and Stefanie van Esveld, MSc (Vrije Universiteit Amsterdam), and Niclas Fischer, MSc (Vrije Universiteit Amsterdam), for assistance in data recruitment and data collection. Bruno de Vos, BA; Wouter Houwen; Ruud op den Kelder, MSc; Bas van Loon; Doruk Eker, MSc; and Rufus van Baardwijk, MSc, recieved financial compensation.

References
1.
Kessler  RC, Berglund  PA, Bruce  ML,  et al.  The prevalence and correlates of untreated serious mental illness.  Health Serv Res. 2001;36(6 pt 1):987-1007.PubMedGoogle Scholar
2.
Fairburn  CG, Patel  V.  The impact of digital technology on psychological treatments and their dissemination.  Behav Res Ther. 2017;88:19-25. doi:10.1016/j.brat.2016.08.012PubMedGoogle ScholarCrossref
3.
de Graaf  R, ten Have  M, van Gool  C, van Dorsselaer  S.  Prevalence of mental disorders and trends from 1996 to 2009: results from the Netherlands Mental Health Survey and Incidence Study-2.  Soc Psychiatry Psychiatr Epidemiol. 2012;47(2):203-213. doi:10.1007/s00127-010-0334-8PubMedGoogle ScholarCrossref
4.
Kessler  RC, Petukhova  M, Sampson  NA, Zaslavsky  AM, Wittchen  HU.  Twelve-month and lifetime prevalence and lifetime morbid risk of anxiety and mood disorders in the United States.  Int J Methods Psychiatr Res. 2012;21(3):169-184. doi:10.1002/mpr.1359PubMedGoogle ScholarCrossref
5.
Depla  MF, ten Have  ML, van Balkom  AJ, de Graaf  R.  Specific fears and phobias in the general population: results from the Netherlands Mental Health Survey and Incidence Study (NEMESIS).  Soc Psychiatry Psychiatr Epidemiol. 2008;43(3):200-208. doi:10.1007/s00127-007-0291-zPubMedGoogle ScholarCrossref
6.
Stinson  FS, Dawson  DA, Patricia Chou  S,  et al.  The epidemiology of DSM-IV specific phobia in the USA: results from the National Epidemiologic Survey on Alcohol and Related Conditions.  Psychol Med. 2007;37(7):1047-1059. doi:10.1017/S0033291707000086PubMedGoogle ScholarCrossref
7.
Trumpf  J, Margraf  J, Vriends  N, Meyer  AH, Becker  ES.  Specific phobia predicts psychopathology in young women.  Soc Psychiatry Psychiatr Epidemiol. 2010;45(12):1161-1166. doi:10.1007/s00127-009-0159-5PubMedGoogle ScholarCrossref
8.
Choy  Y, Fyer  AJ, Goodwin  RD.  Specific phobia and comorbid depression: a closer look at the National Comorbidity Survey data.  Compr Psychiatry. 2007;48(2):132-136. doi:10.1016/j.comppsych.2006.10.010Google ScholarCrossref
9.
Abramowitz  JS.  The practice of exposure therapy: relevance of cognitive-behavioral theory and extinction theory.  Behav Ther. 2013;44(4):548-558. doi:10.1016/j.beth.2013.03.003PubMedGoogle ScholarCrossref
10.
Slater  M, Sanchez-Vives  MV.  Enhancing our lives with immersive virtual reality.  Frontiers Robotics AI.2016;3:76. doi:10.3389/frobt.2016.00074Google ScholarCrossref
11.
Fodor  LA, Coteț  CD, Cuijpers  P, Szamoskozi  Ș, David  D, Cristea  IA.  The effectiveness of virtual reality based interventions for symptoms of anxiety and depression: a meta-analysis.  Sci Rep. 2018;8(1):10323. doi:10.1038/s41598-018-28113-6PubMedGoogle ScholarCrossref
12.
Opriş  D, Pintea  S, García-Palacios  A, Botella  C, Szamosközi  Ş, David  D.  Virtual reality exposure therapy in anxiety disorders: a quantitative meta-analysis.  Depress Anxiety. 2012;29(2):85-93. doi:10.1002/da.20910PubMedGoogle ScholarCrossref
13.
Morina  N, Ijntema  H, Meyerbröker  K, Emmelkamp  PM.  Can virtual reality exposure therapy gains be generalized to real-life? a meta-analysis of studies applying behavioral assessments.  Behav Res Ther. 2015;74:18-24. doi:10.1016/j.brat.2015.08.010PubMedGoogle ScholarCrossref
14.
Carl  E, Stein  AT, Levihn-Coon  A,  et al Virtual reality exposure therapy for anxiety and related disorders: a meta-analysis of randomized controlled trials.  J Anxiety Disord. 2019;61:27-36. doi:10.1016/j.janxdis.2018.08.003Google ScholarCrossref
15.
Parsons  TD, Rizzo  AA.  Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis.  J Behav Ther Exp Psychiatry. 2008;39(3):250-261. doi:10.1016/j.jbtep.2007.07.007PubMedGoogle ScholarCrossref
16.
Benbow  AA, Anderson  PL.  A meta-analytic examination of attrition in virtual reality exposure therapy for anxiety disorders.  J Anxiety Disord. 2019;61:18-26. doi:10.1016/j.janxdis.2018.06.006Google ScholarCrossref
17.
Fernández-Álvarez  J, Rozental  A, Carlbring  P,  et al Deterioration rates in virtual reality therapy: an individual patient data level meta-analysis.  J Anxiety Disord. 2019;61:3-17. doi:10.1016/j.janxdis.2018.06.005Google ScholarCrossref
18.
Lindner  P, Miloff  A, Hamilton  W,  et al.  Creating state of the art, next-generation virtual reality exposure therapies for anxiety disorders using consumer hardware platforms: design considerations and future directions.  Cogn Behav Ther. 2017;46(5):404-420. doi:10.1080/16506073.2017.1280843PubMedGoogle ScholarCrossref
19.
Freeman  D, , Haselton  P, , Freeman  J, ,  et al Automated psychological therapy using immersive virtual reality for treatment of fear of heights: a single-blind, parallel-group, randomised controlled trial.  Lancet Psychiatry. 2018;5(8):625-632. doi:10.1016/S2215-0366(18)30226-8-8Google ScholarCrossref
20.
Donker  T, Van Esveld  S, Fischer  N, Van Straten  A.  0Phobia: towards a virtual cure for acrophobia: study protocol for a randomized controlled trial.  Trials. 2018;19(1):433. doi:10.1186/s13063-018-2704-6PubMedGoogle ScholarCrossref
21.
Cohen  DC.  Comparison of self-report and overt-behavioral procedures for assessing acrophobia.  Behav Ther. 1977;8(1):17-23. doi:10.1016/S0005-7894(77)80116-0Google ScholarCrossref
22.
Steinman  SA, Teachman  BA.  Cognitive processing and acrophobia: validating the Heights Interpretation Questionnaire.  J Anxiety Disord. 2011;25(7):896-902 .doi:10.1016/j.janxdis.2011.05.001Google ScholarCrossref
23.
Kroenke  K, Spitzer  RL, Williams  JB.  The PHQ-9: validity of a brief depression severity measure.  J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.xPubMedGoogle ScholarCrossref
24.
Donker  T, van Straten  A, Marks  I, Cuijpers  P.  A brief Web-based screening questionnaire for common mental disorders: development and validation.  J Med Internet Res. 2009;11(3):e19. doi:10.2196/jmir.1134PubMedGoogle ScholarCrossref
25.
Saghaei  M.  Random allocation software for parallel group randomized trials.  BMC Med Res Methodol. 2004;4:26. doi:10.1186/1471-2288-4-26PubMedGoogle ScholarCrossref
26.
Cuijpers  P, Huibers  M, Ebert  DD, Koole  SL, Andersson  G.  How much psychotherapy is needed to treat depression? a metaregression analysis.  J Affect Disord. 2013;149(1-3):1-13. doi:10.1016/j.jad.2013.02.030PubMedGoogle ScholarCrossref
27.
Emmelkamp  PM, Krijn  M, Hulsbosch  AM, de Vries  S, Schuemie  MJ, van der Mast  CA.  Virtual reality treatment versus exposure in vivo: a comparative evaluation in acrophobia.  Behav Res Ther. 2002;40(5):509-516. doi:10.1016/S0005-7967(01)00023-7PubMedGoogle ScholarCrossref
28.
Abelson  JL, Curtis  GC.  Cardiac and neuroendocrine responses to exposure therapy in height phobics: desynchrony within the ‘physiological response system’.  Behav Res Ther. 1989;27(5):561-567. doi:10.1016/0005-7967(89)90091-0PubMedGoogle ScholarCrossref
29.
Beck  AT, Epstein  N, Brown  G, Steer  RA.  An inventory for measuring clinical anxiety: psychometric properties.  J Consult Clin Psychol. 1988;56(6):893-897. doi:10.1037/0022-006X.56.6.893PubMedGoogle ScholarCrossref
30.
Bangor  A.  An empirical evaluation of the system usability scale.  Int J Hum Comput Interact. 2008;24(6):574-594. doi:10.1080/10447310802205776Google ScholarCrossref
31.
Schubert  T, Friedmann  F, Regenbrecht  H.  The experience of presence: factor analytic insights.  Presence Teleop Virt.2001;10(3):266-281. doi:10.1162/105474601300343603Google ScholarCrossref
32.
Pearlin  LI, Schooler  C.  The structure of coping.  J Health Soc Behav. 1978;19(1):2-21. doi:10.2307/2136319PubMedGoogle ScholarCrossref
33.
LaViola  JJ  Jr.  A discussion of cybersickness in virtual environments.  ACM SIGCHI Bullet. 2000;32(1):47-56. doi:10.1145/333329.333344Google ScholarCrossref
34.
Kennedy  RS, Lane  NE, Berbaum  KS, Lilienthal  MG.  Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness.  Int J Aviat Psychol. 1993;3(3):203-220. doi:10.1207/s15327108ijap0303_3Google ScholarCrossref
35.
Survalyzer Nederland  BV. Utrecht, the Netherlands. http://www.survalyzer.com. Accessed Augst 27, 2018.
36.
Donker  T, Bennett  K, Bennett  A,  et al.  Internet-delivered interpersonal psychotherapy versus internet-delivered cognitive behavioral therapy for adults with depressive symptoms: randomized controlled noninferiority trial.  J Med Internet Res. 2013;15(5):e82. doi:10.2196/jmir.2307.Google ScholarCrossref
37.
Issakidis  C, Andrews  G.  Pretreatment attrition and dropout in an outpatient clinic for anxiety disorders.  Acta Psychiatr Scand. 2004;109(6):426-433. doi:10.1111/j.1600-0047.2004.00264.xPubMedGoogle ScholarCrossref
38.
Lee  DS.  Training, wages, and sample selection: estimating sharp bounds on treatment effects.  Rev Econ Stud. 2009;76(3):1071-1102. doi:10.1111/j.1467-937X.2009.00536.xGoogle ScholarCrossref
39.
Karyotaki  E, Riper  H, Twisk  J,  et al.  Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data.  JAMA Psychiatry. 2017;74(4):351-359. doi:10.1001/jamapsychiatry.2017.0044PubMedGoogle ScholarCrossref
40.
Reliable and clinically significant change. Mounted by Chris Evans, 1998. http://www.psyctc.org/stats/rcsc.htm. Accessed August 18, 2018.
41.
Cohen  DC. Personality predictors and the outcome of varieties of desensitization. Unpublished doctoral dissertation, Harvard University, Cambridge, MA; 1972.
42.
Jacobson  NS, Truax  P.  Clinical significance: a statistical approach to defining meaningful change in psychotherapy research.  J Consult Clin Psychol. 1991;59(1):12-19. doi:10.1037/0022-006X.59.1.12PubMedGoogle ScholarCrossref
43.
Christensen  L, Mendoza  JL.  A method of assessing change in a single subject: an alteration of the RC index.  Behav Ther. 1986;17(3):305-308. doi:10.1016/S0005-7894(86)80060-0Google ScholarCrossref
44.
Furukawa  TA, Leucht  S.  How to obtain NNT from Cohen’s d: comparison of two methods.  PLoS One. 2011;6(4):e19070. doi:10.1371/journal.pone.0019070PubMedGoogle ScholarCrossref
45.
Pedersen  AB, Mikkelsen  EM, Cronin-Fenton  D,  et al.  Missing data and multiple imputation in clinical epidemiological research.  Clin Epidemiol. 2017;9:157-166. doi:10.2147/CLEP.S129785PubMedGoogle ScholarCrossref
46.
Rozental  A, Castonguay  L, Dimidjian  S,  et al.  Negative effects in psychotherapy: commentary and recommendations for future research and clinical practice.  BJPsych Open. 2018;4(4):307-312. doi:10.1192/bjo.2018.42PubMedGoogle ScholarCrossref
47.
Arroll  B, Wallace  HB, Mount  V, Humm  SP, Kingsford  DW.  A systematic review and meta-analysis of treatments for acrophobia.  Med J Aust. 2017;206(6):263-267. doi:10.5694/mja16.00540PubMedGoogle ScholarCrossref
48.
Wolitzky-Taylor  KB, Horowitz  JD, Powers  MB, Telch  MJ.  Psychological approaches in the treatment of specific phobias: a meta-analysis.  Clin Psychol Rev. 2008;28(6):1021-1037. doi:10.1016/j.cpr.2008.02.007PubMedGoogle ScholarCrossref
49.
Krijn  M, Emmelkamp  PMG, Biemond  R, de Wilde de Ligny  C, Schuemie  MJ, van der Mast  CA.  Treatment of acrophobia in virtual reality: the role of immersion and presence.  Behav Res Ther. 2004;42(2):229-239. doi:10.1016/S0005-7967(03)00139-6Google ScholarCrossref
50.
Andersson  G, Titov  N.  Advantages and limitations of Internet-based interventions for common mental disorders.  World Psychiatry. 2014;13(1):4-11. doi:10.1002/wps.20083PubMedGoogle ScholarCrossref
51.
Buntrock  C, Ebert  DD, Lehr  D,  et al.  Effect of a Web-based guided self-help intervention for prevention of major depression in adults with subthreshold depression: a randomized clinical trial.  JAMA. 2016;315(17):1854-1863. doi:10.1001/jama.2016.4326PubMedGoogle ScholarCrossref
52.
Statcounter GlobalStats. Mobile operating system market share Netherlands. Oct 2017- Oct 2018. http://gs.statcounter.com/os-market-share/mobile/netherlands. Accessed November 29, 2018.
×