Comparison of Simulated Treatment and Cost-effectiveness of a Stepped Care Case-Finding Intervention vs Usual Care for Posttraumatic Stress Disorder After a Natural Disaster

Key Points

Question  What is the reach, treatment effectiveness, and cost-effectiveness of a stepped care case-finding intervention for treatment of posttraumatic stress disorder, compared with usual care, after a natural disaster?

Findings  This simulation study of areas of New York City affected by Hurricane Sandy found that stepped care was associated with greater reach, was a more effective treatment, and had a favorable cost-effectiveness profile compared with usual care.

Meaning  Further testing in the field and consideration for implementation are warranted.

Importance  Psychiatric interventions offered after natural disasters commonly address subsyndromal symptom presentations, but often remain insufficient to reduce the burden of chronic posttraumatic stress disorder (PTSD).

Objective  To simulate a comparison of a stepped care case-finding intervention (stepped care [SC]) vs a moderate-strength single-level intervention (usual care [UC]) on treatment effectiveness and incremental cost-effectiveness in the 2 years after a natural disaster.

Design, Setting, and Participants  This study, which simulated treatment scenarios that start 4 weeks after landfall of Hurricane Sandy on October 29, 2012, and ending 2 years later, created a model of 2 642 713 simulated agents living in the areas of New York City affected by Hurricane Sandy.

Interventions  Under SC, cases were referred to cognitive behavioral therapy, an evidence-based therapy that aims to improve symptoms through problem solving and by changing thoughts and behaviors; noncases were referred to Skills for Psychological Recovery, an evidence-informed therapy that aims to reduce distress and improve coping and functioning. Under UC, all patients were referred only to Skills for Psychological Recovery.

Main Outcomes and Measures  The reach of SC compared with UC for 2 years, the 2-year reduction in prevalence of PTSD among the full population, the 2-year reduction in the proportion of PTSD cases among initial cases, and 10-year incremental cost-effectiveness.

Results  This population of 2 642 713 simulated agents was initialized with a PTSD prevalence of 4.38% (115 751 cases) and distributions of sex (52.6% female and 47.4% male) and age (33.9% aged 18-34 years, 49.0% aged 35-64 years, and 17.1% aged ≥65 years) that were comparable with population estimates in the areas of New York City affected by Hurricane Sandy. Stepped care was associated with greater reach and was superior to UC in reducing the prevalence of PTSD in the full population: absolute benefit was clear at 6 months (risk difference [RD], –0.004; 95% CI, –0.004 to –0.004), improving through 1.25 years (RD, –0.015; 95% CI, –0.015 to –0.014). Relative benefits of SC were clear at 6 months (risk ratio, 0.905; 95% CI, 0.898-0.913), with continued gains through 1.75 years (risk ratio, 0.615; 95% CI, 0.609-0.662). The absolute benefit of SC among cases was much stronger, emerging at 3 months (RD, –0.006; 95% CI, –0.007 to –0.005) and increasing through 1.5 years (RD, –0.338; 95% CI, –0.342 to –0.335). Relative benefits of SC among cases were equivalent to those observed in the full population. The incremental cost-effectiveness of SC compared with UC was $3428.71 to $6857.68 per disability-adjusted life year avoided, and $0.80 to $1.61 per PTSD-free day.

Conclusions and Relevance  The results of this simulation study suggest that SC for individuals with PTSD in the aftermath of a natural disaster is associated with greater reach than UC, more effectiveness than UC, and is well within the range of acceptability for cost-effectiveness. Results should be considered in light of limitations inherent to agent-based models.

Hurricane Sandy made landfall in New York City on October 29, 2012, resulting in disruptions of transportation, evacuation, destruction, and damage to homes, businesses, and several hospitals.^1-3 In the New York City area, 44 people died⁴ and injuries were common among evacuees.⁵ Disruptions in day-to-day life were another important consequence of the storm⁶ that may have reduced the pace of normalization and recovery,⁷ while limiting access to health care services.

After the storm, surveillance efforts suggested a high risk for poor mental health outcomes in the New York City boroughs.⁸ In the aftermath of Hurricane Sandy, the New York State Office of Mental Health provided crisis counseling services to an estimated 250 000 individuals between mid-November 2012 and late March 2014, including group and individual crisis counseling, and group education sessions.⁹ This effort used a treatment modality called Skills for Psychological Recovery (SPR), a skills-building intervention aimed at reducing stress and improving coping among survivors.¹⁰

Reflecting the inadequacy of available mental health care services, the estimated prevalence of probable cases of major depression was 8.9% and of hurricane-related PTSD was 2% among areas of New York City heavily affected by Hurricane Sandy approximately 1 to 1.5 years after the storm.¹¹ The same study found that more than half of those with a perceived mental health care service need did not receive such services. Similar patterns of high unmet need have been observed across a variety of disasters including Hurricane Katrina¹² and the September 11, 2001, World Trade Center terrorist attacks.¹³

One potential reason for the persistence of mental health conditions, including PTSD, after disasters could be the lack of effective triage to appropriate levels of care. Although mental health care delivery efforts in the wake of disaster exposures often include psychological first aid—a universal intervention¹⁴—and SPR, a moderate-strength intervention targeting distressed individuals,¹⁰ evidence from stepped care (SC) case-finding models suggests that the most efficient and effective population approach may be screening and triage to the appropriate level of care followed by ongoing systematic reevaluation.^15-17 This approach has been explicated and recommended in the context of disasters,¹⁸ but has not, to our knowledge, been tested in the field. Further investigation of SC models in the treatment of PTSD after a natural disaster is therefore warranted.

Beyond efficient screening and triage, reach, defined as an individual-level measure of participation,¹⁹ is another key feature of successful population-level interventions. Limited reach remains a major critique of specialty mental health care approaches to treatment of anxiety disorders, such as cognitive behavioral therapy (CBT).^20,21 However, SC approaches, which target a broader population base, may provide greater reach than either specialty or nonspecialty mental health care approaches alone. Accordingly, in assessing the potential outcome of SC, we are informed by the population impact framework, which defines population impact as a product of the proportion of intervention candidates, proportion of candidates who received the intervention (ie, reach), and reduction in disease incidence (or prevalence in nonprimary prevention contexts) among those who received the intervention.²²

To our knowledge, there are no real-world trials of SC interventions in disaster contexts, and such studies would be challenging to implement on a large scale. Accordingly, we approached this problem with the use of agent-based models to simulate in silico the effect of SC compared with usual care (UC) on hurricane-related PTSD. Agent-based models can, given appropriate assumptions, provide a useful counterfactual framework for the estimation of treatment effects.²³ Therefore, we aimed to estimate the effect of an SC intervention for PTSD after Hurricane Sandy using an agent-based model. We focused our model on the following 4 questions: What is the reach of SC compared with UC? What is the effect of SC compared with UC on the prevalence of PTSD in the full population during a 2-year period? What is the effect of SC compared with UC on the proportion of PTSD cases remaining during a 2-year period? What is the incremental cost-effectiveness of SC compared with UC during a 10-year period?

This modelling study, which simulated treatment scenarios that start 4 weeks after landfall of Hurricane Sandy on October 29, 2012, and ending 2 years later, focuses on individuals living in hurricane-affected areas of New York City at the time of Hurricane Sandy, and simulates the effect of recovery from PTSD, comparing UC and SC treatment models during 2 years of follow-up. Individuals in the simulation have a PTSD symptom score, reflecting a mix of borough-level demographic composition and hurricane exposure. Posttraumatic stress disorder symptom score can be collapsed into positive or negative case status, which is used to generate effectiveness measures. Every time step approximates 1 week, and individuals can access or discontinue care. Over time, some people will transition out of PTSD status owing to receipt of treatment or spontaneous remission. Among those who recover, a proportion experience recurrence of case status. A schematic overview of our model is provided in Figure 1A, and an overview of SC is provided in Figure 1B. Boston University Institutional Review Board waived approval for this study as all data came from previously published work.

Our population of 2 642 713 agents was initialized to serve as a to-scale approximation of areas of New York City affected by Hurricane Sandy. We applied parameters to our pseudopopulation using sociodemographic (ie, sex, age, and race/ethnicity) and borough characteristics derived from the American Community Survey estimates (2008-2012; 5-year mean) for New York City,^24,25 as shown in Table 1. Figure 2 shows a map of New York City with hurricane-affected areas² shaded in gray.

Initial distributions of the prevalence of and number of PTSD symptoms were estimated from an observational survey conducted among residents of affected areas after Hurricane Sandy.²⁶ This survey used the PTSD checklist to measure hurricane-related PTSD symptoms²⁷; probable cases of PTSD as determined by DSM-IV criteria were classified as those with a symptom score greater than 44. Specifically, we estimated prevalence and symptom counts for 24 strata composed of all possible combinations of sex (male or female), age (18-34, 35-64, or ≥65 years), and race/ethnicity (white non-Hispanic, black non-Hispanic, other non-Hispanic, or Hispanic). These estimates are provided in eTable 1 in the Supplement.

Using data from the Hurricane Sandy survey,²⁶ a receiver operating characteristic curve was used to estimate number of symptoms (range, 0-17) corresponding to a probable diagnosis of PTSD, using a symptom score of 44 on the PTSD checklist as a criterion standard.²⁷ Our cutpoint for case status was 6 symptoms, which maximized sensitivity (1.00) and specificity (0.96) within our observational sample. In turn, in the context of the present simulation, we set our base-case sensitivity and specificity to 0.80 each. In addition, we present estimates of treatment effectiveness and cost-effectiveness for an alternate screening case in which sensitivity is 0.80 and specificity is 1.00 (case A), and another alternate screening case in which sensitivity is 1.00 and specificity is 0.80 (case B).

Among those who do not receive care, 30% undergo remission without treatment, corresponding to the upper range of a meta-analytic estimate for remission among survivors of a broad array of traumatic events in psychotherapy trial waitlist control conditions.²⁸ We chose the upper end of this range, as natural disasters are considered among the less potent types of traumatic events.²⁹

Approximately 15% of recovered individuals undergo recurrence of PTSD after resolution of symptoms. We chose 15% to split the difference between recurrence rates of approximately 30% from samples of mixed traumatic exposure,³⁰ and lack of relapsing-remitting trajectories in studies that examine trajectories of symptoms after natural disasters.³¹ Among agents that underwent symptom relapse, symptoms returned to 80% of initial symptom levels, assuming that symptom levels experienced on relapse will have attenuated from their initial presentation.

Cognitive behavioral therapy takes place in our model for 8 sessions, per PTSD treatment guidelines.³² A course of CBT treatment reduces an agent’s symptoms by 0.36, as shown in the eAppendix in the Supplement. This parameter is a function of the reduction in symptom scale score on the PTSD Symptom Scale experienced during a 3-month period in the CBT arm of a randomized clinical trial³³ testing early treatment of PTSD. Up to 8 extra sessions were allowed among agents who did not recover after the initial 8 sessions.

As described in the SPR manual,¹⁰ the program is an intermediate-level intervention that may lower the need for formal mental health care treatment and aims to reduce distress while improving coping and functioning. In contrast to the reduction of symptoms by 0.36 associated with a course of CBT, 5 sessions of SPR will reduce an agent’s symptoms by a proportion of 0.20, as shown in the eAppendix in the Supplement.

Usual care includes provision of SPR. Stepped care includes a triage screening step at initial entry to services, as illustrated in Figure 1B. Agents are assessed for PTSD case status, and cases are offered CBT, while noncases are offered SPR.

Probability of enrollment in and use of SPR was estimated from a survey of New York City residents in the aftermath of the September 11, 2001, World Trade Center terrorist attacks.³⁴ These models included age, sex, and race/ethnicity and were estimated separately for cases and noncases, as shown in the eAppendix in the Supplement.

Probability of enrolling in CBT treatment was estimated from World Trade Center data using age, sex, and race/ethnicity, and was estimated separately among cases and noncases, as shown in the eAppendix in the Supplement.

We defined reach as the number of individuals per 10 000 who accessed treatment, and report reach attained at the initial time-step of treatment, which provides an optimal overall measure.

We measured treatment effectiveness using risk differences (RDs) (defined in the eAppendix in the Supplement) as a measure of absolute effects and risk ratios (RRs) (eAppendix in the Supplement) as a measure of relative effects. These measures were calculated for each 3-month time increment in the 2-year simulation.

The cost of CBT was set at $60 per session, based on an assessment of in-person delivery costs.³⁵ We did not have data on costs for SPR and made an assumption that the cost per person, per group session would be $15. We also performed sensitivity analyses to determine the effects of raising the cost of CBT to $120 per session and raising the cost of SPR to $30 per session.

Incremental cost-effectiveness ratios (ICERs) were computed among initial cases for each simulated comparison using disability-adjusted life years (DALYs) and PTSD-free days. Disability-adjusted life years provide a standardized method for estimating years of healthy life lost as a result of disability³⁶; their computation is described in the eAppendix in the Supplement. Disability weights used to characterize disability are derived from the Global Burden of Disease study³⁷ and are specific to mild (0.03), moderate (0.149), and severe (0.523) disease states. Posttraumatic stress disorder–free days are the number of days an agent spends in noncase status throughout the simulation. Calculation of ICERs is described in the eAppendix in the Supplement and applied for a 10-year time horizon, with a discounting rate of 3% on future cost and effectiveness. Our model takes a health care sector cost perspective.

Beginning 4 weeks after the hurricane, each of the 2 models (UC and SC) were run for 100 time steps, representing in total the passage of 2 years after Hurricane Sandy.

The model was implemented in Microsoft Visual Studio 2012 (Microsoft Corp) and developed using C++. To account for stochasticity in the modeling process, each model scenario was run 50 times, with mean statistical effect measures reported. The overview, design concept, and details protocol³⁸ for this agent-based model is outlined in the eAppendix in the Supplement, and provides an overview of the model and submodels, pseudocode, and an elaboration of design concepts and model details. We computed the 2.5th percentile and 97.5th percentile calculated across those 50 simulations.

As demonstrated in Table 1, our simulated population distributions of borough, age, sex, and race/ethnicity achieved a good representation of the 2008-2012 American Community Survey 5-year population estimates of the areas of New York City affected by Hurricane Sandy.^24,25

As shown in eTable 2 in the Supplement, the reach of SC (3137.25 per 10 000) exceeded that of UC (2118.60 per 10 000) in the base-case and other simulation scenarios.

Figure 3A demonstrates that SC began to yield a lower prevalence of PTSD starting at 6 months (RD, –0.004; 95% CI, –0.004 to –0.004), with the absolute benefit continuing to gradually improve through 1.25 years (RD, –0.015; 95% CI, –0.015 to –0.014), after which it reached a plateau. The relative benefit of SC, expressed as RRs, increased strongly from essentially no difference at 3 months (RR, 0.994; 95% CI, 0.914-1.081) to an approximately 30% lower risk at 1 year (RR, 0.667; 95% CI, 0.661-0.673), with benefit increasing through 1.75 years (RR, 0.615; 95% CI, 0.609-0.622).

Figure 3B demonstrates that the proportion of PTSD cases remaining under UC exceeded that among cases in the SC group at each 3-month interval. The absolute benefit increases from 3 months (RD, –0.006; 95% CI, –0.007 to –0.005) onward, reaching its maximum at 1.5 years (RD, –0.338; 95% CI, –0.342 to –0.335). The relative benefit of SC among cases was equivalent to that among the full population.

As shown in Table 2, the total cost of UC ranged from $8.37 to $16.75 million and the total cost of SC ranged from $50.94 to $101.87 million. Incremental cost-effectiveness ratios for SC compared with UC ranged from $3428.71 to $6857.68 per DALY averted, while the ICER per PTSD-free day ranged from $0.80 to $1.61.

Under alternate screening case A (eFigure 1 in the Supplement), treatment effectiveness was roughly equivalent to that observed under the base case. Under alternate screening case B (eFigure 2 in the Supplement), effectiveness was superior to that of the base case, with greater absolute and relative benefits of SC. As shown in eTable 3 in the Supplement, under alternate case A, ICERs ranged from $3192.55 per DALY averted to $6562.87 per DALY averted, and $0.75 per PTSD-free day to $1.54 per PTSD-free day. Under alternate case B (eTable 3 in the Supplement), ICERs ranged from $3394.23 per DALY averted to $6793.08 per DALY averted, and $0.80 per PTSD-free day to $1.59 per PTSD-free day.

To test the dependence of our results on treatment strength, we set the strength of each session of CBT and SPR to an equal value of symptom reduction (eFigure 3 in the Supplement). To test the dependence of our results on treatment enrollment probabilities we increased enrollment probabilities for SPR and decreased those for CBT (eFigure 4 in the Supplement). To test the dependence of our results on disability weighting, we decreased the severe PTSD disability weight to be the same strength as the moderate PTSD disability weight (eTable 4 in the Supplement). In all cases, SC was superior to UC and was cost-effective; under the disability weighting change, effectiveness was identical to that of the base case.

Using a simulated counterfactual model comparing UC with an SC case-finding intervention after a natural disaster, we demonstrated that SC was superior to UC in the early treatment of PTSD. Stepped care was associated with greater reach and more effectiveness than UC in reducing the prevalence of hurricane-related PTSD in the population and among cases. These effects emerged between 3 and 6 months after the hurricane and were sustained through the study period. Although SC was more costly overall than UC, the cost per DALY saved ($3428.71-$6857.68), and the cost per PTSD-free day ($0.80-$1.61) were well below what is generally considered a reasonable cost-benefit value.³⁹

These results suggest that the SC scenario we present provides greater population impact than UC and offers excellent value. Our sensitivity analyses suggest acceptable reach, effectiveness, and cost-effectiveness profiles that are robust to a range of screener performance characteristics, an equalization of per-session treatment effectiveness, changes to enrollment probabilities, and strength of disability weighting. Nonetheless, despite the excellent value afforded by SC, deployment of this model requires substantial resources, per our estimated absolute costs of $50.94 million to $101.87 million for deployment in New York City after Hurricane Sandy.

This simulation provides proof of concept for SC in the treatment of PTSD after a natural disaster, and further extends the rationale for this approach. Given the lack of randomized clinical trials or natural experiments to inform this question, these simulation results represent the best evidence to date supporting this approach after disasters. Future studies should consider internet-based interventions, which may extend the reach of SC and improve population outcomes. In addition, given limited uptake of CBT in postdisaster contexts⁴⁰ and other nonspecialty mental health care contexts,^41,42 if CBT reach after a disaster were enhanced, SC performance, as documented here, would likely improve.

The findings of this study are consistent with those of prior studies examining SC care approaches to treating mental health disorders. Stepped care was superior in the treatment of PTSD among injury survivors hospitalized for surgical care, although no cost-effectiveness estimates were provided.¹⁷ A prior study of SC for multiple anxiety disorders in primary care found an RD of –0.143 at the 18-month follow-up,¹⁵ and an ICER of about $4 per anxiety-free day.⁴³ Similarly, a collaborative care intervention for panic disorder, featuring many elements of SC, found an ICER of $8.40 per anxiety-free day.⁴⁴ The higher observed costs in those studies may be explained by the primary care and hospital settings, along with the inclusion of pharmacologic interventions, which are, on average, less cost-effective than psychotherapy.⁴⁵

This study should be considered in light of several limitations. First, and most importantly, this was a simulation, and the findings are based on the assumptions and design of our model. Second, geography is incorporated into our model only at symptom initialization, and not as part of the sub-models on mental health care treatment seeking that were run at each time step. Third, our cost-effectiveness estimates do not include administrative costs.

Working within these limitations, however, our models were initialized with strong empirical data,^26,33,34 providing confidence that our agent-based model provided reasonable estimates of mean treatment effect. Furthermore, we conducted sensitivity analyses to check model robustness under varying screening and cost assumptions and found further support for our main results. In addition, geography beyond the level of symptom initialization is not necessary in attempting to model mean treatment effects in New York City overall. Finally, instead of including administrative costs, we provide a range of cost-effectiveness profiles to allay concerns about underestimation.

This study has shown in a simulation that SC for early treatment of PTSD after a disaster was associated with greater reach, treatment effectiveness, and cost-effectiveness relative to the more universally applied interventions that resemble our UC condition. Accordingly, these results provide further proof of concept for the SC approach to treating PTSD after a disaster, and they warrant further study and application in real-world settings.

Corresponding Author: Gregory H. Cohen, MPhil, MSW, Department of Epidemiology, Boston University School of Public Health, 715 Albany St, Boston, MA 02118 (ghcohen@bu.edu).

Accepted for Publication: August 9, 2017.

Published Online: October 4, 2017. doi:10.1001/jamapsychiatry.2017.3037

Author Contributions: Mr Cohen and Dr Galea had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Cohen, Tamrakar, Lowe, Sampson, Galea.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Cohen, Tamrakar, Linas, Galea.

Critical revision of the manuscript for important intellectual content: Cohen, Tamrakar, Lowe, Sampson, Ettman, Ruggiero.

Statistical analysis: Cohen, Tamrakar, Sampson.

Obtained funding: Ruggiero, Galea.

Administrative, technical, or material support: Tamrakar, Sampson, Ettman, Linas, Galea.

Study supervision: Linas, Galea.

Conflict of Interest Disclosures: None reported.

Funding/Support: Funding for this work was provided by grant HITEP140015 from the US Department of Health and Human Services.

Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: Oliver Gruebner, PhD, Humboldt-Universität zu Berlin, created the map for this article. He was not compensated for his contribution.