[Skip to Navigation]
Sign In
Visual Abstract. Comparison of Teleintegrated Care and Telereferral Care for Treating Complex Psychiatric Disorders in Primary Care
Comparison of Teleintegrated Care and Telereferral Care for Treating Complex Psychiatric Disorders in Primary Care
Figure 1.  CONSORT Flow Diagram
CONSORT Flow Diagram

FQHC indicates federally qualified health center; PER, phone-enhanced referral; TCC, telepsychiatry collaborative care; TER, telepsychiatry/telepsychology–enhanced referral.

Figure 2.  Observed and Adjusted Outcomes for Stage 1 of the SMART Trial
Observed and Adjusted Outcomes for Stage 1 of the SMART Trial

GAD-7 indicates General Anxiety Disorder-7; ISS, Internal State Scale; MCS, Veterans RAND 12-item Health Survey Mental Component Summary; NA, not applicable; PCL-5, PTSD Checklist-5; PTSD, posttraumatic stress disorder; SCL-20, Hopkins Symptom Checklist; TCC, telepsychiatry collaborative care; TER, telepsychiatry/telepsychology–enhanced referral.

aFor clarity, the response set for 3 items was condensed, and scoring was adjusted accordingly.

bEuthymic mood state was modeled with a binary logistic regression; point estimates represent marginal probabilities at the sample mean of all covariates, and the effect size shown on the dot plot is the adjusted risk difference.

Figure 3.  Symptom Trajectories and Within-Group Effect Sizes by Stage 1 Intervention Condition
Symptom Trajectories and Within-Group Effect Sizes by Stage 1 Intervention Condition

GAD-7 indicates General Anxiety Disorder-7; ISS, Internal State Scale; MCS, Veterans RAND 12-item Health Survey Mental Component Summary; PCL-5, PTSD Checklist-5; PTSD, posttraumatic stress disorder; SCL-20, Hopkins Symptom Checklist; TCC, telepsychiatry collaborative care; TER, telepsychiatry/telepsychology–enhanced referral.

Table 1.  Baseline Characteristics of Patients Enrolled in the Study to Promote Innovation in Rural Integrated Telepsychiatry (SPIRIT) Trial
Baseline Characteristics of Patients Enrolled in the Study to Promote Innovation in Rural Integrated Telepsychiatry (SPIRIT) Trial
Table 2.  Observed and Model-Based Marginal Mediation Effects
Observed and Model-Based Marginal Mediation Effects
1.
Wang  PS, Lane  M, Olfson  M, Pincus  HA, Wells  KB, Kessler  RC.  Twelve-month use of mental health services in the United States: results from the National Comorbidity Survey Replication.   Arch Gen Psychiatry. 2005;62(6):629-640. doi:10.1001/archpsyc.62.6.629PubMedGoogle ScholarCrossref
2.
Health Resources and Services Administration. Health center program: impact and growth. Accessed April 4, 2021. https://bphc.hrsa.gov/about/healthcenterprogram/index.html
3.
National Association of Community Health Centers. America's health centers. November, 2017. Accessed August 1, 2021. https://www.nachc.org/wp-content/uploads/2017/11/Americas_Health_Centers_Nov_2017.pdf
4.
National Association of Community Health Centers. Community health center chartbook: 2021. Accessed April 14, 2021. https://www.nachc.org/wp-content/uploads/2021/04/Chartbook-Final-2021.pdf
5.
Thomas  KC, Ellis  AR, Konrad  TR, Holzer  CE, Morrissey  JP.  County-level estimates of mental health professional shortage in the United States.   Psychiatr Serv. 2009;60(10):1323-1328. doi:10.1176/ps.2009.60.10.1323PubMedGoogle ScholarCrossref
6.
Fortney  JC, Heagerty  PJ, Bauer  AM,  et al.  Study to Promote Innovation in Rural Integrated Telepsychiatry (SPIRIT): rationale and design of a randomized comparative effectiveness trial of managing complex psychiatric disorders in rural primary care clinics.   Contemp Clin Trials. 2020;90:105873. doi:10.1016/j.cct.2019.105873PubMedGoogle Scholar
7.
Fortney  JC, Pyne  JM, Edlund  MJ,  et al.  A randomized trial of telemedicine-based collaborative care for depression.   J Gen Intern Med. 2007;22(8):1086-1093. doi:10.1007/s11606-007-0201-9PubMedGoogle ScholarCrossref
8.
Fortney  JC, Pyne  JM, Mouden  SB,  et al.  Practice-based versus telemedicine-based collaborative care for depression in rural federally qualified health centers: a pragmatic randomized comparative effectiveness trial.   Am J Psychiatry. 2013;170(4):414-425. doi:10.1176/appi.ajp.2012.12050696PubMedGoogle ScholarCrossref
9.
Fortney  JC, Pyne  JM, Kimbrell  TA,  et al.  Telemedicine-based collaborative care for posttraumatic stress disorder: a randomized clinical trial.   JAMA Psychiatry. 2015;72(1):58-67. doi:10.1001/jamapsychiatry.2014.1575PubMedGoogle ScholarCrossref
10.
Fortney  JC, Veith  RC, Bauer  AM,  et al.  Developing telemental health partnerships between state medical schools and federally qualified health centers: navigating the regulatory landscape and policy recommendations.   J Rural Health. 2019;35(3):287-297. doi:10.1111/jrh.12323PubMedGoogle ScholarCrossref
11.
Kessler  RC, Akiskal  HS, Angst  J,  et al.  Validity of the assessment of bipolar spectrum disorders in the WHO CIDI 3.0.   J Affect Disord. 2006;96(3):259-269. doi:10.1016/j.jad.2006.08.018PubMedGoogle ScholarCrossref
12.
Lang  AJ, Stein  MB.  An abbreviated PTSD checklist for use as a screening instrument in primary care.   Behav Res Ther. 2005;43(5):585-594. doi:10.1016/j.brat.2004.04.005PubMedGoogle ScholarCrossref
13.
Fortney  JC, Pyne  JM, Ward-Jones  S,  et al.  Implementation of evidence-based practices for complex mood disorders in primary care safety net clinics.   Fam Syst Health. 2018;36(3):267-280. doi:10.1037/fsh0000357PubMedGoogle ScholarCrossref
14.
Almirall  D, Nahum-Shani  I, Sherwood  NE, Murphy  SA.  Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research.   Transl Behav Med. 2014;4(3):260-274. doi:10.1007/s13142-014-0265-0PubMedGoogle ScholarCrossref
15.
Jakupcak  M, Wagner  A, Paulson  A, Varra  A, McFall  M.  Behavioral activation as a primary care-based treatment for PTSD and depression among returning veterans.   J Trauma Stress. 2010;23(4):491-495. doi:10.1002/jts.20543PubMedGoogle ScholarCrossref
16.
Unützer  J, Choi  Y, Cook  IA, Oishi  S.  A web-based data management system to improve care for depression in a multicenter clinical trial.   Psychiatr Serv. 2002;53(6):671-673, 678. doi:10.1176/ps.53.6.671PubMedGoogle ScholarCrossref
17.
Fortney  JC, Unützer  J, Wrenn  G,  et al.  A tipping point for measurement-based care.   Psychiatr Serv. 2017;68(2):179-188. doi:10.1176/appi.ps.201500439PubMedGoogle ScholarCrossref
18.
Monson  CM, Schnurr  PP, Resick  PA, Friedman  MJ, Young-Xu  Y, Stevens  SP.  Cognitive processing therapy for veterans with military-related posttraumatic stress disorder.   J Consult Clin Psychol. 2006;74(5):898-907. doi:10.1037/0022-006X.74.5.898PubMedGoogle ScholarCrossref
19.
Otto  MW, Reilly-Harrington  NA, Knauz  RO, Henin  A, Kogan  JN, Sachs  GS.  Managing Bipolar Disorder: A Cognitive Behavior Treatment Program. Oxford University Press; 2008.
20.
Jones  D, Kazis  L, Lee  A,  et al.  Health status assessments using the Veterans SF-12 and SF-36: methods for evaluating outcomes in the Veterans Health Administration.   J Ambul Care Manage. 2001;24(3):68-86. doi:10.1097/00004479-200107000-00011PubMedGoogle ScholarCrossref
21.
Blevins  CA, Weathers  FW, Davis  MT, Witte  TK, Domino  JL.  The Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5): development and initial psychometric evaluation.   J Trauma Stress. 2015;28(6):489-498. doi:10.1002/jts.22059PubMedGoogle ScholarCrossref
22.
Derogatis  LR. Symptom Checklist-90–Revised (SCL-90-R). In: American Psychiatric Association; Rush AJ, ed.  Handbook of Psychiatric Measures. American Psychiatric Association; 2000:81-84.
23.
Altman  EG, Hedeker  D, Peterson  JL, Davis  JM.  The Altman Self-Rating Mania scale.   Biol Psychiatry. 1997;42(10):948-955. doi:10.1016/S0006-3223(96)00548-3PubMedGoogle ScholarCrossref
24.
Bauer  MS, Vojta  C, Kinosian  B, Altshuler  L, Glick  H.  The Internal State Scale: replication of its discriminating abilities in a multisite, public sector sample.   Bipolar Disord. 2000;2(4):340-346. doi:10.1034/j.1399-5618.2000.020409.xPubMedGoogle ScholarCrossref
25.
Spitzer  RL, Kroenke  K, Williams  JB, Löwe  B.  A brief measure for assessing generalized anxiety disorder: the GAD-7.   Arch Intern Med. 2006;166(10):1092-1097. doi:10.1001/archinte.166.10.1092PubMedGoogle ScholarCrossref
26.
Corrigan  PW, Salzer  M, Ralph  RO, Sangster  Y, Keck  L.  Examining the factor structure of the Recovery Assessment Scale.   Schizophr Bull. 2004;30(4):1035-1041. doi:10.1093/oxfordjournals.schbul.a007118PubMedGoogle ScholarCrossref
27.
Fortney  JC, Pyne  JM, Hawrilenko  M,  et al.  Psychometric properties of the Assessment of Perceived Access to Care (APAC) instrument.   J Ambul Care Manage. 2021;44(1):31-45. doi:10.1097/JAC.0000000000000358PubMedGoogle ScholarCrossref
28.
McNeish  D, Stapleton  LM.  Modeling clustered data with very few clusters.   Multivariate Behav Res. 2016;51(4):495-518. doi:10.1080/00273171.2016.1167008PubMedGoogle ScholarCrossref
29.
VanderWeele  TJ.  A unification of mediation and interaction: a 4-way decomposition.   Epidemiology. 2014;25(5):749-761. doi:10.1097/EDE.0000000000000121PubMedGoogle ScholarCrossref
30.
Muthén  B, Asparouhov  T.  Causal effects in mediation modeling: an introduction with applications to latent variables.   Struct Equation Modeling. 2015;22(1):12-23. doi:10.1080/10705511.2014.935843Google ScholarCrossref
31.
Graham  JW, Olchowski  AE, Gilreath  TD.  How many imputations are really needed? some practical clarifications of multiple imputation theory.   Prev Sci. 2007;8(3):206-213. doi:10.1007/s11121-007-0070-9PubMedGoogle ScholarCrossref
32.
Tang  F, Ishwaran  H.  Random forest missing data algorithms.   Stat Anal Data Min. 2017;10(6):363-377. doi:10.1002/sam.11348PubMedGoogle ScholarCrossref
33.
Stekhoven  DJ, Bühlmann  P.  MissForest—non-parametric missing value imputation for mixed-type data.   Bioinformatics. 2012;28(1):112-118. doi:10.1093/bioinformatics/btr597PubMedGoogle ScholarCrossref
34.
Shah  AD, Bartlett  JW, Carpenter  J, Nicholas  O, Hemingway  H.  Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study.   Am J Epidemiol. 2014;179(6):764-774. doi:10.1093/aje/kwt312PubMedGoogle ScholarCrossref
35.
Enders  CK.  Multiple imputation as a flexible tool for missing data handling in clinical research.   Behav Res Ther. 2017;98:4-18. doi:10.1016/j.brat.2016.11.008PubMedGoogle ScholarCrossref
36.
Cerimele  JM, LePoire  E, Fortney  JC, Hawrilenko  M, Unützer  J, Bauer  AM.  Bipolar disorder and PTSD screening and telepsychiatry diagnoses in primary care.   Gen Hosp Psychiatry. 2020;65:28-32. doi:10.1016/j.genhosppsych.2020.05.006PubMedGoogle ScholarCrossref
37.
Bauer  MS, McBride  L, Williford  WO,  et al; Cooperative Studies Program 430 Study Team.  Collaborative care for bipolar disorder: part II. impact on clinical outcome, function, and costs.   Psychiatr Serv. 2006;57(7):937-945. doi:10.1176/ps.2006.57.7.937PubMedGoogle ScholarCrossref
38.
Salyers  MP, Bosworth  HB, Swanson  JW, Lamb-Pagone  J, Osher  FC.  Reliability and validity of the SF-12 health survey among people with severe mental illness.   Med Care. 2000;38(11):1141-1150. doi:10.1097/00005650-200011000-00008PubMedGoogle ScholarCrossref
39.
Kudla  D, Lambert  M, Domin  S, Kasper  S, Naber  D.  Effectiveness, tolerability, and safety of ziprasidone in patients with schizophrenia or schizoaffective disorder: results of a multi-centre observational trial.   Eur Psychiatry. 2007;22(3):195-202. doi:10.1016/j.eurpsy.2006.06.004PubMedGoogle ScholarCrossref
40.
Kilbourne  AM, Post  EP, Nossek  A, Drill  L, Cooley  S, Bauer  MS.  Improving medical and psychiatric outcomes among individuals with bipolar disorder: a randomized controlled trial.   Psychiatr Serv. 2008;59(7):760-768. doi:10.1176/ps.2008.59.7.760PubMedGoogle ScholarCrossref
41.
Kilbourne  AM, Goodrich  D, Miklowitz  DJ, Austin  K, Post  EP, Bauer  MS.  Characteristics of patients with bipolar disorder managed in VA primary care or specialty mental health care settings.   Psychiatr Serv. 2010;61(5):500-507. doi:10.1176/ps.2010.61.5.500PubMedGoogle ScholarCrossref
42.
Bartels  SJ, Coakley  EH, Zubritsky  C,  et al; PRISM-E Investigators.  Improving access to geriatric mental health services: a randomized trial comparing treatment engagement with integrated versus enhanced referral care for depression, anxiety, and at-risk alcohol use.   Am J Psychiatry. 2004;161(8):1455-1462. doi:10.1176/appi.ajp.161.8.1455PubMedGoogle ScholarCrossref
43.
Areán  PA, Ayalon  L, Jin  C,  et al.  Integrated specialty mental health care among older minorities improves access but not outcomes: results of the PRISMe study.   Int J Geriatr Psychiatry. 2008;23(10):1086-1092. doi:10.1002/gps.2100PubMedGoogle ScholarCrossref
44.
Hays  RD, Morales  LS.  The RAND-36 measure of health-related quality of life.   Ann Med. 2001;33(5):350-357. doi:10.3109/07853890109002089PubMedGoogle ScholarCrossref
45.
Engel  CC, Jaycox  LH, Freed  MC,  et al.  Centrally assisted collaborative telecare for posttraumatic stress disorder and depression among military personnel attending primary care: a randomized clinical trial.   JAMA Intern Med. 2016;176(7):948-956. doi:10.1001/jamainternmed.2016.2402PubMedGoogle ScholarCrossref
46.
Mueser  KT, Rosenberg  SD, Xie  H,  et al.  A randomized controlled trial of cognitive-behavioral treatment for posttraumatic stress disorder in severe mental illness.   J Consult Clin Psychol. 2008;76(2):259-271. doi:10.1037/0022-006X.76.2.259PubMedGoogle ScholarCrossref
47.
Meredith  LS, Eisenman  DP, Han  B,  et al.  Impact of collaborative care for underserved patients with PTSD in primary care: a randomized controlled trial.   J Gen Intern Med. 2016;31(5):509-517. doi:10.1007/s11606-016-3588-3PubMedGoogle ScholarCrossref
48.
van der Voort  TY, van Meijel  B, Goossens  PJ,  et al.  Collaborative care for patients with bipolar disorder: randomised controlled trial.   Br J Psychiatry. 2015;206(5):393-400. doi:10.1192/bjp.bp.114.152520PubMedGoogle ScholarCrossref
49.
Suppes  T, Rush  AJ, Dennehy  EB,  et al; Texas Medication Algorithm Project.  Texas Medication Algorithm Project, phase 3 (TMAP-3): clinical results for patients with a history of mania.   J Clin Psychiatry. 2003;64(4):370-382. doi:10.4088/JCP.v64n0403PubMedGoogle ScholarCrossref
50.
Craske  MG, Stein  MB, Sullivan  G,  et al.  Disorder-specific impact of coordinated anxiety learning and management treatment for anxiety disorders in primary care.   Arch Gen Psychiatry. 2011;68(4):378-388. doi:10.1001/archgenpsychiatry.2011.25PubMedGoogle ScholarCrossref
51.
Schnurr  PP, Friedman  MJ, Oxman  TE,  et al.  RESPECT-PTSD: re-engineering systems for the primary care treatment of PTSD, a randomized controlled trial.   J Gen Intern Med. 2013;28(1):32-40. doi:10.1007/s11606-012-2166-6PubMedGoogle ScholarCrossref
52.
Watts  BV, Schnurr  PP, Mayo  L, Young-Xu  Y, Weeks  WB, Friedman  MJ.  Meta-analysis of the efficacy of treatments for posttraumatic stress disorder.   J Clin Psychiatry. 2013;74(6):e541-e550. doi:10.4088/JCP.12r08225PubMedGoogle ScholarCrossref
53.
Carlo  AD, Drake  L, Ratzliff  ADH, Chang  D, Unützer  J.  Sustaining the collaborative care model (CoCM): billing newly available CoCM CPT codes in an academic primary care system.   Psychiatr Serv. 2020;71(9):972-974. doi:10.1176/appi.ps.201900581PubMedGoogle ScholarCrossref
54.
Fortney  JC, Pyne  JM, Turner  EE,  et al.  Telepsychiatry integration of mental health services into rural primary care settings.   Int Rev Psychiatry. 2015;27(6):525-539. doi:10.3109/09540261.2015.1085838PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    August 25, 2021

    Comparison of Teleintegrated Care and Telereferral Care for Treating Complex Psychiatric Disorders in Primary Care: A Pragmatic Randomized Comparative Effectiveness Trial

    Author Affiliations
    • 1Department of Psychiatry and Behavioral Sciences, School of Medicine, University of Washington, Seattle
    • 2Department of Veterans Affairs, Health Services Research and Development, Center of Innovation for Veteran-Centered and Value-Driven Care, Seattle, Washington
    • 3Department of Psychiatry and Behavioral Sciences, College of Medicine, University of Arkansas for Medical Sciences, Little Rock
    • 4Department of Veterans Affairs, Health Services Research and Development, Center for Mental Healthcare and Outcomes Research, Little Rock, Arkansas
    • 5University of Michigan Medical School, Ann Arbor
    • 6Department of Veterans Affairs, Health Services Research and Development, Center for Clinical Management Research, VA Ann Arbor Healthcare System, Ann Arbor, Michigan
    • 7Department of Biostatistics, School of Public Health, University of Washington, Seattle
    • 8Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Palo Alto, California
    • 9Department of Bioethics and Humanities, University of Washington, Seattle
    • 10School of Economic Sciences, Washington State University, Pullman
    • 11Community Health Plan of Washington, Seattle
    • 12SPIRIT Consumer Advisory Board, Eldorado, Arkansas
    JAMA Psychiatry. 2021;78(11):1189-1199. doi:10.1001/jamapsychiatry.2021.2318
    Key Points

    Question  Which is more effective, an integrated or referral approach to using clinic-to-clinic interactive video to deliver evidence-based mental health treatments to patients with complex psychiatric disorders in primary care clinics?

    Findings  In this pragmatic randomized comparative effectiveness trial including 1004 adult participants, both approaches significantly and substantially improved clinical outcomes. The referral approach used substantially more mental health specialist time than the integrated approach.

    Meaning  Based on findings from this trial, from a health care system perspective, clinical leadership should implement whichever approach is most sustainable; from a societal perspective, policy makers should incentivize the integrated approach because it required less scarce mental health specialist time.

    Abstract

    Importance  Only one-third of patients with complex psychiatric disorders engage in specialty mental health care, and only one-tenth receive adequate treatment in primary care. Scalable approaches are critically needed to improve access to effective mental health treatments in underserved primary care settings.

    Objective  To compare 2 clinic-to-clinic interactive video approaches to delivering evidence-based mental health treatments to patients in primary care clinics.

    Design, Setting, and Participants  This pragmatic comparative effectiveness trial used a sequential, multiple-assignment, randomized trial (SMART) design with patient-level randomization. Adult patients treated at 24 primary care clinics without on-site psychiatrists or psychologists from 12 federally qualified health centers in 3 states who screened positive for posttraumatic stress disorder and/or bipolar disorder and who were not already receiving pharmacotherapy from a mental health specialist were recruited from November 16, 2016, to June 30, 2019, and observed for 12 months.

    Interventions  Two approaches were compared: (1) telepsychiatry/telepsychology–enhanced referral (TER), where telepsychiatrists and telepsychologists assumed responsibility for treatment, and (2) telepsychiatry collaborative care (TCC), where telepsychiatrists provided consultation to the primary care team. TER included an adaptive intervention (phone-enhanced referral [PER]) for patients not engaging in treatment, which involved telephone outreach and motivational interviewing.

    Main Outcomes and Measures  Survey questions assessed patient-reported outcomes. The Veterans RAND 12-item Health Survey Mental Component Summary (MCS) score was the primary outcome (range, 0-100). Secondary outcomes included posttraumatic stress disorder symptoms, manic symptoms, depressive symptoms, anxiety symptoms, recovery, and adverse effects.

    Results  Of 1004 included participants, 701 of 1000 (70.1%) were female, 660 of 994 (66.4%) were White, and the mean (SD) age was 39.4 (12.9) years. Baseline MCS scores were 2 SDs below the US mean; the mean (SD) MCS scores were 39.7 (14.1) and 41.2 (14.2) in the TCC and TER groups, respectively. There was no significant difference in 12-month MCS score between those receiving TCC and TER (β = 1.0; 95% CI, −0.8 to 2.8; P = .28). Patients in both groups experienced large and clinically meaningful improvements from baseline to 12 months (TCC: Cohen d = 0.81; 95% CI, 0.67 to 0.95; TER: Cohen d = 0.90; 95% CI, 0.76 to 1.04). For patients not engaging in TER at 6 months, there was no significant difference in 12-month MCS score between those receiving PER and TER (β = 2.0; 95% CI, −1.7 to 5.7; P = .29).

    Conclusions and Relevance  In this comparative effectiveness trial of patients with complex psychiatric disorders randomized to receive TCC or TER, significantly and substantially improved outcomes were observed in both groups. From a health care system perspective, clinical leadership should implement whichever approach is most sustainable.

    Trial Registration  ClinicalTrials.gov Identifier: NCT02738944

    Introduction

    Only one-third of individuals with bipolar disorder (BD) and posttraumatic stress disorder (PTSD) receive specialty mental health care during the course of a year.1 In primary care settings, only one-tenth of patients with BD and PTSD receive adequate care compared with more than half in specialty mental health.1 Managing complex psychiatric disorders is especially challenging for primary care clinicians in federally qualified health centers (FQHCs). There are nearly 1400 FQHCs with more than 13 000 clinic locations that provide services to 30 million individuals in the US.2 Almost half (44%) of patients treated at FQHCs live in rural areas,3 91% live in poverty,4 and 62% are from racial or ethnic minority groups.4 While 97% of FQHCs offer on-site mental health services, only 12% of mental health staff are psychiatrists or licensed clinical psychologists.4 The shortage of psychiatrists and psychologists in rural and poverty-stricken areas precipitates this chronic staffing problem in FQHCs.1,5

    The widespread adoption of telepsychiatry and telepsychology owing to the COVID-19 pandemic could potentially increase access for primary care patients with complex psychiatric disorders living in underserved areas. The Study to Promote Innovation in Rural Integrated Telepsychiatry (SPIRIT) trial was a pragmatic trial (PCS-1406-19295) designed to identify the best approach to delivering telemental health services to primary care clinics. Specifically, the SPIRIT trial compared the effectiveness of telepsychiatry collaborative care (TCC) and telepsychiatry/telepsychology–enhanced referral (TER) to treat BD and PTSD.6

    Quiz Ref IDTCC is an integrated population-based model of care.7-9 By integrating BD and PTSD treatment into primary care and taking a population-based care management approach, TCC is expected to engage a higher proportion of patients in treatment than TER, which is a traditional referral model of care that focuses exclusively on patients attending scheduled appointments. However, TCC telepsychiatrists only provide consultation to the primary care team, while TER telepsychiatrists and telepsychologists provide direct ongoing treatment to patients. Therefore, the as-treated effectiveness of TCC may be lower than TER. TCC and TER represent clinical equipoise with respect to intent-to-treat effectiveness, with TCC expected to have greater engagement but lower as-treated effectiveness for those engaged in treatment and TER expected to have lower engagement but greater as-treated effectiveness for those engaged in treatment. We hypothesized that the greater engagement in TCC would result in better intent-to-treat effectiveness. Although not a common practice, telemental health referral models could adopt a more population-based approach, and the SPIRIT trial was also designed to test the intent-to-treat effectiveness of such an approach. For patients not engaging in treatment, we also hypothesized that phone-enhanced referral (PER), which uses telephone outreach to encourage patients to initiate or reengage in treatment, would result in better engagement and outcomes than continued TER.

    Methods

    Human subjects protection oversight was provided by the institutional review boards of the University of Arkansas for Medical Sciences, University of Michigan, and University of Washington. Written informed consent was obtained for all study participants. The trial was designed and conducted in close collaboration with consumer and policy advisory boards. The trial protocol can be found in Supplement 1.

    Study Sites

    A total of 24 clinics from 12 FQHCs in 3 states (Arkansas, Michigan, and Washington) participated. Clinics were eligible if they had no psychiatrists or licensed clinical psychologists practicing on site. Telepsychiatrists and telepsychologists from state medical schools were credentialed and privileged to practice at the FQHC and documented their clinical assessment and treatment plan in their electronic health record.10 Study participants received up to 12 months of treatment and presented to the clinic for interactive video encounters.

    Study Population

    Patients screening positive for BD and/or PTSD were enrolled from November 16, 2016, to June 30, 2019 (Figure 1). Inclusion criteria were based on screening instruments (Composite International Diagnostic Interview version 3.011 for BD and PTSD Checklist-6 for PTSD12) administered during annual wellness visits rather than structured clinical assessments to reflect real-world practice and to maximize external validity. To minimize false-positives and screening burden,13 only patients screening positive for depression (Patient Health Questionnaire-9 score of 10 or greater) were screened for BD and PTSD. Exclusion criteria were minimal: (1) age younger than 18 years, (2) unable to communicate in English or Spanish, (3) lacked capacity to consent, (4) no future FQHC visits planned, and (5) already being prescribed psychotropic medications by a psychiatrist or psychiatric nurse practitioner at baseline. Patients prescribed psychotropic medications by a primary care clinician were included.

    Randomization

    A sequential, multiple-assignment, randomized trial (SMART) design was used because TER is an adaptive intervention.14 Patients were initially randomized to TCC and TER using blocking and stratification by FQHC and screening status. Patients screening positive for both BD and PTSD were categorized as having BD for stratification purposes. At 6 months, patients assigned to TER with 2 or less interactive video encounters were randomized a second time (using blocking and stratification by FQHC and screening status) to either continued TER or PER.

    TCC

    Quiz Ref IDOn-site behavioral health care managers (eg, social workers or nurses) and off-site telepsychiatrist consultants supported primary care clinicians, who prescribed all psychotropic medications. Care managers provided psychoeducation, conducted outreach and treatment engagement activities, and delivered behavioral activation (BA) psychotherapy.15 Care managers used a web-based registry, the Care Management Tracking System (CMTS),16 to monitor engagement and symptom severity (ie, measurement-based care17). Telepsychiatrist consultants conducted an initial diagnostic assessment for all patients. The telepsychiatrists met weekly with care managers for case reviews (approximately 10 minutes per patient) to identify patients not engaging in or responding to care and suggested treatment recommendations to primary care clinicians. TCC care was documented in CMTS and the electronic health record.

    TER

    Patients initially had a telepsychiatry encounter to establish diagnosis and develop a treatment plan. Telepsychiatrists ordered laboratory tests and electronically prescribed medications in the FQHC’s electronic health record. If referred by telepsychiatrists, telepsychologists delivered either cognitive processing therapy (CPT)18 for PTSD or cognitive behavioral therapy (CBT)19 for BD. FQHC staff scheduled appointments and sent appointment reminders. Telepsychiatrists and telepsychologists monitored symptoms using CMTS, and treatment was documented in CMTS and the electronic health record.

    PER

    The adaptive intervention involved reaching out telephonically to patients. FQHC staff scheduled telephone appointments with the telephone psychologist, who encouraged patients to attend telepsychiatry/telepsychology interactive video encounters in the clinic. In contrast to TER, which focuses on treating patients attending appointments, PER focuses on engaging patients in treatment.

    Survey

    Quiz Ref IDTelephone or web-based surveys were administered at baseline and 6 and 12 months later. Treatment group assignment was masked for telephone interviewers. The primary outcome was mental health functioning at 12 months as measured by the Veterans RAND 12-item Health Survey Mental Component Summary (MCS) score (range, 0 to 100).20 The MCS is a non–disease-specific assessment of vitality, role functioning, social functioning, and feeling calm and peaceful, and scores represent an outcome that is highly relevant to patients. Secondary outcomes included the (1) PTSD Checklist-5 (PCL-5)21 for PTSD, (2) Hopkins Symptom Checklist (SCL-20)22 for depression, (3) Altman Mania Rating Scale (AMRS)23 for mania, (4) Internal State Scale (ISS)24 for mood state, (5) General Anxiety Disorder-725 for general anxiety, (6) Recovery Assessment Scale,26 and (7) number of moderate and severe adverse effects from psychotropic medications. Access was measured using the Assessment of Perceived Access to Care (APAC).27 Engagement mediators included self-reported psychotropic medication prescription and adherence and number of psychotherapy encounters (BA, CPT, or CBT) documented in CMTS.

    Statistical Analysis

    Intent-to-treat hypotheses were tested using a 2-level model with longitudinal observations (level 1) nested within patients (level 2). To account for stratified randomization, screening status and FQHCs were included as fixed effects.28 Mixed models included a random intercept, random linear slope, and 6-month and 12-month indicators to allow for nonlinear change over time. The adjusted difference between TCC and TER at each time point was tested with a group × time interaction term, using a 2-sided α level of .05. Outcomes at 12 months for the TER group represented the average effect of being randomized to PER or continued TER in months 6 to 12 for nonresponders. The same modeling approach was used to analyze the second-stage randomization, except that only 1 group × time interaction term was used to estimate the difference between TER and PER at 12 months. Disorder-specific outcomes (PCL-5, ISS, and AMRS scores) were analyzed only in the subgroup that screened positive for the disorder. Euthymic mood based on the ISS was specified as a binary variable and modeled with a log link for relative risks. Adverse effects were analyzed as a count variable and modeled with a negative binomial distribution.

    Assuming a 30% loss to follow-up, a sample size of 1000 was needed to have 80% power (α = .05) to detect a small effect size (Cohen d = 0.21) for MCS between first-stage randomization arms and 80% power to detect a medium effect size (Cohen d = 0.43) between second-stage randomization arms (assuming 50% did not engage in TER). For secondary outcomes, we calculated 95% CIs but did not test hypotheses.

    To test hypotheses about baseline treatment effect modifiers, moderator × group × time interaction terms were added. In this model, baseline MCS scores were incorporated as a level-2 covariate, and time was centered at 6 months. To test mediation hypotheses, a multivariate structural equation model was specified. Treatment effects on the mediators (A paths in eFigure 1 in Supplement 2), psychotherapy engagement, and medication engagement (prescribed medications and always/mostly adherent) were modeled using negative binomial and logit links, respectively. The mediators were included as time-varying predictors (B paths in eFigure 1 in Supplement 2) of changes in MCS, with a treatment-mediator interaction29 to account for the possibility that psychotherapy and medication management had differential effectiveness across condition. Indirect effects were calculated using the delta method.30

    To account for missing data at follow-up, 100 complete data sets were imputed31 using random forest imputation.32-34 Imputation was stratified by initial randomization group to allow potentially different effects and separate covariance structures by group.35 Potential missingness mechanisms were examined by correlating loss to follow-up status with key baseline characteristics. To gauge the sensitivity of results to violations of the missing-at-random assumption, we calculated how the primary outcome estimate changed under different proportions and effect sizes of nonignorable missingness. Statistical analyses were conducted using Mplus version 8 (Muthen & Muthen).

    Results

    Most patients screening positive for BD and/or PTSD and assessed for eligibility (1931 of 2464 [78.4%]) did not meet any exclusion criteria, and two-thirds of eligible patients (1214 of 1931 [62.9%]) consented to participate (Figure 1). The most common reason for ineligibility was already being prescribed psychotropic medications by a psychiatrist or psychiatric nurse practitioner at baseline (453 of 533 [85.0%]; Figure 1).

    Of 1004 included participants, 701 of 1000 (70.1%) were female, 660 of 994 (66.4%) were White, and the mean (SD) age was 39.4 (12.9) years (Table 1). A total of 503 of 1003 patients (50.1%) lived in a rural area, 222 of 1002 (22.2%) did not graduate from high school, 660 of 1002 (65.9%) were unmarried, 789 of 976 (80.8%) were not employed full time, 620 of 946 (65.5%) lived below the 2016 federal poverty level, and 827 of 997 (83.0%) were publicly insured or uninsured. Most participants (880 of 975 [90.3%]) reported a perceived need for mental health treatment, and 691 of 975 (70.9%) were taking psychotropic medications prescribed by their primary care clinician at baseline. The mean (SD) MCS score (range, 0 to 100) at baseline was 30.8 (11.2). Nearly all participants screened positive for PTSD (978 [97.4%]) with moderate severity (mean [SD] PCL-5 score, 48.0 [17.7]), and 760 (78.1%) reported a traumatic event meeting PTSD diagnostic criterion. One-third of participants (367 [36.6%]) screened positive for lifetime BD, and of these, only 34 (9.3%) were euthymic at baseline. Physical and mental health comorbidities were common. Of the 767 participants with a TCC or TER telepsychiatry encounter during the 12-month treatment period, 357 (46.5%) were diagnosed with PTSD only, 135 (17.6%) were diagnosed with BD and PTSD, and 57 (7.4%) were diagnosed with BD only.36 Other telepsychiatrist-assigned diagnoses included unipolar depression (457 [59.6%]), anxiety (287 [37.4%]), alcohol use disorder (103 [13.4%]), other substance use disorder (109 [14.2%]), personality disorder (75 [9.8%]), and schizophrenia (29 [3.8%]).

    A total of 464 of 508 patients randomized to TCC (91.3%) had 1 or more care manager encounters, and among these patients, the mean (SD) number of care manager encounters was 10.3 (7.9). A total of 403 (79.3%) had 1 or more BA encounters, and of these, the mean (SD) number of encounters was 9.6 (7.7). A total of 389 (76.6%) had a telepsychiatry consultation, and of these, the mean (SD) number of encounters was 1.4 (0.9). Overall, the TCC sample used 1.06 telepsychiatry encounters per randomized patient. In addition, 447 patients (88.0%) had a case review by the care manager and telepsychiatrist, and of these, the mean (SD) number of encounters was 6.4 (4.1). A total of 388 of 496 patients randomized to TER (78.2%) had 1 or more telepsychiatry encounters, and of these patients, the mean (SD) number of encounters was 4.3 (3.1). A total of 223 (45.0%) had 1 or more telepsychology encounter, and of these, the mean (SD) number of encounters was 6.4 (4.4). Overall, the TER sample used 6.4 telepsychiatry/telepsychology encounters per randomized patient.

    Quiz Ref IDBaseline MCS scores were 2 SDs below the US mean; the mean (SD) MCS scores were 39.7 (14.1) and 41.2 (14.2) in the TCC and TER groups, respectively. MCS scores were not clinically or significantly different between those randomized to TER and TCC at 12 months (β = 1.0; 95% CI, −0.8 to 2.8; P = .28; Figure 2). Treatment effects measured by MCS were not significantly different across age, sex, race or ethnicity, screening results, or baseline MCS scores. Analyses examining the sensitivity of results to missing-at-random assumption violations found that even under extreme violations, bias was likely to be very small (eFigure 2 in Supplement 2). Patients in both groups experienced large and clinically meaningful improvements in MCS scores from baseline to 12 months (TCC: Cohen d = 0.81; 95% CI, 0.67 to 0.95; TER: Cohen d = 0.90; 95% CI, 0.76 to 1.04; Figure 3). While we could rule out a null effect, group differences in PCL-5 scores were not clinically meaningful (β = –5.1; 95% CI, −7.7 to −2.4; Figure 2). Both groups experienced large clinically meaningful decreases in PCL-5 scores from baseline to 12 months (TCC: Cohen d = −0.72; 95% CI, −0.84 to −0.59; TER: Cohen d = −0.96; 95% CI, −1.12 to −0.88; Figure 3). Group differences in the proportions reporting euthymic mood were near zero (Figure 2), and both groups had large increases in the proportion euthymic by 12 months (TCC: adjusted difference in proportion, 0.17; 95% CI, 0.09 to 0.25; TER: adjusted difference in proportion, 0.19; 95% CI, 0.10 to 0.28; Figure 3). Group differences for the remaining secondary outcomes were not clinically meaningful, and patients in both groups exhibited clinical improvement (Figure 3). For the second-stage randomization, 228 trial participants (46.0%) did not engage in TER and were rerandomized. There was no meaningful between-group difference in medication engagement between those randomized to TER and PER (adjusted difference in proportion, −0.03; 95% CI, −0.15 to 0.07), but a substantially greater proportion randomized to PER had a psychotherapy encounter (adjusted difference in proportion, 0.09; 95% CI, 0.02 to 0.16). However, there was no significant between-group difference in 12-month MCS scores between those randomized to TER and PER (β = 2.0; 95% CI, −1.7 to 5.7; P = .29; eFigure 3 in Supplement 2) or other secondary outcomes.

    Perceived access to mental health, as measured by APAC scores, were not significantly different between those randomized to TER and TCC at 12 months (β = 0.11; 95% CI, −0.03 to 0.25; P = .10). Patients in both groups experienced statistically significant increases in perceived access from baseline to 12 months (TCC: Cohen d = 0.29; 95% CI, 0.17 to 0.41; TER: Cohen d = 0.43; 95% CI, 0.31 to 0.55). Results from the mediation analysis (Table 2) indicated no meaningful differences in self-reported medication engagement between those randomized to TER and TCC at 12 months (adjusted difference in proportion, −0.03; 95% CI, −0.09 to 0.03). However, patients randomized to TER averaged fewer psychotherapy encounters than those randomized to TCC (β = −4.2; 95% CI, −5.0 to −3.5). The effect of psychotherapy engagement on 12-month MCS scores was different between groups (β = 0.55; 95% CI, 0.26 to 0.84). For patients randomized to TER, each additional psychotherapy encounter was associated with an improvement in the 12-month MCS score of 0.33 (β = 0.41; 95% CI, 0.16 to 0.66). For patients randomized to TCC, the association was small and not significantly different from zero (β = −0.14; 95% CI, −0.28 to 0).

    Discussion

    Primary care patients enrolled in the trial had baseline MCS scores 2 SDs below the national mean, a level of functioning that is lower than typically seen in specialty mental health settings.37-41 TCC and TER both improved perceived access and treatment engagement, with more than three-fourths of patients attending telepsychiatry appointments. For those not engaging in TER, telephone-based outreach (PER) significantly increased engagement in psychotherapy (but not pharmacotherapy), although clinical outcomes were not improved.

    Patients in both the TCC and TER arms experienced fewer adverse effects from psychotropic medications and statistically significant and clinically meaningful improvements in outcomes. These results are in sharp contrast to the Primary Care Research in Substance Abuse and Mental Health for Elderly (PRISM-E) trial, which compared integrated and referral care and found low levels of engagement42 and little clinical improvement.43 TCC represents a more intensive intervention than the integrated care provided in the PRISM-E trial, and TER was virtually colocated in primary care whereas the PRIMS-E trial used off-site referrals. TCC and TER also both used measurement-based care, which the PRISM-E trial interventions did not.

    The improvement in clinical outcomes is unlikely to reflect regression to the mean for 3 reasons. First, inclusion criteria were based on highly sensitive screeners administered during routine annual wellness visits rather than on structured diagnostic interviews administered to patients seeking treatment for a new episode of care. Second, the 9-point to 10-point increase in the MCS represents an improvement of a standard deviation and is considerably higher than the 3-point to 5-point minimum clinically important difference.44 We are aware of only 2 mental health trials in which the intervention group experienced a greater than 9-point improvement in MCS.7,8 Third, in the usual care groups of BD and PTSD trials, MCS scores do not improve,37,45-47 and with one exception,47 disorder-specific symptom severity does not improve.9,37,45,48-51

    Mediation analysis suggested that engagement in telepsychologist-delivered CPT and/or CBT in the TER arm, but not care manager-delivered BA in the TCC arm, was positively associated with MCS improvement. These results are consistent with a meta-analysis of PTSD psychotherapy trials, which found that CPT has a larger effect size than other psychotherapies.52

    Importantly, there were no clinically meaningful differences in outcomes between patients randomized to TCC and TER and no evidence of treatment heterogeneity. From a health care system perspective, results suggest that clinical leadership should implement whichever evidence-based practice is most sustainable. TCC is billable under new billing codes.53 From a societal perspective, TCC should be incentivized by policy makers because it leverages scarce telepsychiatrist capacity through consultation and case-review.54 Telepsychiatry encounters were 3-fold (mean [SD] of 4.3 [3.1] vs 1.4 [0.9]) greater in TER than TCC.

    Limitations

    Quiz Ref IDThe SPIRIT trial was the largest mental health trial conducted in rural primary care clinics and one of the largest trials ever conducted in FQHCs. However, there were some limitations. The trial design did not include a usual care group. Survey follow-up rates at 6 months and 12 months were relatively low. However, there were only slight differences between survey completers and noncompleters (eTable in the Supplement), and results were not sensitive to alternative assumptions about missing data. In addition, the SPIRIT trial made some compromises regarding pragmatism by using research funds to provide some of the clinical services and using telepsychiatrists and telepsychologists from state medical schools.

    Conclusions

    In summary, implementing TCC and/or TER in primary clinics in rural and underserved areas increased access to and engagement in effective treatments and substantially improved outcomes. By leveraging scarce telepsychiatrist capacity, TCC is able to serve more patients than TER.

    Back to top
    Article Information

    Accepted for Publication: June 29, 2021.

    Published Online: August 25, 2021. doi:10.1001/jamapsychiatry.2021.2318

    Corresponding Author: John C. Fortney, PhD, Department of Psychiatry and Behavioral Sciences, School of Medicine, University of Washington, 1959 NE Pacific St, Box 356560, Seattle, WA 98195 (fortneyj@uw.edu).

    Author Contributions: Drs Fortney and Hawrilenko had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Study concept and design: Fortney, Bauer, Cerimele, Pyne, Heagerty, Kaysen, Bowen, Moore, Metzger, Hafer, Unützer.

    Acquisition, analysis, or interpretation of data: Fortney, Bauer, Cerimele, Pyne, Pfeiffer, Heagerty, Hawrilenko, Zielinski, Kaysen, Bowen, Moore, Ferro, Shushan, Nolan, Dalack, Unützer.

    Drafting of the manuscript: Fortney, Hawrilenko, Bowen, Moore, Metzger, Hafer.

    Critical revision of the manuscript for important intellectual content: Bauer, Cerimele, Pyne, Pfeiffer, Heagerty, Hawrilenko, Zielinski, Kaysen, Moore, Ferro, Shushan, Nolan, Dalack, Unützer.

    Statistical analysis: Fortney, Heagerty, Hawrilenko.

    Obtained funding: Fortney, Kaysen, Hafer.

    Administrative, technical, or material support: Fortney, Bauer, Pfeiffer, Zielinski, Kaysen, Bowen, Moore, Ferro, Metzger, Shushan, Hafer, Nolan, Dalack, Unützer.

    Study supervision: Fortney, Pyne, Kaysen, Unützer.

    Conflict of Interest Disclosures: Drs Fortney, Bauer, Cerimele, Heagerty, Kaysen, Shushan, and Dalack have received grants from the Patient-Centered Outcomes Research Institute during the conduct of the study. Dr Bauer has received grants from the National Institutes of Health, Washington State Legislature Integrated Care Training Program, and Premera Blue Cross Rural Mental Health Integration Initiative outside the submitted work. Dr Pyne has received grants from the University of Arkansas for Medical Sciences during the conduct of the study. Dr Kaysen has received personal fees from Elsevier Publishing and honorarium for conducting trainings or workshops in cognitive processing therapy outside the submitted work. Dr Moore has received support from the University of Washington during the conduct of the study. No other disclosures were reported.

    Funding/Support: This work was supported by a grant PCS-1406-19295 from the Patient-Centered Outcomes Research Institute to Drs Fortney and Unützer.

    Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The statements in this article are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute, its board of governors, or methodology committee.

    Data Sharing Statement: See Supplement 3.

    Additional Contributions: We thank the patients and staff of Family Health Centers, Omak, Washington; Moses Lake Community Health Center, Moses Lake, Washington; Sea Mar Community Health Centers, Seattle, Washington; Yakima Neighborhood Health Services, Yakima, Washington; Boston Mountain Rural Health Centers, Marshal, Arkansas; East Arkansas Family Health Center, West Memphis; Lee County Cooperative Clinic, Marianna, Arkansas; Cherry Health, Grand Rapids, Michigan; Family Medical Center, Temperance, Michigan; Great Lakes Bay Health Centers, Saginaw, Michigan; InterCare Community Health Network, Bangor, Michigan; and Upper Great Lakes Family Health Center, Calumet, Michigan. We thank the staff of the Community Health Plan of Washington, the Community Health Centers of Arkansas, and the Michigan Primary Care Association. We thank our policy advisory board: Donald Weaver, MD (National Association of Community Health Centers, Bethesda, Maryland); Ed Larkins, MHA (Family Medical Center); Jay Shore, MD (University of Colorado at Denver); LaShannon Spencer, MPA, MHSA (Community Health Centers of Arkansas, Little Rock); Nancy Speck, PhD (National Association for Rural Mental Health, Houston, Texas); Sara Coates, MA, MPH, and Dawne Velianoff, LMSW (Michigan Primary Care Association, Lansing); and Susan Ward-Jones, MD (East Arkansas Family Health Center). We thank our consumer advisor board: Bernadette McDonald, BS; Florence Fee, JD (No Health Without Mental Health, Arlington, Virginia); Ingrid Deetz, BS, Kevin Einbinder, BS, and Betsy O’Brien, BS (Depression and Bipolar Support Alliance, Chicago, Illinois); Kim Arnold, BS, Marquitta Magnini, BS, and Shawn McCown, BS (National Alliance on Mental Illness, Little Rock, Arkansas); Jode Freyholtz-London, AAS (Wellness in the Woods, Eagle Bend, Minnesota); and Tammy Heral and Jeremy Mullins. We thank the members of our external data safety monitoring board: Thomas Belin, PhD (University of California, Los Angeles); John Williams, MD (Duke University, Durham, North Carolina); and Mark Williams, MD (Mayo Clinic, Rochester, Minnesota). Also, thanks to our research staff Suzanne Hunter, BS, Jared Bechtel, BS, Diane Powers, MA, MBA, Shiyu Chen, MS, and Morgan Johnson, MS (Department of Psychiatry, University of Washington, Seattle); and Nick Ponomarev, PhD, and Dan Vakoch, MS (Social and Economic Sciences Research Center, Washington State University, Pullman). We thank the following electronic health record vendors for giving us substantial discounts on the user licenses for the telepsychiatrists and telepsychologists: NextGen Healthcare, eClinicalWorks, Centricity (GE Healthcare), Allscripts, and SuccessEHS, Inc. Members of the policy advisory board and consumer advisory board as well as our research staff were compensated for their work.

    References
    1.
    Wang  PS, Lane  M, Olfson  M, Pincus  HA, Wells  KB, Kessler  RC.  Twelve-month use of mental health services in the United States: results from the National Comorbidity Survey Replication.   Arch Gen Psychiatry. 2005;62(6):629-640. doi:10.1001/archpsyc.62.6.629PubMedGoogle ScholarCrossref
    2.
    Health Resources and Services Administration. Health center program: impact and growth. Accessed April 4, 2021. https://bphc.hrsa.gov/about/healthcenterprogram/index.html
    3.
    National Association of Community Health Centers. America's health centers. November, 2017. Accessed August 1, 2021. https://www.nachc.org/wp-content/uploads/2017/11/Americas_Health_Centers_Nov_2017.pdf
    4.
    National Association of Community Health Centers. Community health center chartbook: 2021. Accessed April 14, 2021. https://www.nachc.org/wp-content/uploads/2021/04/Chartbook-Final-2021.pdf
    5.
    Thomas  KC, Ellis  AR, Konrad  TR, Holzer  CE, Morrissey  JP.  County-level estimates of mental health professional shortage in the United States.   Psychiatr Serv. 2009;60(10):1323-1328. doi:10.1176/ps.2009.60.10.1323PubMedGoogle ScholarCrossref
    6.
    Fortney  JC, Heagerty  PJ, Bauer  AM,  et al.  Study to Promote Innovation in Rural Integrated Telepsychiatry (SPIRIT): rationale and design of a randomized comparative effectiveness trial of managing complex psychiatric disorders in rural primary care clinics.   Contemp Clin Trials. 2020;90:105873. doi:10.1016/j.cct.2019.105873PubMedGoogle Scholar
    7.
    Fortney  JC, Pyne  JM, Edlund  MJ,  et al.  A randomized trial of telemedicine-based collaborative care for depression.   J Gen Intern Med. 2007;22(8):1086-1093. doi:10.1007/s11606-007-0201-9PubMedGoogle ScholarCrossref
    8.
    Fortney  JC, Pyne  JM, Mouden  SB,  et al.  Practice-based versus telemedicine-based collaborative care for depression in rural federally qualified health centers: a pragmatic randomized comparative effectiveness trial.   Am J Psychiatry. 2013;170(4):414-425. doi:10.1176/appi.ajp.2012.12050696PubMedGoogle ScholarCrossref
    9.
    Fortney  JC, Pyne  JM, Kimbrell  TA,  et al.  Telemedicine-based collaborative care for posttraumatic stress disorder: a randomized clinical trial.   JAMA Psychiatry. 2015;72(1):58-67. doi:10.1001/jamapsychiatry.2014.1575PubMedGoogle ScholarCrossref
    10.
    Fortney  JC, Veith  RC, Bauer  AM,  et al.  Developing telemental health partnerships between state medical schools and federally qualified health centers: navigating the regulatory landscape and policy recommendations.   J Rural Health. 2019;35(3):287-297. doi:10.1111/jrh.12323PubMedGoogle ScholarCrossref
    11.
    Kessler  RC, Akiskal  HS, Angst  J,  et al.  Validity of the assessment of bipolar spectrum disorders in the WHO CIDI 3.0.   J Affect Disord. 2006;96(3):259-269. doi:10.1016/j.jad.2006.08.018PubMedGoogle ScholarCrossref
    12.
    Lang  AJ, Stein  MB.  An abbreviated PTSD checklist for use as a screening instrument in primary care.   Behav Res Ther. 2005;43(5):585-594. doi:10.1016/j.brat.2004.04.005PubMedGoogle ScholarCrossref
    13.
    Fortney  JC, Pyne  JM, Ward-Jones  S,  et al.  Implementation of evidence-based practices for complex mood disorders in primary care safety net clinics.   Fam Syst Health. 2018;36(3):267-280. doi:10.1037/fsh0000357PubMedGoogle ScholarCrossref
    14.
    Almirall  D, Nahum-Shani  I, Sherwood  NE, Murphy  SA.  Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research.   Transl Behav Med. 2014;4(3):260-274. doi:10.1007/s13142-014-0265-0PubMedGoogle ScholarCrossref
    15.
    Jakupcak  M, Wagner  A, Paulson  A, Varra  A, McFall  M.  Behavioral activation as a primary care-based treatment for PTSD and depression among returning veterans.   J Trauma Stress. 2010;23(4):491-495. doi:10.1002/jts.20543PubMedGoogle ScholarCrossref
    16.
    Unützer  J, Choi  Y, Cook  IA, Oishi  S.  A web-based data management system to improve care for depression in a multicenter clinical trial.   Psychiatr Serv. 2002;53(6):671-673, 678. doi:10.1176/ps.53.6.671PubMedGoogle ScholarCrossref
    17.
    Fortney  JC, Unützer  J, Wrenn  G,  et al.  A tipping point for measurement-based care.   Psychiatr Serv. 2017;68(2):179-188. doi:10.1176/appi.ps.201500439PubMedGoogle ScholarCrossref
    18.
    Monson  CM, Schnurr  PP, Resick  PA, Friedman  MJ, Young-Xu  Y, Stevens  SP.  Cognitive processing therapy for veterans with military-related posttraumatic stress disorder.   J Consult Clin Psychol. 2006;74(5):898-907. doi:10.1037/0022-006X.74.5.898PubMedGoogle ScholarCrossref
    19.
    Otto  MW, Reilly-Harrington  NA, Knauz  RO, Henin  A, Kogan  JN, Sachs  GS.  Managing Bipolar Disorder: A Cognitive Behavior Treatment Program. Oxford University Press; 2008.
    20.
    Jones  D, Kazis  L, Lee  A,  et al.  Health status assessments using the Veterans SF-12 and SF-36: methods for evaluating outcomes in the Veterans Health Administration.   J Ambul Care Manage. 2001;24(3):68-86. doi:10.1097/00004479-200107000-00011PubMedGoogle ScholarCrossref
    21.
    Blevins  CA, Weathers  FW, Davis  MT, Witte  TK, Domino  JL.  The Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5): development and initial psychometric evaluation.   J Trauma Stress. 2015;28(6):489-498. doi:10.1002/jts.22059PubMedGoogle ScholarCrossref
    22.
    Derogatis  LR. Symptom Checklist-90–Revised (SCL-90-R). In: American Psychiatric Association; Rush AJ, ed.  Handbook of Psychiatric Measures. American Psychiatric Association; 2000:81-84.
    23.
    Altman  EG, Hedeker  D, Peterson  JL, Davis  JM.  The Altman Self-Rating Mania scale.   Biol Psychiatry. 1997;42(10):948-955. doi:10.1016/S0006-3223(96)00548-3PubMedGoogle ScholarCrossref
    24.
    Bauer  MS, Vojta  C, Kinosian  B, Altshuler  L, Glick  H.  The Internal State Scale: replication of its discriminating abilities in a multisite, public sector sample.   Bipolar Disord. 2000;2(4):340-346. doi:10.1034/j.1399-5618.2000.020409.xPubMedGoogle ScholarCrossref
    25.
    Spitzer  RL, Kroenke  K, Williams  JB, Löwe  B.  A brief measure for assessing generalized anxiety disorder: the GAD-7.   Arch Intern Med. 2006;166(10):1092-1097. doi:10.1001/archinte.166.10.1092PubMedGoogle ScholarCrossref
    26.
    Corrigan  PW, Salzer  M, Ralph  RO, Sangster  Y, Keck  L.  Examining the factor structure of the Recovery Assessment Scale.   Schizophr Bull. 2004;30(4):1035-1041. doi:10.1093/oxfordjournals.schbul.a007118PubMedGoogle ScholarCrossref
    27.
    Fortney  JC, Pyne  JM, Hawrilenko  M,  et al.  Psychometric properties of the Assessment of Perceived Access to Care (APAC) instrument.   J Ambul Care Manage. 2021;44(1):31-45. doi:10.1097/JAC.0000000000000358PubMedGoogle ScholarCrossref
    28.
    McNeish  D, Stapleton  LM.  Modeling clustered data with very few clusters.   Multivariate Behav Res. 2016;51(4):495-518. doi:10.1080/00273171.2016.1167008PubMedGoogle ScholarCrossref
    29.
    VanderWeele  TJ.  A unification of mediation and interaction: a 4-way decomposition.   Epidemiology. 2014;25(5):749-761. doi:10.1097/EDE.0000000000000121PubMedGoogle ScholarCrossref
    30.
    Muthén  B, Asparouhov  T.  Causal effects in mediation modeling: an introduction with applications to latent variables.   Struct Equation Modeling. 2015;22(1):12-23. doi:10.1080/10705511.2014.935843Google ScholarCrossref
    31.
    Graham  JW, Olchowski  AE, Gilreath  TD.  How many imputations are really needed? some practical clarifications of multiple imputation theory.   Prev Sci. 2007;8(3):206-213. doi:10.1007/s11121-007-0070-9PubMedGoogle ScholarCrossref
    32.
    Tang  F, Ishwaran  H.  Random forest missing data algorithms.   Stat Anal Data Min. 2017;10(6):363-377. doi:10.1002/sam.11348PubMedGoogle ScholarCrossref
    33.
    Stekhoven  DJ, Bühlmann  P.  MissForest—non-parametric missing value imputation for mixed-type data.   Bioinformatics. 2012;28(1):112-118. doi:10.1093/bioinformatics/btr597PubMedGoogle ScholarCrossref
    34.
    Shah  AD, Bartlett  JW, Carpenter  J, Nicholas  O, Hemingway  H.  Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study.   Am J Epidemiol. 2014;179(6):764-774. doi:10.1093/aje/kwt312PubMedGoogle ScholarCrossref
    35.
    Enders  CK.  Multiple imputation as a flexible tool for missing data handling in clinical research.   Behav Res Ther. 2017;98:4-18. doi:10.1016/j.brat.2016.11.008PubMedGoogle ScholarCrossref
    36.
    Cerimele  JM, LePoire  E, Fortney  JC, Hawrilenko  M, Unützer  J, Bauer  AM.  Bipolar disorder and PTSD screening and telepsychiatry diagnoses in primary care.   Gen Hosp Psychiatry. 2020;65:28-32. doi:10.1016/j.genhosppsych.2020.05.006PubMedGoogle ScholarCrossref
    37.
    Bauer  MS, McBride  L, Williford  WO,  et al; Cooperative Studies Program 430 Study Team.  Collaborative care for bipolar disorder: part II. impact on clinical outcome, function, and costs.   Psychiatr Serv. 2006;57(7):937-945. doi:10.1176/ps.2006.57.7.937PubMedGoogle ScholarCrossref
    38.
    Salyers  MP, Bosworth  HB, Swanson  JW, Lamb-Pagone  J, Osher  FC.  Reliability and validity of the SF-12 health survey among people with severe mental illness.   Med Care. 2000;38(11):1141-1150. doi:10.1097/00005650-200011000-00008PubMedGoogle ScholarCrossref
    39.
    Kudla  D, Lambert  M, Domin  S, Kasper  S, Naber  D.  Effectiveness, tolerability, and safety of ziprasidone in patients with schizophrenia or schizoaffective disorder: results of a multi-centre observational trial.   Eur Psychiatry. 2007;22(3):195-202. doi:10.1016/j.eurpsy.2006.06.004PubMedGoogle ScholarCrossref
    40.
    Kilbourne  AM, Post  EP, Nossek  A, Drill  L, Cooley  S, Bauer  MS.  Improving medical and psychiatric outcomes among individuals with bipolar disorder: a randomized controlled trial.   Psychiatr Serv. 2008;59(7):760-768. doi:10.1176/ps.2008.59.7.760PubMedGoogle ScholarCrossref
    41.
    Kilbourne  AM, Goodrich  D, Miklowitz  DJ, Austin  K, Post  EP, Bauer  MS.  Characteristics of patients with bipolar disorder managed in VA primary care or specialty mental health care settings.   Psychiatr Serv. 2010;61(5):500-507. doi:10.1176/ps.2010.61.5.500PubMedGoogle ScholarCrossref
    42.
    Bartels  SJ, Coakley  EH, Zubritsky  C,  et al; PRISM-E Investigators.  Improving access to geriatric mental health services: a randomized trial comparing treatment engagement with integrated versus enhanced referral care for depression, anxiety, and at-risk alcohol use.   Am J Psychiatry. 2004;161(8):1455-1462. doi:10.1176/appi.ajp.161.8.1455PubMedGoogle ScholarCrossref
    43.
    Areán  PA, Ayalon  L, Jin  C,  et al.  Integrated specialty mental health care among older minorities improves access but not outcomes: results of the PRISMe study.   Int J Geriatr Psychiatry. 2008;23(10):1086-1092. doi:10.1002/gps.2100PubMedGoogle ScholarCrossref
    44.
    Hays  RD, Morales  LS.  The RAND-36 measure of health-related quality of life.   Ann Med. 2001;33(5):350-357. doi:10.3109/07853890109002089PubMedGoogle ScholarCrossref
    45.
    Engel  CC, Jaycox  LH, Freed  MC,  et al.  Centrally assisted collaborative telecare for posttraumatic stress disorder and depression among military personnel attending primary care: a randomized clinical trial.   JAMA Intern Med. 2016;176(7):948-956. doi:10.1001/jamainternmed.2016.2402PubMedGoogle ScholarCrossref
    46.
    Mueser  KT, Rosenberg  SD, Xie  H,  et al.  A randomized controlled trial of cognitive-behavioral treatment for posttraumatic stress disorder in severe mental illness.   J Consult Clin Psychol. 2008;76(2):259-271. doi:10.1037/0022-006X.76.2.259PubMedGoogle ScholarCrossref
    47.
    Meredith  LS, Eisenman  DP, Han  B,  et al.  Impact of collaborative care for underserved patients with PTSD in primary care: a randomized controlled trial.   J Gen Intern Med. 2016;31(5):509-517. doi:10.1007/s11606-016-3588-3PubMedGoogle ScholarCrossref
    48.
    van der Voort  TY, van Meijel  B, Goossens  PJ,  et al.  Collaborative care for patients with bipolar disorder: randomised controlled trial.   Br J Psychiatry. 2015;206(5):393-400. doi:10.1192/bjp.bp.114.152520PubMedGoogle ScholarCrossref
    49.
    Suppes  T, Rush  AJ, Dennehy  EB,  et al; Texas Medication Algorithm Project.  Texas Medication Algorithm Project, phase 3 (TMAP-3): clinical results for patients with a history of mania.   J Clin Psychiatry. 2003;64(4):370-382. doi:10.4088/JCP.v64n0403PubMedGoogle ScholarCrossref
    50.
    Craske  MG, Stein  MB, Sullivan  G,  et al.  Disorder-specific impact of coordinated anxiety learning and management treatment for anxiety disorders in primary care.   Arch Gen Psychiatry. 2011;68(4):378-388. doi:10.1001/archgenpsychiatry.2011.25PubMedGoogle ScholarCrossref
    51.
    Schnurr  PP, Friedman  MJ, Oxman  TE,  et al.  RESPECT-PTSD: re-engineering systems for the primary care treatment of PTSD, a randomized controlled trial.   J Gen Intern Med. 2013;28(1):32-40. doi:10.1007/s11606-012-2166-6PubMedGoogle ScholarCrossref
    52.
    Watts  BV, Schnurr  PP, Mayo  L, Young-Xu  Y, Weeks  WB, Friedman  MJ.  Meta-analysis of the efficacy of treatments for posttraumatic stress disorder.   J Clin Psychiatry. 2013;74(6):e541-e550. doi:10.4088/JCP.12r08225PubMedGoogle ScholarCrossref
    53.
    Carlo  AD, Drake  L, Ratzliff  ADH, Chang  D, Unützer  J.  Sustaining the collaborative care model (CoCM): billing newly available CoCM CPT codes in an academic primary care system.   Psychiatr Serv. 2020;71(9):972-974. doi:10.1176/appi.ps.201900581PubMedGoogle ScholarCrossref
    54.
    Fortney  JC, Pyne  JM, Turner  EE,  et al.  Telepsychiatry integration of mental health services into rural primary care settings.   Int Rev Psychiatry. 2015;27(6):525-539. doi:10.3109/09540261.2015.1085838PubMedGoogle ScholarCrossref
    ×