Visit and Between-Visit Interaction Frequency Before and After COVID-19 Telehealth Implementation

Key Points Question How did use of visits and between-visit interactions change from April 2019 to March 2021, and did differences by age, race and ethnicity, language, or socioeconomic status that existed before the COVID-19 pandemic continue after COVID-19–related telehealth implementation? Findings In this cohort study of 15 148 patients with diabetes, preexisting growth in total encounters persisted after COVID-19–related implementation, driven by growth in between-visit interactions. Pre–COVID-19 differences in visits by patient characteristics decreased, but differences in between-visit interactions persisted. Meaning This study found that telehealth adoption affected utilization of both visits and between-visit interactions, with variation by patient characteristics; future research should explore the impact of growing between-visit interactions on equity, outcomes, and experience.


Introduction
0][11][12] The impact of increased telehealth adoption on outpatient patient-clinician interactions broadly among patients with chronic diseases remains unknown.
4][15][16] Visits occur in-person or remotely by video or telephone and are typically scheduled in advance and billed.They occur most often between patients and physicians or advance practice practitioners (APPs) but also occur with nonbilling health care team members, such as nurses.6][7][8][21][22][23][24][25][26][27][28][29][30][31][32] Few studies have accounted for between-visit interactions and visits with nonbilling clinicians, likely due to reliance on payer data, which excludes most of these encounters. 3,21,24,31,33,34To gain a better understanding of changes in patterns of care for patients with chronic diseases, it is imperative to evaluate both visits and between-visit interactions. 184,27 Close attention to these distinct periods is needed because while telehealth visits substituted in-person care during the shelter-in-place period, both telehealth visits and in-person visits were available during the hybrid period.Moreover, given the wide variation across health systems in providing video or telephone-only telehealth visits, inclusion of safety-net settings, which often provide more telephone visits, in telehealth evaluations is particularly pressing. 5,6,21,22,35,36fety-net systems serve a disproportionate share of disadvantaged patients who face barriers to digital health uptake.[39] To our knowledge, no studies have assessed the impact of PHE-related telehealth implementation on the quantity of patient-health care team encounters overall.In this study, we evaluate rates of change in both visits and between-visit interactions among patients with diabetes with multiple health care team members in 2 health systems with different telehealth implementations.We examine these rates of change across 3 time periods (pre-PHE, shelter-inplace, and hybrid) to understand, after these abrupt changes to care delivery, the sustained rates of change in care utilization in the hybrid period.1][42] We also compare patterns of differences by patient characteristics known to influence access to digital health care (age, preferred language, race and ethnicity, and neighborhood socioeconomic status [nSES]) 43 in the pre-PHE vs hybrid periods.

Study Setting and Data Source
Using electronic health record (EHR) data, we conducted a 2-year retrospective cohort study (April

Study Period
We evaluated

Study Sample
In each health system, we identified a cohort of patients (Ն18 years old) with diabetes and empaneled in primary care as of April 1, 2019.We used EHR data to identify cohorts from problem lists, disease registries, laboratory results, and/or medications (eMethods in Supplement 1).To ensure patients were actively receiving care, we excluded those with no encounters from April 1, 2019, to March 31, 2021, with their primary care or endocrinology teams (hereafter jointly referred to as "health care team").Encounters with the following team members were included: billing clinicians (physicians, APPs); nurses; social workers; pharmacists; dietitians; medical assistants; and embedded behavioral health clinicians.

Variables
Healthcare utilization was measured at the patient level as mean number of encounters per patient per month, categorized into 4 outcomes (A to D) (eTable 1 in Supplement 1).Outcome A was total encounters, defined as any EHR-documented encounter (visit or between-visit interaction) between patients and any member of their health care team (ie, outcomes B and D combined).Visits consisted of scheduled in-person, video, or telephone appointments.Between-visit interactions consisted of all care that occurred between visits, documented in the EHR as unscheduled telephone calls or patient portal messages.Between-visit interactions can be initiated by patients or members of their health care team.A single between-visit interaction can include several back-and-forth calls or messages pertaining to the reason for the interaction (eg, one interaction could include a patient messaging to request a medication refill then the nurse forwarding the message with a comment to a doctor then the doctor replying to the patient that the medication has been refilled).Because they often involve multiple team members, we were unable to reliably identify between-visit interactions

JAMA Network Open | Health Informatics
Interaction Frequency Before and After COVID-19 Telehealth Implementation by clinician type.While patients with diabetes are advised to have a visit every 3 to 6 months (0.17-0.33 visits/patient/mo), there is no benchmark for the frequency of between-visit interactions.
Outcome B was visits with any team member.Outcome C was visits with billing clinicians only (ie, a subset of outcome B).Outcome D was all between-visit interactions.

Statistical Analysis Descriptive Statistics
Each health system cohort was analyzed separately due to differences in patient population and primary telehealth modality.We used descriptive statistics to report all variables.For each of the 4 encounter-type outcomes, we report monthly means and the overall means within the pre-PHE and hybrid-PHE periods.For visits with any team member and between-visit interactions, we also report monthly means by modality.

Interrupted Time-Series Analyses
We used an interrupted time-series model with interruptions (intercept and slope changes) at the start of PHE-SIP and hybrid-PHE to model rates of change in outcomes A to D between study periods (pre-PHE vs PHE-SIP vs hybrid-PHE).While initial changes in care utilization at the start of the pandemic have been well documented, we chose an interrupted time-series model rather than an alternative analysis (eg, difference in differences) to focus our evaluation on sustained rates of change (slopes) in care utilization during the hybrid-PHE period.For each outcome A to D in the interrupted time-series analysis, we conducted Poisson regression models using generalized estimating equations with robust standard errors clustered on person (eMethods in Supplement 1).
We used a multiple imputation by chained equations procedure (R version 4.

[R Project for
Statistical Computing]) to address missing values for baseline hemoglobin A 1c and BP measurements. 44Final models included all key patient-level independent variables and all covariates as well as slope changes at each interruption and statistically significant interaction terms between key independent variables and the PHE-SIP and hybrid-PHE intercept change variables (determined using Wald test with threshold of P < .05).We used model coefficients and 95% CIs to estimate the rate of change in encounters (slope) in the pre-PHE and hybrid-PHE periods and the magnitude of change (slope change) between the rates in these 2 periods.

Use in Pre-PHE vs PHE-Hybrid Periods by Patient-Level Characteristics
To examine differences by key patient-level independent variables in the pre-PHE and PHE-hybrid periods, we generated marginal means using ggeffects and plotted them using ggplot2. 45We conducted 2-sided pairwise comparisons of the estimated marginal means using the emmeans package. 46We used the Tukey method for pairwise comparisons (threshold of .05).Given the highly unusual nature of clinical care during the PHE-SIP period, we focused pairwise comparisons on the pre-PHE and PHE-hybrid periods.

JAMA Network Open | Health Informatics
Interaction Frequency Before and After COVID-19 Telehealth Implementation
In both health systems, the increase in total encounters over time was driven largely by an increase in between-visit interactions, which comprised more than half of all encounters in both systems in most periods (79%-90% at UCSF, 48%-61% at SFHN decrease in the slope from the pre-PHE period to the hybrid-PHE period.Rates of visits with any team member and billing clinicians were increasing during the pre-PHE period (any team member: 0.9% per patient/mo; 95% CI, 0.2% to 1.6% per patient/mo; billing clinician: 1.0% per patient/mo; 95% CI, 0.3% to 1.7% per patient/mo; estimated 3 visits/1000 patients/mo for both) and were stable during the hybrid-PHE period (any team member: 0.2% per patient/mo; 95% CI, −0.4% to 0.7% per patient/mo; billing clinician: 0.1%; 95% CI, −0.5% to 0.6% per patient/mo).

Differential Utilization Across Time by Key Demographic Characteristics
Differences by key patient-level characteristics (age, race and ethnicity, language, nSES) in outcomes A to D within each period based on marginal means are presented in eTable 5 in Supplement 1.

Age
There were no significant differences by age at UCSF.At SFHN, younger individuals had fewer total encounters, visits with billing clinicians, and between-visit interactions during the pre-PHE period compared with older individuals (eFigure 8 in Supplement 1); these differences were no longer significant during the hybrid-PHE period.During the hybrid-PHE period, patients aged 50 to 64 years had more visits with any team member compared with those aged 75 years or older.

Race and Ethnicity
Patients identifying as Asian had fewer total encounters and between-visit interactions compared with those identifying as non-Hispanic White, Black, or Hispanic/Latinx during the pre-PHE period at  remained significant in the hybrid-PHE period at UCSF.At UCSF, patients identifying as Black had more visits (with billing clinicians and any team member) compared with non-Hispanic White and Asian in all periods (eFigure 9 in Supplement 1).

Language Preference
At UCSF, there were no differences by language in any outcome in the pre-PHE period (Figure 4 and eFigure 10 in Supplement 1).However, in the hybrid-PHE period, patients with a preferred language other than English, Spanish, or Chinese had fewer total encounters compared with English speakers.
At SFHN in the pre-PHE period, Chinese speakers had fewer total encounters and between-visit interactions compared with English speakers and fewer visits with billing clinicians compared with all other groups.This difference persisted for between-visit interactions in the hybrid-PHE period (0.52 [95% CI, 0.47-0.58]vs 0.61 [95% CI, 0.56-0.66]between-visit interactions/patient/mo; P = .03), but not for visits with billing clinicians.Spanish speakers had fewer between-visit interactions in the pre-PHE period compared with English speakers, but this difference was no longer significant in hybrid-PHE.

nSES
There were no differences in outcomes A to D by nSES at either site.Results appear in eTables 2 and 3 in Supplement 1.

Discussion
Despite different telehealth implementation strategies and patient populations in the academic (UCSF) and safety-net (SFHN) settings, total encounters continued to increase significantly during the hybrid period at both health systems, driven by between-visit interactions.These between-visit interactions-primarily patient messages at UCSF and a combination of telephone calls and patient messages at SFHN-are crucial to advancing health, especially when team-based care is the standard of care, but they are understudied. 18,28Within our investigation of differences in utilization by demographic characteristics, we found fewer differences in visit frequency by age and language preference during the hybrid-PHE period compared with the pre-PHE period at SFHN.In contrast, among between-visit interactions, differences persisted by language preference, decreased by age at SFHN, and decreased by race and ethnicity at UCSF.
Prior literature has shown recent growth in patient portal messages. 25,26We found that unscheduled telephone calls have also been growing, particularly in the safety-net system. 26,47Work is needed to understand the cause and impact of the growth in between-visit interactions, which address care concerns between scheduled visits. 18This growth may be because synchronous telehealth visits are inadequate and require more between-visit care, patients do not have timely access to clinician visits, or visits are inconvenient, suggesting between-visit interactions may be an important access point for patients who would otherwise forego care.9][50] To alleviate clinician burnout associated with these increased tasks, some payers have begun reimbursing some patient portal messages. 51However, similar attention has not been paid to the growing number of telephone-based between-visit interactions, nor has attention been paid to the impact of this growing between-visit workload on clinical teams more broadly.The uptake by clinicians of billing for patient portal messages remains low, similar to the past introduction of other billing codes. 51,52Billing for patient portal messages may inhibit their use due to copays, further limiting access for patients. 53Thus, while the best approach to cope with the growing number of between-visit interactions remains unclear, our findings indicate that their impact on clinician wellbeing and health outcomes merit examination.
Of particular importance to equity is our finding that utilization patterns by sociodemographic characteristics differed during the pre-PHE period compared with the hybrid-PHE period and by  23,54 We found that when a mix of in-person, video, and telephone options

Conclusions
In these 2 urban health care systems, patients with diabetes were increasingly engaging with their health care teams through between-visit patient messages or telephone calls.In contrast, visits, both with billing clinicians and with any team member, were stable during the hybrid-PHE period.Patterns of utilization by key sociodemographic characteristics differed in the hybrid-PHE period compared with the pre-PHE period, with nuanced implications for health equity.More research is needed to evaluate the drivers and impact of between-visit interactions, particularly on clinical outcomes and clinician well-being, including among nonphysician and APP members of the health care team.Our findings encourage broadening current conversations regarding the impact of telehealth on care utilization to consider how a range of encounter types can be optimized to ensure equitable access to team-based care for chronic disease management.
Figures of interrupted time-series analyses stratified by age, race and ethnicity, and language are presented in eFigures 2 to 7 in Supplement 1.

Figure 3 .
Figure 3. Adjusted Encounter Means by Race and Ethnicity in the Pre-Public Health Emergency (PHE) and Hybrid-PHE Periods Among 4967 Patients Receiving Care at University of California San Francisco (UCSF) and 8975 Patients Receiving Care at San Francisco Health Network (SFHN)

Figure 4 .
Figure 4. Adjusted Encounter Means by Language Preference in the Pre-Public Health Emergency (PHE) and Hybrid-PHE Periods Among 4967 Patients Receiving Care at University of California San Francisco (UCSF) and 8975 Patients Receiving Care at San Francisco Health Network

eTable 5 . 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 .
Estimated Marginal Means eFigure Interrupted Time-Series Analysis of Patient-Clinician Encounters per Month at UCSF (n = 4967) and SFHN (n = 8975) eFigure Interrupted Time-Series Analysis of Patient-Clinician Encounters per Month at UCSF (n = 4967) Stratified by Patient Age eFigure Interrupted Time-Series Analysis of Patient-Clinician Encounters per Month at SFHN (n = 8975) Stratified by Patient Age eFigure Interrupted Time-Series Analysis of Patient-Clinician Encounters per Month at UCSF (n = 4967) Stratified by Patient Race and Ethnicity eFigure Interrupted Time-Series Analysis of Patient-Clinician Encounters per Month at SFHN (n = 8975) Stratified by Patient Race and Ethnicity eFigure Interrupted Time-Series Analysis of Patient-Clinician Encounters per Month at UCSF (n = 4967) Stratified by Patient Language eFigure Interrupted Time-Series Analysis of Patient-Clinician Encounters per Month at SFHN (n = 8975) Stratified by Patient Language eFigure Adjusted Encounter Means by Age in Pre-PHE and Hybrid-PHE Periods at UCSF (n = 4967) and SFHN (n = 8975) eFigure Adjusted Encounter Means by Race and Ethnicity for Visits With Any Team Member in Pre-PHE and Hybrid-PHE Periods at UCSF (n = 4967) and SFHN (n = 8975) eFigure 10.Adjusted Encounter Means by Language for Visits With Any Team Member in Pre-PHE and Hybrid-PHE Periods at UCSF (n = 4967) and SFHN (n = 8975) SUPPLEMENT 2. Data Sharing Statement Interaction Frequency Before and After COVID-19 Telehealth Implementation JAMA Network Open.2023;6(9):e2333944.doi:10.1001/jamanetworkopen.2023.33944(Reprinted) September 15, 2023 2/16 Downloaded From: https://jamanetwork.com/ on 10/06/2023 1,   2019toMarch 31, 2021)of patients with diabetes (eMethods in Supplement 1).All patients received primary care at 1 of 2 health care systems: University of California San Francisco (UCSF) or the San Francisco Health Network (SFHN).UCSF, an academic tertiary care center, has 4 primary care practices.SFHN, San Francisco County's safety-net health system, has 14 primary care practices.The UCSF institutional review board approved this study.Informed consent was waived due to the low risk of the study and procedures involving only review of data collected for routine clinical care.This report follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE)

Table .
Characteristics of Included Participants hybrid-PHE: 1.13 [0.09] per mo vs 1.21 [0.10] per mo; SFHN, pre-PHE vs hybrid-PHE: 0.84 [0.09] per mo vs 1.00 [0.08] per mo).Mean (SD) visits with any team member and with billing clinicians decreased at UCSF in the pre-PHE vs hybrid-PHE periods (any team member: 0.35 [0.03] per mo vs 0.30 [0.02] per mo; billing clinicians: 0.31 [0.03] per mo vs 0.26 [0.01] per mo).Mean (SD) visits with any team member increased slightly at SFHN between the pre-PHE and hybrid-PHE periods (0.36 [0.03] per mo to 0.39 [0.03] per mo), while visits with billing clinicians remained stable.At the start of PHE-SIP, in-person visits were replaced mostly by video visits at UCSF and telephone visits at SFHN, although by the end of hybrid-PHE, in-person visits occurred at approximately the same rate as video visits at UCSF and as phone visits at SFHN (Figure b Due to small sample size, American Indian or Alaska Native and Native Hawaiian or Other Pacific Islander were grouped with other race and ethnicity in analyses.c Participants with missing data (301 [5.7%] at UCSF d Healthy Workers is a city-provided insurance for home caregivers.e f Data presented is based on values where there were no missing data.As described in the Methods section, for analyses missing values were imputed for baseline A 1c (940 [17.8%] at UCSF; 3679 [37.2%] at SFHN) and baseline blood pressure (457 [8.7%] at UCSF; 1126 [11.4%] at SFHN).month was higher in the hybrid-PHE period than in the pre-PHE period at both sites (UCSF, pre-PHE Figure 1.Mean Number of Patient-Clinician Encounters per Month Among 5268 Patients Receiving Care at University of California San Francisco (UCSF) and 9880 Patients Receiving Care at San Francisco Health Network (SFHN) From April 2019 to March 2021 1, 2019, to March 31, 2020, at UCSF; September 1, 2019, to March 31, 2020, at SFHN; PHE-SIP, public health emergency shelter in place, April 1 to June 30, 2020; and hybrid-PHE, hybrid public health emergency, July 1, 2020, to March 31, 2021.vs ). Mean (SD) between-visit interactions per patient per month increased 14% at UCSF (from 0.79 [0.07] per mo pre-PHE to 0.90 [0.09] per mo hybrid-PHE) and 23% at SFHN (from 0.48 [0.06] per mo pre-PHE to 0.59 [0.06] per mo hybrid-PHE).At UCSF (Figure 2), this growth was due to an increase in mean (SD) patient portal messages (from 0.30 [0.04] per mo pre-PHE to 0.43 [0.05] per mo hybrid-PHE).SFHN had an Figure 2. Mean Number of Patient-Clinician Encounters by Visit and Between-Visit Interaction Modality per Month Among 5268 Patients Receiving Care at University of California San Francisco (UCSF) and 9880 Patients Receiving Care at San Francisco Health Network (SFHN) From April 2019 to March 2021 At SFHN, the rate of total encounters was increasing in both the pre-PHE and hybrid-PHE periods (pre-PHE: 3.1% per patient/mo; 95% CI, 2.5% to 3.8% per patient/mo; estimated 25 encounters/1000 patients/mo; hybrid-PHE: 1.8% per patient/mo; 95% CI, 1.3% to 2.2% per patient/mo; estimated 18 encounters/1000 patients/mo) (eFigure 1 and eTable 4 in Supplement 1) although there was a 1.3% (95% CI, 0.5% to 2.1%) decrease in the slope from the pre-PHE period to the hybrid-PHE period.Similarly, the rate of between-visit interactions was increasing in both periods (pre-PHE: 4.9% per patient/mo; 95% CI, 4.1% to 5.7% per patient/mo; estimated 21 interactions/ 1000 patients/mo; hybrid-PHE 2.9% per patient/mo; 95% CI, 2.3% to 3.4% per patient/mo; estimated 17 interactions/1000 patients/mo), although there was a 1.9% (95% CI, 1.0% to 2.8%) Interaction Frequency Before and After COVID-19 Telehealth Implementation These differences persisted in the hybrid-PHE period at SFHN; specifically Asian patients had fewer monthly mean between-visit interactions compared with White patients (0.46 [95% CI, 0.42-0.50]vs 0.59 [95% CI, 0.53-0.66]between-visit interactions/patient/mo; P < .001).However, only the difference between Asian and Black patients JAMA Network Open.2023;6(9):e2333944.doi:10.1001/jamanetworkopen.2023.33944(Reprinted)September15,2023 8/16 Downloaded From: https://jamanetwork.com/ on 10/06/2023both UCSF and SFHN (Figure3).

on 10/06/2023 encounter
type.Differences in visits by race and ethnicity and preferred language that existed during the pre-PHE period disappeared during the hybrid-PHE period, except for Black patients, who continued to have more visits than other patients.Prior studies have expressed concerns about equity in telehealth access.
Downloaded From: https://jamanetwork.com/ Pre-PHE indicates April 1, 2019, to March 31, 2020, at UCSF and September 1, 2019, to March 31, 31, 2020; hybrid-PHE, July 1, 2020 to March 31, 2021.Horizontal bars indicate significant difference at P < .05(allcomparisonstested).Error bars indicate the 95% CI for the adjusted mean.wereavailable during the hybrid-PHE period, there were fewer differences in utilization by race and ethnicity and preferred language compared with when visits were primarily conducted in person, suggesting that telehealth options may reduce disparities.Adults younger than 35 years at SFHN had fewer between-visit interactions compared with older patients during the pre-PHE period, but this difference was no longer present during the hybrid-PHE period.This may represent greater engagement of younger patients via telehealth and/or barriers to telehealth utilization among older adults due to lower digital literacy.While both Chinese speakers and Spanish speakers had a lower frequency of between-visit interactions at SFHN compared with English speakers during the pre-PHE period, this difference persisted over time for Chinese speakers but disappeared for Spanish speakers during the hybrid-PHE period, possibly because language access for Chinese speakers remains limited on patient portals or when patients call health care systems. 55mitationsOur study is limited by only including 2 urban health systems and data through March 2021, which may not be generalizable to later periods during the pandemic when concerns over virus exposure decreased.In identifying our cohorts, the accuracy of EHR problem lists or medication lists is unknown; we tried to mitigate this through medical record reviews for data validation.We could not account for unmeasured confounders, such as losing access to health care due to PHE-related job loss.New EHR adoption with a new patient portal at SFHN in late 2019 may have impacted our results, though less than 5% of patients had active patient portal accounts in the prior EHR.We did not have access to data beyond our 2 health systems, so we could not account for patients having external encounters; we restricted our cohorts to patients actively receiving care at UCSF or SFHN to mitigate this.