[Skip to Navigation]
Sign In
Figure 1.  Study Flowchart
Study Flowchart

EHR indicates electronic health record.

Figure 2.  Monthly Trends in Core Metrics for Electronic Health Record (EHR) and Inbox Time, Normalized to 8 Hours of Scheduled Clinical Time and Stratified by Turnover Status During the Study Period
Monthly Trends in Core Metrics for Electronic Health Record (EHR) and Inbox Time, Normalized to 8 Hours of Scheduled Clinical Time and Stratified by Turnover Status During the Study Period
Table 1.  Physician Characteristics Stratified by Turnover Status
Physician Characteristics Stratified by Turnover Status
Table 2.  Factors Associated With Physician Turnover in Generalized Estimation Equation Models for Longitudinal Data Among 5728 Months for 314 Nonteaching, Ambulatory Physicians
Factors Associated With Physician Turnover in Generalized Estimation Equation Models for Longitudinal Data Among 5728 Months for 314 Nonteaching, Ambulatory Physicians
1.
Shanafelt  T, Goh  J, Sinsky  C.  The business case for investing in physician well-being.   JAMA Intern Med. 2017;177(12):1826-1832. doi:10.1001/jamainternmed.2017.4340PubMedGoogle ScholarCrossref
2.
Han  S, Shanafelt  TD, Sinsky  CA,  et al.  Estimating the attributable cost of physician burnout in the United States.   Ann Intern Med. 2019;170(11):784-790. doi:10.7326/M18-1422PubMedGoogle ScholarCrossref
3.
Shanafelt  TD, West  CP, Sinsky  C,  et al.  Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2017.   Mayo Clin Proc. 2019;94(9):1681-1694. doi:10.1016/j.mayocp.2018.10.023PubMedGoogle ScholarCrossref
4.
Colwill  JM, Cultice  JM, Kruse  RL.  Will generalist physician supply meet demands of an increasing and aging population?   Health Aff (Millwood). 2008;27(3)(suppl 1):w232-w241. doi:10.1377/hlthaff.27.3.w232PubMedGoogle Scholar
5.
Dall  T, West  T, Chakrabarti  R, Iacobucci  W.  The Complexities of Physician Supply and Demand: Projections From 2013 to 2025. Association of American Medical Colleges; 2015.
6.
Hann  M, Reeves  D, Sibbald  B.  Relationships between job satisfaction, intentions to leave family practice and actually leaving among family physicians in England.   Eur J Public Health. 2011;21(4):499-503. doi:10.1093/eurpub/ckq005PubMedGoogle ScholarCrossref
7.
Shanafelt  TD, Hasan  O, Dyrbye  LN,  et al.  Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014.   Mayo Clin Proc. 2015;90(12):1600-1613. doi:10.1016/j.mayocp.2015.08.023PubMedGoogle ScholarCrossref
8.
Sinsky  CA, Dyrbye  LN, West  CP, Satele  D, Tutty  M, Shanafelt  TD.  Professional satisfaction and the career plans of US physicians.   Mayo Clin Proc. 2017;92(11):1625-1635. doi:10.1016/j.mayocp.2017.08.017PubMedGoogle ScholarCrossref
9.
Hamidi  MS, Bohman  B, Sandborg  C,  et al.  Estimating institutional physician turnover attributable to self-reported burnout and associated financial burden: a case study.   BMC Health Serv Res. 2018;18(1):851. doi:10.1186/s12913-018-3663-zPubMedGoogle ScholarCrossref
10.
Willard-Grace  R, Knox  M, Huang  B, Hammer  H, Kivlahan  C, Grumbach  K.  Burnout and health care workforce turnover.   Ann Fam Med. 2019;17(1):36-41. doi:10.1370/afm.2338PubMedGoogle ScholarCrossref
11.
Shanafelt  TD, Mungo  M, Schmitgen  J,  et al.  Longitudinal study evaluating the association between physician burnout and changes in professional work effort.   Mayo Clin Proc. 2016;91(4):422-431. doi:10.1016/j.mayocp.2016.02.001PubMedGoogle ScholarCrossref
12.
Arndt  BG, Beasley  JW, Watkinson  MD,  et al.  Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations.   Ann Fam Med. 2017;15(5):419-426. doi:10.1370/afm.2121PubMedGoogle ScholarCrossref
13.
Tai-Seale  M, Olson  CW, Li  J,  et al.  Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine.   Health Aff (Millwood). 2017;36(4):655-662. doi:10.1377/hlthaff.2016.0811PubMedGoogle ScholarCrossref
14.
Young  RA, Burge  SK, Kumar  KA, Wilson  JM, Ortiz  DF.  A time-motion study of primary care physicians’ work in the electronic health record era.   Fam Med. 2018;50(2):91-99. doi:10.22454/FamMed.2018.184803PubMedGoogle ScholarCrossref
15.
Sinsky  C, Colligan  L, Li  L,  et al.  Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties.   Ann Intern Med. 2016;165(11):753-760. doi:10.7326/M16-0961PubMedGoogle ScholarCrossref
16.
Nguyen  OT, Jenkins  NJ, Khanna  N,  et al.  A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being.   J Am Med Inform Assoc. 2021;28(5):974-984. doi:10.1093/jamia/ocaa339PubMedGoogle ScholarCrossref
17.
Melnick  ER, Dyrbye  LN, Sinsky  CA,  et al.  The association between perceived electronic health record usability and professional burnout among US physicians.   Mayo Clin Proc. 2020;95(3):476-487. doi:10.1016/j.mayocp.2019.09.024PubMedGoogle ScholarCrossref
18.
Shanafelt  TD, Dyrbye  LN, Sinsky  C,  et al.  Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction.   Mayo Clin Proc. 2016;91(7):836-848. doi:10.1016/j.mayocp.2016.05.007PubMedGoogle ScholarCrossref
19.
Tai-Seale  M, Dillon  EC, Yang  Y,  et al.  Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records.   Health Aff (Millwood). 2019;38(7):1073-1078. doi:10.1377/hlthaff.2018.05509PubMedGoogle ScholarCrossref
20.
Adler-Milstein  J, Zhao  W, Willard-Grace  R, Knox  M, Grumbach  K.  Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians.   J Am Med Inform Assoc. 2020;27(4):531-538. doi:10.1093/jamia/ocz220PubMedGoogle ScholarCrossref
21.
Adler-Milstein  J, Adelman  JS, Tai-Seale  M, Patel  VL, Dymek  C.  EHR audit logs: a new goldmine for health services research?   J Biomed Inform. 2020;101:103343. doi:10.1016/j.jbi.2019.103343PubMedGoogle Scholar
22.
Melnick  ER, Ong  SY, Fong  A,  et al.  Characterizing physician EHR use with vendor derived data: a feasibility study and cross-sectional analysis.   J Am Med Inform Assoc. 2021;28(7):1383-1392. doi:10.1093/jamia/ocab011PubMedGoogle ScholarCrossref
23.
Melnick  ER, Sinsky  CA, Krumholz  HM.  Implementing measurement science for electronic health record use.   JAMA. 2021;325(21):2149-2150. doi:10.1001/jama.2021.5487PubMedGoogle ScholarCrossref
24.
Sinsky  CA, Rule  A, Cohen  G,  et al.  Metrics for assessing physician activity using electronic health record log data.   J Am Med Inform Assoc. 2020;27(4):639-643. doi:10.1093/jamia/ocz223PubMedGoogle ScholarCrossref
25.
Micek  MA, Arndt  B, Tuan  W-J,  et al.  Physician burnout and timing of electronic health record use.   ACI Open. 2020;04(01):e1-e8. doi:10.1055/s-0039-3401815Google Scholar
26.
Willard-Grace  R, Hessler  D, Rogers  E, Dubé  K, Bodenheimer  T, Grumbach  K.  Team structure and culture are associated with lower burnout in primary care.   J Am Board Fam Med. 2014;27(2):229-238. doi:10.3122/jabfm.2014.02.130215PubMedGoogle ScholarCrossref
27.
Shaw  JG, Brown-Johnson  C, Chan  G, Mahoney  M, Winget  M.  Physician burnout and joy of practice: early mixed-methods findings from the implementation of Stanford Primary Care 2.0.   J Gen Intern Med. 2017;32(2):S272. Accessed September 8, 2021. https://link.springer.com/article/10.1007/s11606-017-4028-8Google Scholar
28.
Contratto  E, Romp  K, Estrada  CA, Agne  A, Willett  LL.  Physician order entry clerical support improves physician satisfaction and productivity.   South Med J. 2017;110(5):363-368.PubMedGoogle Scholar
29.
Smith  CD, Balatbat  C, Corbridge  S,  et al.  Implementing optimal team-based care to reduce clinician burnout.   NAM Perspectives. 2018. doi:10.31478/201809cGoogle Scholar
30.
Lyon  C, English  AF, Chabot Smith  P.  A team-based care model that improves job satisfaction.   Fam Pract Manag. 2018;25(2):6-11.PubMedGoogle Scholar
31.
Sinsky  CA, Bodenheimer  T.  Powering-up primary care teams: advanced team care with in-room support.   Ann Fam Med. 2019;17(4):367-371. doi:10.1370/afm.2422PubMedGoogle ScholarCrossref
32.
Smith  PC, Lyon  C, English  AF, Conry  C.  Practice transformation under the University of Colorado’s primary care redesign model.   Ann Fam Med. 2019;17(suppl 1):S24-S32. doi:10.1370/afm.2424PubMedGoogle ScholarCrossref
33.
McPeek-Hinz  E, Boazak  M, Sexton  JB,  et al.  Clinician burnout associated with sex, clinician type, work culture, and use of electronic health records.   JAMA Netw Open. 2021;4(4):e215686. doi:10.1001/jamanetworkopen.2021.5686PubMedGoogle Scholar
34.
Olson  K, Sinsky  C, Rinne  ST,  et al.  Cross-sectional survey of workplace stressors associated with physician burnout measured by the Mini-Z and the Maslach Burnout Inventory.   Stress Health. 2019;35(2):157-175. doi:10.1002/smi.2849PubMedGoogle ScholarCrossref
35.
Gardner  RL, Cooper  E, Haskell  J,  et al.  Physician stress and burnout: the impact of health information technology.   J Am Med Inform Assoc. 2019;26(2):106-114. doi:10.1093/jamia/ocy145PubMedGoogle ScholarCrossref
36.
Baxter  SL, Apathy  NC, Cross  DA, Sinsky  C, Hribar  MR.  Measures of electronic health record use in outpatient settings across vendors.   J Am Med Inform Assoc. 2021;28(5):955-959. doi:10.1093/jamia/ocaa266PubMedGoogle ScholarCrossref
37.
Sinsky  C.  Advancing practice science with electronic health record use data.   JAMA Intern Med. 2021;181(2):260-261. doi:10.1001/jamainternmed.2020.7068PubMedGoogle ScholarCrossref
Original Investigation
Health Informatics
October 12, 2021

Analysis of Electronic Health Record Use and Clinical Productivity and Their Association With Physician Turnover

Author Affiliations
  • 1Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut
  • 2Department of Biostatistics, Yale School of Public Health, New Haven, Connecticut
  • 3MedStar Health National Center for Human Factors in Healthcare, Columbia, Maryland
  • 4Northeast Medical Group, Yale–New Haven Health, Stratford, Connecticut
  • 5American Medical Association, Chicago, Illinois
  • 6Department of Urology, MedStar Health, Washington, DC
  • 7MedStar Health Research Institute, Hyattsville, Maryland
  • 8Department of Psychiatry, Georgetown University Medical Center, Washington, DC
JAMA Netw Open. 2021;4(10):e2128790. doi:10.1001/jamanetworkopen.2021.28790
Key Points

Question  Can vendor-derived electronic health record (EHR) use data platforms be used to understand and explain physician turnover?

Findings  In this cohort study of 314 physicians in a large ambulatory care network, physician productivity and EHR use metrics were associated with physician departure. Counterintuitively, less time spent on the EHR (in particular inbox management) was associated with physician departure.

Meaning  These findings suggest that prospectively tracking these metrics along with standardizing metrics across vendor products could identify physicians at high risk of departure who would benefit from early, team-based, targeted interventions.

Abstract

Importance  Physician turnover takes a heavy toll on patients, physicians, and health care organizations. Survey research has established associations of electronic health record (EHR) use with professional burnout and reduction in professional effort, but these findings are subject to response fatigue and bias.

Objective  To evaluate the association of physician productivity and EHR use patterns, as determined by vendor-derived EHR use data platforms, with physician turnover.

Design, Setting, and Participants  This retrospective cohort study was conducted among nonteaching ambulatory physicians at a large ambulatory practice network based in New England. Data were collected from March 2018 to February 2020.

Main Outcomes and Measures  Physician departure from the practice network; 4 time-based core measures of EHR use, normalized to 8 hours of scheduled clinical time; teamwork, percentage of a physician’s orders that are placed by other members of the care team; and productivity measures of patient volume, intensity, and demand.

Results  Among 335 physicians assessed for eligibility, 314 unique physicians (89.2%) were included in the analysis (123 [39%] women; 100 [32%] aged 45-54 years), with 5663 physician-months of data. The turnover rate was 5.1%/year (32 of 314 physicians). Physicians completed a mean 2.6 appointments/hour (95% CI, 2.5-2.6 appointments/hour) and 206 appointments/month (95% CI, 197-215 appointments/month) with 5.5 hours (95% CI, 5.3-5.8 hours) of EHR time for every 8 hours of scheduled patient time. After controlling for gender, medical specialty, and time, the following variables were associated with turnover: inbox time (odds ratio [OR], 0.70; 95% CI, 0.61-0.82; P < .001), teamwork (OR, 0.68; 95% CI, 0.52-0.87; P = .003), demand (ie, proportion of available appointments filled: OR, 0.49; 95% CI, 0.35-0.70; P < .001), and age 45 to 54 years vs 25 to 34 years (OR, 0.19; 95% CI, 0.04-0.93; P = .04).

Conclusions and Relevance  In this study, physician productivity and EHR use metrics were associated with physician departure. Prospectively tracking these metrics could identify physicians at high risk of departure who would benefit from early, team-based, targeted interventions. The counterintuitive finding that less time spent on the EHR (in particular inbox management) was associated with physician departure warrants further investigation.

Introduction

Physician turnover disrupts patients’ continuity of care, the lives of physicians and their families, and strains health care organizations’ culture, climate, and finances. Physician replacement costs to a health care organization (including recruitment, lost revenue, onboarding, and time to return to optimal efficiency) have been quantified at as much as $1 million per physician departure.1 This translates to approximately $4.6 billion annually at the national level.2 These costs are particularly worrisome given high rates of physician burnout3 and a projected shortfall of 35 000 to 90 000 physicians by 2025.4,5

Cross-sectional analyses have established associations between professional satisfaction, burnout, intention to leave,6-10 and actual reduction in professional effort.11 Given excessive physician time spent on electronic health record (EHR) activities12-15 and its association with professional burnout,16-18 it is not surprising that physician dissatisfaction with the EHR has been associated with intention to reduce clinical work in the next 12 months (in national sample of 6880 physicians: OR, 1.44; 95% CI, 1.16-1.80; P = .001) and to leave one’s current position in the next 24 months (OR, 1.57; 95% CI, 1.27-1.93; P < .001).8 In particular, EHR inbox message volume has been strongly associated with physician burnout.19,20

Survey research is subject to the limitations of self-report, including response fatigue and bias. EHR audit log data has the potential to overcome these limitations by providing an automated, objective data source for measuring physician productivity and EHR use.21,22 Recognizing that physician departure is multifactorial (ie, attributed to reasons that are personal and financial and related to work culture, work climate, workload, and so on), we assume an association exists between physician workload and departure. Given this assumption, reliable measurement of physician productivity and EHR use could help to identify physicians at increased risk of departure.23 In 2020, 7 core EHR use metrics were proposed to standardize ambulatory physician EHR use measurement normalized to 8 hours of scheduled patient time.24 Five of these metrics could be implemented for ambulatory, nonteaching physicians in 2 health care systems operating different EHR vendor products.22

To our knowledge, no previous studies have examined physician productivity and EHR use patterns and their association with physician turnover. Therefore, we conducted a study of ambulatory physicians’ turnover across an ambulatory care delivery network over a 2-year period to determine the association of physicians’ productivity and EHR use patterns with departure. Based on their positive association with physician burnout,19,20,25 we hypothesized that increased time on EHR activities (in particular inbox management) would be associated with higher rates of physician departure.

Methods
Study Design and Setting

This retrospective cohort study of ambulatory physician turnover examined physician turnover, productivity, and EHR use from March 2018 to February 2020 in a large ambulatory practice network in New England. Given known challenges implementing core EHR use metrics with vendor-derived data in the previous implementation of the 7 core metrics for ambulatory physician EHR use,22 the study cohort was limited to the ambulatory division of a practice network that employs ambulatory-based physicians who are out of training and have no teaching responsibilities. The network includes 141 practice sites in Connecticut, New York, and Rhode Island and operates on a single installation of the Epic EHR (Epic Systems). The study protocol was approved by Yale University’s institutional review board. Informed consent was waived, as all study data were deidentified, no private health information was collected, and obtaining consent would jeopardize the scientific validity of the findings given the likelihood of introducing participation bias. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines for cohort studies.

Participants

All nontrainee ambulatory physicians in the practice network were eligible for inclusion in the analysis. The inclusion criteria from the previously reported feasibility study (>30 scheduled ambulatory clinical hours)22 were applied to each month’s data to best approximate which physicians worked exclusively in the ambulatory setting without teaching duties. These inclusion criteria also inherently exclude physicians decreasing their clinical effort in preparation to depart their position. To identify EHR use patterns associated with departure in time for future interventions, we excluded all data within 3 months prior of departure. Physicians with missing data for variables in the final model were excluded from the analysis.

Data Sources

The primary data source for this study was EHR metadata from vendor-derived data platforms. EHR use data were collected from the Epic Signal platform, and scheduling data were collected from the Epic Clarity database. Signal tracks EHR user actions using a 5-second latency period, ie, once the user is idle for 5 seconds, tracking pauses until user actions resume. User actions are defined as any mouse action (ie, click, movement, scroll) or keyboard action (ie, keystrokes). This 5-second latency period was internally validated by Epic with a time-motion analysis (Provider Efficiency Technical Services team, email, April 6, 2020). A similar time-motion analysis has been performed on other Epic event logs.12 It is important to note that time on EHR tasks away from the computer will not reliably be captured, eg, a telephone call to patient to resolve an inbox message.

Physician demographic data, including termination status and departure date, were obtained from human resources rosters. These rosters did not include a specific reason for departure (ie, we are unable to determine whether a physician left their position to practice in another location, to leave the profession of medicine, or to retire).

Measurements and Outcomes

Physician characteristics included age, gender, medical specialty, and termination date (if the physician departed their position during the study period). The primary outcome was physician turnover (ie, leaving one’s position in the practice network during the study period). To control for temporal trends in EHR use (eg, physician-level variability in EHR proficiency or efficiency over time), each month of data for each physician was assigned a monthly time trend variable corresponding to that physician’s study month number.

Physician productivity measures were analyzed to account for differences in clinical workload in terms of patient volume, intensity of work, and demand for a physician’s services. Patient volume was defined as the number of completed appointments per month. Intensity was defined as the number of completed appointments per scheduled hour of work. Demand was defined as the percentage of available patient time that was scheduled with appointments.

EHR use measures included the 5 core EHR use metrics that were feasible for implementation in vendor-derived EHR data platforms: total EHR time, normalized to 8 hours of scheduled patient time; work outside of scheduled clinical hours, normalized to 8 hours of scheduled patient time; encounter note documentation time, normalized to 8 hours of scheduled patient time; active time on inbox, normalized to 8 hours of scheduled patient time; and teamwork for orders, the percentage of a physician’s orders that are placed by other members of the care team.22,24 This final non–time-based metric was specifically proposed due to the extensive literature that team-based care can reduce professional burnout.26-32

Statistical Analysis

Data were summarized using standard descriptive statistics for continuous variables and frequencies and percentages for categorical variables. The longitudinal data set consisted of monthly measures for each physician. Bivariate analyses were conducted using 2-sample t tests; associations for continuous variables and χ2, 2-sample proportions tests for categorical variables were used for cross-sectional data, such as physician characteristics or to compare the means of the mean values for all metrics at the physician level. Mean differences between groups based on repeated measures were also calculated by departure status using linear mixed models to adjust for within-physician association over time. To account for within-physician association due to the multiple observations for each physician, the associations between departure status, physician productivity, and EHR use patterns were examined on a monthly basis for each physician using generalized estimation equations (GEEs) for panel data (xtgee in Stata; family: binomial; link function: logit, exchangeable correlation structure) that allowed robust standard errors. Two models were built to model physician turnover as a function of EHR and productivity metrics: 1 using total EHR time as a variable of primary interest and another using time on inbox, note documentation, and work outside scheduled clinical hours, given that total EHR time is an approximate combination of these 3 metrics. Both models were adjusted by age, gender, physician specialty, and study month. Specialties were grouped into 3 categories: primary care, medical specialties, and surgical specialties. Since both patient volume and intensity include completed appointments per month, we chose to control for intensity in the final model (instead of volume alone) to control for additional dimensions of productivity, thereby potentially explaining more variance in physician turnover that is due to non-EHR factors. Variation in physician turnover that could be explained by the models (McKelvey and Zavoina R2) was computed using multivariable logistic regression models adjusted for correlated physician-level data (clustered by physician) since GEE did not provide a similar goodness-of-fit statistic. To obtain physician-level indicators of the estimating power of the models, area under the curve (AUC), positive and negative predictive values, sensitivity, and specificity were computed based on cross-sectional logistic regression models using means of the physicians across time for EHR metrics and productivity variables. The mean values of the total EHR time and time on inbox per month across physicians starting with the first month for each physician were illustrated using Lowess curves. All statistical analyses were conducted in Stata version 15 (StataCorp). Statistical significance was set at P < .05, and all tests were 2-tailed.

Results

Among 335 physicians assessed for eligibility, 314 unique physicians (89.2%) met criteria and were included in the analysis, with a corresponding 5728 physician-months of data (Figure 1; Table 1); 123 (39%) were women, 100 (32%) were aged 45 to 54 years, and the turnover rate was 5.1%/year (10.2% total; 32 physicians departed their position during the study period). Six physicians were excluded due to missing data for the variables included in the multivariable models. The mean values for physician productivity were as follows: demand, 77% (95% CI, 75%-79%); patient volume, 206 completed appointments/month (95% CI, 197-215 appointments/month); and intensity, 2.6 patients/hour (95% CI, 2.5-2.6 patients/hour). Mean EHR use values were as follows: total EHR time, 5.5 hours (95% CI, 5.3-5.8 hours); note documentation, 1.9 hours (95% CI, 1.7-2.0 hours); work outside scheduled clinical hours, 0.8 hours (95% CI, 0.8-0.9 hours); time on inbox, 0.70 hours (95% CI, 0.66-0.74 hours); and teamwork, 23% (95% CI, 21%-25%).

A wide range of ambulatory medical specialties were represented in the analysis, with a wide range of departure rates. The specialties with the highest turnover rates were other medical subspecialties (8 of 46 [17.4%]), surgical specialties (4 of 25 [16.0%]), gastroenterology (2 of 17 [11.8%]), and internal medicine (13 of 117 [11.1%]). In bivariate analyses, the following variables were associated with physician turnover: demand (difference between departed and remaining physicians, −0.12; 95% CI, −0.18 to −0.05; P < .001) and time on inbox (difference between departed and remaining physicians, −0.2; 95% CI, −0.3 to −0.04; P = .01) (Table 1). Monthly trends for total EHR time and time on inbox are presented in Figure 2, stratified by turnover status during the study period.

After controlling for physician age, gender, medical specialty, and study month in the generalized estimation equations model (Table 2), the following variables were associated with physician departure in the main model: time on inbox (odds ratio [OR], 0.78; 95% CI, 0.68-0.90; P = .001), teamwork (OR, 0.61; 95% CI, 0.46-0.82; P = .001), demand (OR, 0.56; 95% CI, 0.41-0.75; P < .001), and age 45 to 54 years vs 25 to 34 years (OR, 0.19; 95% CI, 0.04-0.95; P = .04). In the secondary model, total EHR time was also associated with physician turnover, with an adjusted OR of 0.96 (95% CI, 0.93-0.99; P = .01). McKelvey and Zavoina R2 was 0.35 for the main model. Adding EHR use metrics increased R2 from 0.21 to 0.35, indicating that the model accounts for 35% of variation in turnover and that EHR metrics accounted for 14% of variation in turnover. The estimating power was calculated as an AUC of 0.82, with a positive predictive value of 66.7%, negative predictive value of 90.9%, sensitivity of 12.5%, and specificity of 99.3%.

Discussion
Key Results

Despite continued limitations to vendor-derived EHR use data platforms, the findings from this cohort study of physician turnover, productivity, and EHR use, as assessed by vendor-derived EHR use data platforms in a large ambulatory care network, indicate that physician productivity and EHR use metrics were associated with departure. Physician age, gender, and specialty were not associated with departure in bivariate or multivariate analyses. After controlling for these factors, less time on the EHR and inbox were both associated with physician departure. The direction of the association between these time-based EHR use metrics and physician departure was opposite of the hypothesized direction based on assumptions about the association between EHR usability and physician burnout. Lower rates of teamwork on order entry were associated with physician departure, indicating that teamwork may help to prevent physician turnover. This finding is consistent with evidence that team-based care can reduce professional burnout and optimize team performance.26-32 Taken together, these findings suggest that a refined, prospective model of vendor-derived EHR data could help to identify physicians at high risk of departure who might benefit from targeted team-based care interventions. Indeed, low demand for a physician’s service, time on the EHR and inbox, and rates of teamwork on orders may very well be lead indicators for physicians preparing to leave practice.

Interpretation

Using vendor-derived EHR data to explore physician departure offers several advantages over traditional survey research. Vendor-derived EHR data are more objective, more available (given their automated collection without the burden of survey completion on the population being surveyed), and potentially more representative of the population being studied (since they are not subject to the response bias inherent in survey research). Even though we excluded the last 3 months of data for physicians who left their positions, the physician productivity metrics found to be associated with departure in this analysis are consistent with expectations that physicians leaving their position wind down their practice in anticipation of departure or may leave their position if there is not demand for their services. Physicians practicing in a productivity-based compensation network (like that studied here) who find themselves with consistently low patient volumes may choose to leave the network due to lower than expected income.

EHR use metric values for physicians who left their positions are more counterintuitive. If physicians who leave are in less demand or winding down their practice, they may have less EHR work to do or may be able to complete more of their EHR work during scheduled clinical hours. This could falsely decrease their EHR use time regardless of their efficiency or proficiency with EHR-related work. Alternatively, physicians who leave their practice could be more proficient with the EHR and, therefore, more marketable to move to a new position. This would suggest that more proficient EHR users are at an increased likelihood for departure. However, we are unable to conclude from this study any more details of the EHR proficiency of physicians who departed. Future studies could more definitively assess the association of burnout and departure with EHR use by linking survey data with EHR use and productivity metrics with more detailed information on departure. Furthermore, incorporating more detailed analysis of non–time-based EHR use patterns (eg, wrong-patient orders, number of clicks per patient) could help to better determine whether physicians who leave practice and spend less time on EHR work are indeed more proficient than their peers who stay in practice.23

Despite the effect size of inbox time and teamwork for orders, these metrics’ comparatively smaller R2 indicate that physician characteristics and productivity are likely more strongly associated with physician turnover than EHR use. This finding is consistent with a survey study of 1310 ambulatory clinicians, which found that work culture explained more variation in burnout than the EHR (R2, 17.6% vs 1.3%; P < .001).33 Similarly, physician-perceived EHR usability explained only 5.8% of burnout.17 Although 2 recent physician surveys reported that excessive EHR time was the most prevalent stressor associated with burnout (OR, 1.99; 95% CI, 1.21-3.27)34 and that physicians who were frustrated with their EHR had 2.4 times higher odds of burnout (95% CI, 1.6-3.7; P < .001),35 the findings from the present analysis suggest either a smaller contribution of the EHR to physician turnover or a more complex relationship between EHR use, burnout, and attrition. The counterintuitive direction of the association of time spent on EHR activities and physician departure warrants further investigation.

Additional future research could prospectively track physician productivity and EHR use patterns to identify physicians at risk of departure, thereby potentially allowing practice leaders to intervene and retain physicians at the highest risk of departure. Additional possibilities include strengthening model performance by: (1) extending study duration to allow earlier identification of physicians at high risk of departure in time for more robust retention attempts, (2) exploring additional practice dimensions that might be associated with departure, and (3) focusing model outcomes on reduction in professional effort or specific reasons for departure (ie, moving jobs, retiring, or leaving the profession). Specific practice dimensions worth exploring include (1) EHR dimensions related to note quality, teamwork on inbox, or inbox volume19,20; (2) panel size and complexity; (3) practice setting (including academic or telehealth); (4) medical specialty; and (5) temporal factors, such as the COVID-19 pandemic and vacation time. Additional work in this space would benefit from standardizing vendor-derived EHR data definitions in a way that is clinically relevant and important as well as qualitative exit interviews to provide further insight into existing data (eg, why physicians left their position) and the association between EHR use, burnout, and physician attrition.

Limitations

This study has limitations. The main limitation was that the primary outcome—physician departure—did not include a reason, ie, whether a physician left their position to take a new job, retire, or leave the profession of medicine altogether. Subsequently, the study findings are more likely applicable to departure in general than a specific reason. This study was also subject to limitations related to vendor-derived data quality. For example, some data definitions were changed internally by the vendor during the study period. More importantly, vendor-derived data platforms could not reliably distinguish ambulatory care context and scheduled hours for teaching attendings. This limitation is not unique to this vendor or practice network.22 However, we cannot conclude whether the findings from this analysis are generalizable vs specific to the practice settings studied here based on local differences such as culture, climate, and compensation models.

Conclusions

As the first study that we know of to use vendor-derived EHR data to model physician turnover, this study identified physician demand, inbox time, and teamwork on orders as key variables associated with physician departure. Counterintuitively, less time spent on the EHR (in particular on inbox management) was associated with physician departure and warrants further investigation. Standardizing vendor-derived data definitions (across and within vendor products) with better integration of clinical schedules23,36,37 could improve data validity and reliability. With greater data validity and reliability, future models could prospectively identify physicians at high risk of departure who would benefit from targeted interventions to improve retention.

Back to top
Article Information

Accepted for Publication: August 8, 2021.

Published: October 12, 2021. doi:10.1001/jamanetworkopen.2021.28790

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Melnick ER et al. JAMA Network Open.

Corresponding Author: Edward R. Melnick, MD, MHS, Department of Emergency Medicine, Yale School of Medicine, 464 Congress Ave, Ste 260, New Haven, CT 06519 (edward.melnick@yale.edu).

Author Contributions: Drs Melnick and Mete had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Marchalik and Mete contributed equally as co–last authors.

Concept and design: Melnick, Fong, Nath, Williams, Ratwani, Goldstein, O’Connell, Marchalik, Mete.

Acquisition, analysis, or interpretation of data: Melnick, Fong, Goldstein, Sinsky, Mete.

Drafting of the manuscript: Melnick, Fong, Nath, Marchalik, Mete.

Critical revision of the manuscript for important intellectual content: Melnick, Fong, Williams, Ratwani, Goldstein, O’Connell, Sinsky, Marchalik, Mete.

Statistical analysis: Melnick, Fong, Mete.

Obtained funding: Melnick.

Administrative, technical, or material support: Melnick, Fong, Nath, Williams, Ratwani, Goldstein, O’Connell, Marchalik.

Supervision: Melnick, Goldstein, O’Connell, Marchalik, Mete.

Conflict of Interest Disclosures: Dr Melnick reported receiving grants from the American Medical Association (AMA) during the conduct of the study and receiving grants from the National Institute on Drug Abuse outside the submitted work. Dr Fong reported receiving grants from AMA, subcontracted through Yale School of Medicine, during the conduct of the study. Dr Ratwani reported receiving grants from AMA during the conduct of the study. Dr Sinsky reported being employed by the AMA. No other disclosures were reported.

Funding/Support: This work was supported by 2 AMA Practice Transformation Initiatives (contract Nos. 36648 and 36650).

Role of the Funder/Sponsor: Dr Sinsky of the AMA participated in the design of the study, analysis and interpretation of the data, preparation, review, approval of the manuscript, and the decision to submit the manuscript for publication. The AMA had no role in conduct of the study and collection and management of the data.

Disclaimer: The opinions expressed in this article are those of the author(s) and should not be interpreted as AMA policy. The content is solely the responsibility of the authors and does not necessarily represent the official views of the AMA.

Additional Contributions: The authors would like to thank Becky Tylutki (Joint Data Analytics Team, Yale–New Haven Health System) for data collection and honest broker and Peter Kahn, MD, MPH, ThM (Yale School of Medicine), for variable selection. Neither of these individuals were compensated for their time.

References
1.
Shanafelt  T, Goh  J, Sinsky  C.  The business case for investing in physician well-being.   JAMA Intern Med. 2017;177(12):1826-1832. doi:10.1001/jamainternmed.2017.4340PubMedGoogle ScholarCrossref
2.
Han  S, Shanafelt  TD, Sinsky  CA,  et al.  Estimating the attributable cost of physician burnout in the United States.   Ann Intern Med. 2019;170(11):784-790. doi:10.7326/M18-1422PubMedGoogle ScholarCrossref
3.
Shanafelt  TD, West  CP, Sinsky  C,  et al.  Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2017.   Mayo Clin Proc. 2019;94(9):1681-1694. doi:10.1016/j.mayocp.2018.10.023PubMedGoogle ScholarCrossref
4.
Colwill  JM, Cultice  JM, Kruse  RL.  Will generalist physician supply meet demands of an increasing and aging population?   Health Aff (Millwood). 2008;27(3)(suppl 1):w232-w241. doi:10.1377/hlthaff.27.3.w232PubMedGoogle Scholar
5.
Dall  T, West  T, Chakrabarti  R, Iacobucci  W.  The Complexities of Physician Supply and Demand: Projections From 2013 to 2025. Association of American Medical Colleges; 2015.
6.
Hann  M, Reeves  D, Sibbald  B.  Relationships between job satisfaction, intentions to leave family practice and actually leaving among family physicians in England.   Eur J Public Health. 2011;21(4):499-503. doi:10.1093/eurpub/ckq005PubMedGoogle ScholarCrossref
7.
Shanafelt  TD, Hasan  O, Dyrbye  LN,  et al.  Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014.   Mayo Clin Proc. 2015;90(12):1600-1613. doi:10.1016/j.mayocp.2015.08.023PubMedGoogle ScholarCrossref
8.
Sinsky  CA, Dyrbye  LN, West  CP, Satele  D, Tutty  M, Shanafelt  TD.  Professional satisfaction and the career plans of US physicians.   Mayo Clin Proc. 2017;92(11):1625-1635. doi:10.1016/j.mayocp.2017.08.017PubMedGoogle ScholarCrossref
9.
Hamidi  MS, Bohman  B, Sandborg  C,  et al.  Estimating institutional physician turnover attributable to self-reported burnout and associated financial burden: a case study.   BMC Health Serv Res. 2018;18(1):851. doi:10.1186/s12913-018-3663-zPubMedGoogle ScholarCrossref
10.
Willard-Grace  R, Knox  M, Huang  B, Hammer  H, Kivlahan  C, Grumbach  K.  Burnout and health care workforce turnover.   Ann Fam Med. 2019;17(1):36-41. doi:10.1370/afm.2338PubMedGoogle ScholarCrossref
11.
Shanafelt  TD, Mungo  M, Schmitgen  J,  et al.  Longitudinal study evaluating the association between physician burnout and changes in professional work effort.   Mayo Clin Proc. 2016;91(4):422-431. doi:10.1016/j.mayocp.2016.02.001PubMedGoogle ScholarCrossref
12.
Arndt  BG, Beasley  JW, Watkinson  MD,  et al.  Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations.   Ann Fam Med. 2017;15(5):419-426. doi:10.1370/afm.2121PubMedGoogle ScholarCrossref
13.
Tai-Seale  M, Olson  CW, Li  J,  et al.  Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine.   Health Aff (Millwood). 2017;36(4):655-662. doi:10.1377/hlthaff.2016.0811PubMedGoogle ScholarCrossref
14.
Young  RA, Burge  SK, Kumar  KA, Wilson  JM, Ortiz  DF.  A time-motion study of primary care physicians’ work in the electronic health record era.   Fam Med. 2018;50(2):91-99. doi:10.22454/FamMed.2018.184803PubMedGoogle ScholarCrossref
15.
Sinsky  C, Colligan  L, Li  L,  et al.  Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties.   Ann Intern Med. 2016;165(11):753-760. doi:10.7326/M16-0961PubMedGoogle ScholarCrossref
16.
Nguyen  OT, Jenkins  NJ, Khanna  N,  et al.  A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being.   J Am Med Inform Assoc. 2021;28(5):974-984. doi:10.1093/jamia/ocaa339PubMedGoogle ScholarCrossref
17.
Melnick  ER, Dyrbye  LN, Sinsky  CA,  et al.  The association between perceived electronic health record usability and professional burnout among US physicians.   Mayo Clin Proc. 2020;95(3):476-487. doi:10.1016/j.mayocp.2019.09.024PubMedGoogle ScholarCrossref
18.
Shanafelt  TD, Dyrbye  LN, Sinsky  C,  et al.  Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction.   Mayo Clin Proc. 2016;91(7):836-848. doi:10.1016/j.mayocp.2016.05.007PubMedGoogle ScholarCrossref
19.
Tai-Seale  M, Dillon  EC, Yang  Y,  et al.  Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records.   Health Aff (Millwood). 2019;38(7):1073-1078. doi:10.1377/hlthaff.2018.05509PubMedGoogle ScholarCrossref
20.
Adler-Milstein  J, Zhao  W, Willard-Grace  R, Knox  M, Grumbach  K.  Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians.   J Am Med Inform Assoc. 2020;27(4):531-538. doi:10.1093/jamia/ocz220PubMedGoogle ScholarCrossref
21.
Adler-Milstein  J, Adelman  JS, Tai-Seale  M, Patel  VL, Dymek  C.  EHR audit logs: a new goldmine for health services research?   J Biomed Inform. 2020;101:103343. doi:10.1016/j.jbi.2019.103343PubMedGoogle Scholar
22.
Melnick  ER, Ong  SY, Fong  A,  et al.  Characterizing physician EHR use with vendor derived data: a feasibility study and cross-sectional analysis.   J Am Med Inform Assoc. 2021;28(7):1383-1392. doi:10.1093/jamia/ocab011PubMedGoogle ScholarCrossref
23.
Melnick  ER, Sinsky  CA, Krumholz  HM.  Implementing measurement science for electronic health record use.   JAMA. 2021;325(21):2149-2150. doi:10.1001/jama.2021.5487PubMedGoogle ScholarCrossref
24.
Sinsky  CA, Rule  A, Cohen  G,  et al.  Metrics for assessing physician activity using electronic health record log data.   J Am Med Inform Assoc. 2020;27(4):639-643. doi:10.1093/jamia/ocz223PubMedGoogle ScholarCrossref
25.
Micek  MA, Arndt  B, Tuan  W-J,  et al.  Physician burnout and timing of electronic health record use.   ACI Open. 2020;04(01):e1-e8. doi:10.1055/s-0039-3401815Google Scholar
26.
Willard-Grace  R, Hessler  D, Rogers  E, Dubé  K, Bodenheimer  T, Grumbach  K.  Team structure and culture are associated with lower burnout in primary care.   J Am Board Fam Med. 2014;27(2):229-238. doi:10.3122/jabfm.2014.02.130215PubMedGoogle ScholarCrossref
27.
Shaw  JG, Brown-Johnson  C, Chan  G, Mahoney  M, Winget  M.  Physician burnout and joy of practice: early mixed-methods findings from the implementation of Stanford Primary Care 2.0.   J Gen Intern Med. 2017;32(2):S272. Accessed September 8, 2021. https://link.springer.com/article/10.1007/s11606-017-4028-8Google Scholar
28.
Contratto  E, Romp  K, Estrada  CA, Agne  A, Willett  LL.  Physician order entry clerical support improves physician satisfaction and productivity.   South Med J. 2017;110(5):363-368.PubMedGoogle Scholar
29.
Smith  CD, Balatbat  C, Corbridge  S,  et al.  Implementing optimal team-based care to reduce clinician burnout.   NAM Perspectives. 2018. doi:10.31478/201809cGoogle Scholar
30.
Lyon  C, English  AF, Chabot Smith  P.  A team-based care model that improves job satisfaction.   Fam Pract Manag. 2018;25(2):6-11.PubMedGoogle Scholar
31.
Sinsky  CA, Bodenheimer  T.  Powering-up primary care teams: advanced team care with in-room support.   Ann Fam Med. 2019;17(4):367-371. doi:10.1370/afm.2422PubMedGoogle ScholarCrossref
32.
Smith  PC, Lyon  C, English  AF, Conry  C.  Practice transformation under the University of Colorado’s primary care redesign model.   Ann Fam Med. 2019;17(suppl 1):S24-S32. doi:10.1370/afm.2424PubMedGoogle ScholarCrossref
33.
McPeek-Hinz  E, Boazak  M, Sexton  JB,  et al.  Clinician burnout associated with sex, clinician type, work culture, and use of electronic health records.   JAMA Netw Open. 2021;4(4):e215686. doi:10.1001/jamanetworkopen.2021.5686PubMedGoogle Scholar
34.
Olson  K, Sinsky  C, Rinne  ST,  et al.  Cross-sectional survey of workplace stressors associated with physician burnout measured by the Mini-Z and the Maslach Burnout Inventory.   Stress Health. 2019;35(2):157-175. doi:10.1002/smi.2849PubMedGoogle ScholarCrossref
35.
Gardner  RL, Cooper  E, Haskell  J,  et al.  Physician stress and burnout: the impact of health information technology.   J Am Med Inform Assoc. 2019;26(2):106-114. doi:10.1093/jamia/ocy145PubMedGoogle ScholarCrossref
36.
Baxter  SL, Apathy  NC, Cross  DA, Sinsky  C, Hribar  MR.  Measures of electronic health record use in outpatient settings across vendors.   J Am Med Inform Assoc. 2021;28(5):955-959. doi:10.1093/jamia/ocaa266PubMedGoogle ScholarCrossref
37.
Sinsky  C.  Advancing practice science with electronic health record use data.   JAMA Intern Med. 2021;181(2):260-261. doi:10.1001/jamainternmed.2020.7068PubMedGoogle ScholarCrossref
×