eAppendix. Methods: Sensitivity Analyses
eTable A1. Association between EHR use and clinical events and office visits: Sensitivity analysis among continuously enrolled patients
eTable A2. Association between EHR use and clinical events and office visits: Sensitivity analysis using linear regression model with patient-level random effect
B. Results Unadjusted Descriptive Data
eFigure 1. Unadjusted monthly rate of hospitalizations during study period 2004-2009 (months 1-72) across all medical centers (N=169,711)
eFigures 2-16. Unadjusted monthly rate of hospitalizations during study period 2004-2009 (months 1-72) for each medical center (1-17)
C. Results Hospitalization for specific primary diagnosis
Table A3. Association between EHR use and Hospitalization for a Diabetes-related or Cardiovascular primary diagnoses
Reed M, Huang J, Brand R, Graetz I, Neugebauer R, Fireman B, Jaffe M, Ballard DW, Hsu J. Implementation of an Outpatient Electronic Health Record and Emergency Department Visits, Hospitalizations, and Office Visits Among Patients With Diabetes. JAMA. 2013;310(10):1060-1065. doi:10.1001/jama.2013.276733
The US federal government is spending billions of dollars in physician incentives to encourage the meaningful use of electronic health records (EHRs). Although the use of EHRs has potential to improve patient health outcomes, the existing evidence has been limited and inconsistent.
To examine the association between implementing a commercially available outpatient EHR and emergency department (ED) visits, hospitalizations, and office visits for patients with diabetes mellitus.
Design, Setting, and Population
Staggered EHR implementation across outpatient clinics in an integrated delivery system (Kaiser Permanente Northern California) between 2005 and 2008 created an opportunity for studying changes associated with EHR use. Among a population-based sample of 169 711 patients with diabetes between 2004 and 2009, we analyzed 4 997 585 person-months before EHR implementation and 4 648 572 person-months after an EHR was being used by patients’ physicians.
Main Outcomes and Measures
We examined the association between EHR use and unfavorable clinical events (ED visits and hospitalizations) and office visit use among patients with diabetes, using multivariable regression with patient-level fixed-effect analyses and adjustment for trends over time.
In multivariable analyses, use of the EHR was associated with a statistically significantly decreased number of ED visits, 28.80 fewer visits per 1000 patients annually (95% CI, 20.28 to 37.32), from a mean of 519.12 visits per 1000 patients annually without using the EHR to 490.32 per 1000 patients when using the EHR. The EHR was also associated with 13.10 fewer hospitalizations per 1000 patients annually (95% CI, 7.37 to 18.82), from a mean of 251.60 hospitalizations per 1000 patients annually with no EHR to 238.50 per 1000 patients annually when using the EHR. There were similar statistically significant reductions in nonelective hospitalizations (10.92 fewer per 1000 patients annually) and hospitalizations for ambulatory care–sensitive conditions (7.08 fewer per 1000 patients annually). There was no statistically significant association between EHR use and office visit rates.
Conclusions and Relevance
Among patients with diabetes, use of an outpatient EHR in an integrated delivery system was associated with modest reductions in ED visits and hospitalizations but not office visit rates. Further studies are needed to quantify the association of EHR use with changes in costs.
The Health Information Technology for Economic and Clinical Health Act authorizes up to $27 billion during 10 years to promote meaningful use of electronic health records (EHRs),1 with penalties for lack of EHR use beginning in 2015. With these substantial incentives, it is not surprising that EHR adoption in the United States appears to be increasing.2,3
Electronic health records increase access to timely and complete patient information at the point of care, with potential to improve the quality and efficiency of care delivered, including improved care coordination. Although EHRs have also been proposed as a mechanism for controlling health care costs, these benefits have yet to be consistently shown.4,5 With medical care for patients with chronic diseases representing 75% of US health care costs6 and hospitalizations representing one-third of all US health care expenditures,7 better management of chronic medical conditions such as diabetes represents one clinical area in which improved care theoretically could reduce spending.
There is, however, limited and mixed evidence on the effect of EHRs on health outcomes or clinical events.8- 14 Previous studies have not reported the effect of outpatient EHR use on use of health care services, including downstream unfavorable clinical events represented by emergency department (ED) visits and hospitalizations. Studies of EHR effects with rigorous design and controls are limited, and many studies have examined use of individual health information technology (IT) tools instead of the comprehensive functionality required in current federally certified complete EHRs or meeting minimum criteria for meaningful use.15
In this study, we examined the association between EHR implementation and ED visits, hospitalizations, and office visits among patients with diabetes. We took advantage of the stepwise implementation of a commercially available and federally certified complete outpatient EHR across multiple practice sites within a large integrated health care delivery system (IDS).
This study was conducted at Kaiser Permanente Northern California, a large prepaid IDS, which provides comprehensive medical care for more than 3 million members, including outpatient, inpatient, ED, pharmacy, and laboratory services.16
The Kaiser Foundation Research Institute institutional review board reviewed and approved the study protocol, waiving the requirement for informed consent.
Between 2005 and 2008, Kaiser Permanente Northern California implemented a commercially available complete outpatient EHR. The implementation schedule was staggered stepwise across outpatient clinics at facilities within medical centers.14 This provided the opportunity for a study design examining EHR use with adjustment for secular trends in diabetes care practice unrelated to the EHR. Although rollout of the certified EHR was not randomly ordered, the sequence was determined a priori and not systematically designed according to the medical centers’ diabetes care quality or their ability to implement the EHR and did not coincide with any other large systematic organizational changes.14
The outpatient EHR completely replaced the paper-based medical record and a limited patchwork of preexisting nonintegrated health IT tools. Use of those early health IT tools was limited because paper-based alternatives were still in use. The EHR is an EpicCare-based integrated health IT system that increased the amount of information available at the point of care, presenting integrated clinical information in an electronic medical record, with computer-based physician order entry, diabetes-specific decision support for laboratory testing and treatment intensification, and secure messaging capability. This system has been certified as a complete outpatient EHR, thereby qualifying users who demonstrate meaningful use of the EHR for federal payments.
Our study population included all IDS members (>1 year old) who were in the health plan diabetes clinical registry in the last quarter of 2003. Our study period extended from 2004 to 2009, spanning the EHR implementation timeline, and patients remained in the study cohort unless they disenrolled from the IDS or died.
We linked each patient in the study population to the medical facility where they sought care and defined a facility as using the EHR once it was used for at least 80% of outpatient visits in a given calendar month (generally within 1 month from initial implementation).
We identified all ED visits, hospital stays, and office visits for patients in the study population and constructed monthly counts for each patient for each type of care throughout the study period from 2004 to 2009. The study included office visits with physicians, nurse practitioners, or physician assistants of any specialty.
Among all hospitalizations, we categorized those that were emergency or urgent as nonelective. We also categorized the subset of all hospitalizations that were for ambulatory care–sensitive conditions,17 including those specifically for a diabetes-related or cardiovascular condition,18 according to the primary diagnosis for the hospitalization. We were not able to categorize ED visits because there were changes in the diagnostic coding system used for them during our study observation period.
We examined the change in the mean number of events (ED visits, hospitalizations, or office visits) per patient in each month after the EHR was implemented at the patient’s facility compared with before the EHR was implemented. A 2-tailed significance level of .05 was used in all hypothesis tests.
As a guard against possible omission of unmeasured time-stable confounders, we used a fixed-effect estimation strategy (Stata xtreg, fe option), which is widely used in econometrics19 and is an extension of change score analysis that can be applied when there are repeated measurements per patient. We applied a linear regression model with fixed effects at the patient level to analyze repeated monthly event counts adjusting for seasonality with 11 indicator variables for calendar month, and secular trend with a quadratic form for the month (1-72) in the study period. Medical centers may vary in their diabetes care practices or policies over time unrelated to the EHR. Therefore, we allowed medical center–specific secular trends by including interaction terms between the medical center and trend. We calculated predicted event rates for periods with and without the EHR, using the coefficients from the regression model and treating all patients as if they were in the period with or without EHR, respectively. We multiplied monthly per-patient findings by 12 to estimate annual effects.
We then repeated these analyses in the subset of hospitalizations that were nonelective, for ambulatory care–sensitive conditions, or for cardiovascular- and diabetes-related complications.
As a sensitivity analysis (results shown in the eAppendix in the Supplement), we also used random-effect analyses (Stata xtreg, re option), clustered at the patient level, adjusting for the patient’s age, sex, race/ethnicity, neighborhood socioeconomic status, baseline Diagnostic Cost Group risk score20 (with interaction by Medicare insurance status), medical center, seasonality (calendar month), and medical center–specific secular trend. We also repeated all the analyses only among the subset of continuously enrolled patients (eAppendix). We found similar results across the analytical approaches. All analyses were implemented with Stata version 10.21
Across the 45 facilities in the 17 medical centers, the study included all 169 711 patients in the health plan’s clinical diabetes registry at the beginning of our study period. Between 2004 and 2009, we analyzed 4 997 585 person-months before the implementation of the outpatient EHR and 4 648 572 afterward. Late in the study period, the IDS also began to implement an ED and inpatient EHR at some study hospitals in medical centers that had already implemented the outpatient EHR. However, in this study, only 2.3% of the outpatient EHR observation time was from patients who had also been hospitalized at a hospital that was using an inpatient EHR.
Baseline characteristics of the patient population are shown in Table 1. Patients left the study cohort when they first disenrolled from the IDS (mean, 4.9% per year) or died (mean, 2.6% per year). We used sensitivity analyses among patients continuously enrolled in the study to confirm that attrition did not significantly bias our results (eTable 1 in the Supplement).
The study included 100 510 hospitalizations, 211 623 ED visits, and 2 574 472 office visits before EHR implementation; our study included 96 684 hospitalizations, 194 486 ED visits, and 2 412 882 office visits after the EHR was in use. Before EHR implementation, there were 241 hospitalizations, 508 ED visits, and 6182 hospitalizations per 1000 patients per year; after the EHR was implemented, there were 250 hospitalizations, 502 ED visits, and 6229 office visits per 1000 patients per year.
In multivariable analyses, shown in Table 2, use of the EHR was associated with a statistically significantly decreased rate of ED visits. After use of the EHR, there were 28.80 fewer ED visits (95% CI, 20.28 to 37.32) per 1000 patients per year. This decline represents a 5.54% difference from a mean predicted baseline rate of 519.12 ED visits per 1000 patients annually without the EHR to 490.32 ED visits per 1000 patients with an EHR.
Use of the EHR was also associated with a significantly decreased overall rate of hospitalizations (eFigures 2-16 in the Supplement). After use of the EHR, there were 13.10 fewer hospitalizations (95% CI, 7.37 to 18.82) per 1000 patients annually, which was a 5.21% difference from a predicted baseline mean rate of 251.60 hospitalizations per 1000 patients annually with no EHR to 238.50 hospitalizations per 1000 patients annually with an EHR.
There were 10.92 fewer nonelective hospitalizations (95% CI, 6.48 to 15.48) per 1000 patients annually when the EHR was in use. This was a 6.14% difference from a predicted baseline mean rate of 177.84 hospitalizations per 1000 patients annually with no EHR to 166.80 hospitalizations per 1000 patients annually with an EHR.
Hospitalizations specifically for ambulatory care–sensitive conditions also declined significantly with EHR use, a 10.50% difference from a mean of 67.44 hospitalizations per 1000 patients annually to 60.36 (difference of 7.08 visits per 1000 patients per year; 95% CI, 4.44 to 9.60). The EHR was not associated with statistically significant differences in hospitalizations specifically for diabetes exacerbations, long-term diabetes complications, or cardiovascular conditions. However, these events were rare, even in our large cohort (eTable 3 in the Supplement).
There was no statistically significant difference in office visit rates with the implementation of an EHR (5.4 fewer office visits; 95% CI, −40.80 to 30.00) (Table 2). This represents a 0.09% difference from a predicted baseline mean rate of 6207.12 office visits per 1000 patients annually with no EHR to 6201.72 office visits per 1000 patients annually with an EHR.
Across the staggered implementation of a commercially available certified outpatient EHR, we examined the association between EHR use and unfavorable clinical events and office visit use in patients with diabetes. We found that use of the EHR was associated with modestly lower rates of several unfavorable downstream clinical events that should be affected by better initial care. Specifically, use of the EHR was associated with statistically significantly fewer ED visits, hospitalizations for any condition, nonelective hospitalizations, and hospitalizations for ambulatory care–sensitive conditions. In contrast, we did not find that use of the EHR was associated with any statistically significant change in the rate of office visits among our population of patients with diabetes.
Although the magnitude of the EHR-related associations that we identified was modest, the association between EHR use and hospitalizations was consistent when we examined nonelective hospitalizations or the subset classified as nonelective or as ambulatory care–sensitive conditions, indicating that EHR use may have widespread effect across conditions. The rates of ED visits and hospitalizations in our study population were comparable with national estimates among patients with diabetes.22 Although we did not find statistically significant differences in event rates for diabetes or cardiovascular-specific diagnoses, the adjusted association between EHR use and events in each of these subgroups was in the same direction as in the main analysis across all diagnoses (eAppendix). Our ability to identify statistically significant changes in these specific diagnosis groups was limited because these events were rare, even in our large cohort. None of our analyses identified any significant adverse outcomes associated with EHR use, which contrasts with results of early studies in different care settings that reported negative or mixed effects of health IT use.10,11,23 Future studies should continue to carefully examine the potential for unintended consequences of EHR implementation and should describe the longer-term effects of EHR use.
With ongoing federal incentive payments for meaningful use of EHRs and financial penalties for lack of use beginning in 2015, there has been an increase in adoption of EHRs in the United States.3 These programs encourage EHR use with the goal of gaining improved health care efficiency and care quality. Although several previous studies have found improvements in care processes for patients with diabetes associated with EHR use,9- 13,23 few have found EHR-related improvements in physiologic outcomes such as glycemic or lipid control,24 and to our knowledge no studies have reported effects of outpatient EHR use on adverse health outcomes such as ED visits or hospitalizations. Although the literature capturing care process improvements is important in understanding the pathways for EHR effects, ultimately improvements in process measures matter most to the extent that they lead to improvements in health outcomes and reductions in unfavorable clinical events. Another study14 in the same health care setting and patient population as the current study found that EHR use was associated with improvements in medication treatment intensification after elevated glycosylated hemoglobin or low-density lipoprotein levels, rates of follow-up laboratory testing, and reductions in glycosylated hemoglobin and low-density lipoprotein levels. That study found that patients with poorer disease control experienced greater benefit from EHR use. In this article, we extend the evidence for EHR-related improvements in care delivery by further describing statistically significant modest reductions in downstream adverse health outcomes measured by ED visits and hospitalizations.
EHR adoption has broad-reaching potential to affect health care delivery across many conditions and clinical pathways.9,25 Beyond the specific set of improved diabetes care processes examined in the earlier study,14 we hypothesize that EHR use may also act through many other pathways for other conditions to produce the overall reductions in ED visits and hospitalizations that we found. These additional changes in care delivery may come through mechanisms similar to those that led to improvements in glycemic and lipid control, such as through other types of laboratory tests and prescribing. There are also likely many additional ways that EHRs help physicians and patients avoid clinical events, including improving access to accurate clinical information at the point of care, electronic order entry, or electronic communication between members of the health care team and with patients.10,26 Because a complete EHR system like the one we studied can be used in numerous ways to manage patients with diabetes, including many with multiple conditions, we believe that our finding of reduced ED visits and hospitalizations may represent not just improvements in diabetes care but also the cumulative effect of the EHR across many different care pathways and conditions. Also, our study found EHR-related improvement in care quality and outcomes without changes in office visit rates, which may reflect greater efficiency during visits or care delivery between visits. Future studies should continue to examine the detailed workflow and communication changes associated with EHR use to better understand the many ways the EHRs affect health care delivery.
Although a complete analysis of the cost influence of EHR use was beyond the scope of our study, we provide a simple translation of event rates to dollars by using standard unit costs. Nationally, the mean cost of a hospitalization in 2009 was $9200,27 and the mean cost of an ED visit was $1318.28 We applied the predicted change in these events identified in our study, multiplying the mean costs with 13.10 fewer hospitalizations per 1000 patients and 28.80 fewer ED visits per 1000 patients. Together, these amount to a reduction in ED and inpatient care costs for patients with diabetes of $158 478 annually per 1000 patients. This translation of event rates into dollars is not an analysis of the effect of EHR use on health care costs and does not include improvements in guideline-recommended laboratory monitoring and prescription drug treatment identified in our previous study, the cost implications of any other EHR-related changes in care delivery, or the resources needed to purchase, implement, and maintain an EHR across a practice, including for patients without diabetes. Still, the estimated reductions in ED visits and hospitalizations that we identified for patients with diabetes may have potential to affect ED and hospitalization costs.
In studying associations with EHR use through an observational study, it is important to disentangle any changes associated with the EHR itself from ongoing noise of secular changes in clinical practice or operational policy. Our analysis was designed to isolate differences in event rates attributable specifically to the outpatient EHR implementation from the influence of any baseline tools or programs and from the many ongoing changes in practice patterns and event rates during our study that were unrelated to EHR implementation. Although our study was designed to capture the association between EHR use and event rates across the IDS as a whole, specific experiences varied across the 17 medical centers included in our study and not all centers experienced a decline in their event rates after the EHR implementation (eFigures 2-16 in the Supplement). This study design allowed for rigorous controls and adjustment for background secular time trends in event rates during the study period. However, because this is an observational study and the magnitude of our findings is modest, we cannot rule out unmeasured confounding. Still, our analytic approach focusing on comparisons of within-patient event rates helps to minimize confounding from time-stable characteristics, and our findings were robust across sensitivity analyses using different analytic models.
Although our study identified statistically significant reductions in hospitalizations and ED visits associated with EHR use, the magnitude of these reductions may be limited by several characteristics of the study setting. A limited set of nonintegrated health IT tools was available to physicians along with traditional paper-based systems before EHR implementation; these baseline tools collectively would not have met federal EHR certification criteria. At baseline the study delivery system also already used chronic disease registries to develop programs to manage diabetes and other chronic conditions. It is possible that EHR-associated effects are different in other settings, including different delivery systems and practices that do not use any health IT before EHR implementation or depend on EHR-generated data to develop these efforts to improve care quality.
Overall, we found that among patients with diabetes, use of an outpatient EHR in an integrated delivery system was associated with modest reductions in ED visits and hospitalizations but was not associated with a statistically significant change in office visit rates. Further studies are needed to quantify the association of EHR use with changes in costs.
Corresponding Author: Mary Reed, DrPH, Division of Research, Kaiser Permanente, 2000 Broadway, Oakland, CA 94610 (firstname.lastname@example.org).
Author Contributions: Dr Reed had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Huang, Brand, Graetz, Jaffe, Hsu, Reed.
Acquisition of data: Reed.
Analysis and interpretation of data: Huang, Brand, Graetz, Neugebauer, Fireman, Jaffe, Ballard, Hsu, Reed.
Drafting of the manuscript: Brand, Neugebauer, Jaffe, Ballard, Hsu, Reed.
Critical revision of the manuscript for important intellectual content: Huang, Brand, Graetz, Fireman, Jaffe, Ballard, Hsu, Reed.
Statistical analysis: Huang, Brand, Neugebauer, Fireman, Reed.
Obtained funding: Brand, Reed.
Administrative, technical, or material support: Graetz, Hsu, Reed.
Study supervision: Jaffe, Reed.
Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.
Funding/Support: Research reported herein was supported by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health under award R01DK085070 and the Agency for Healthcare Research and Quality under award R01HS015280.
Role of the Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.