Electronic Health Record Use and the Quality of Ambulatory Care in the United States

Background  Electronic health records (EHRs) have been proposed as a sustainable solution for improving the quality of medical care. We assessed the association between EHR use, as implemented, and the quality of ambulatory care in a nationally representative survey.

Methods  We performed a retrospective, cross-sectional analysis of visits in the 2003 and 2004 National Ambulatory Medical Care Survey. We examined EHR use throughout the United States and the association of EHR use with 17 ambulatory quality indicators. Performance on quality indicators was defined as the percentage of applicable visits in which patients received recommended care.

Results  Electronic health records were used in 18% (95% confidence interval [CI], 15%-22%) of the estimated 1.8 billion ambulatory visits (95% CI, 1.7-2.0 billion) in the United States in 2003 and 2004. For 14 of the 17 quality indicators, there was no significant difference in performance between visits with vs without EHR use. Categories of these indicators included medical management of common diseases, recommended antibiotic prescribing, preventive counseling, screening tests, and avoiding potentially inappropriate medication prescribing in elderly patients. For 2 quality indicators, visits to medical practices using EHRs had significantly better performance: avoiding benzodiazepine use for patients with depression (91% vs 84%; P = .01) and avoiding routine urinalysis during general medical examinations (94% vs 91%; P = .003). For 1 quality indicator, visits to practices using EHRs had significantly worse quality: statin prescribing to patients with hypercholesterolemia (33% vs 47%; P = .01).

Conclusion  As implemented, EHRs were not associated with better quality ambulatory care.

Patients in the United States receive about half of recommended medical care, and there have been widespread calls to improve health care quality.^1,2 Health information technology (HIT) and, in particular, electronic health records (EHRs) have been touted as cost-effective, sustainable solutions for improving quality in medical care.^3-6

Although some computer- and EHR-based decision support efforts to improve quality have been successful, others have not.^7-10 A recent systematic review¹¹ conducted for the Agency for Healthcare Research and Quality found that HIT systems, including EHRs, can increase the delivery of guideline-adherent care, improve quality of care through clinical monitoring, and reduce rates of medical errors. However, much of the research supporting these findings in the United States comes from 4 “benchmark” institutions with largely internally developed EHR systems. Other settings using other systems may not have achieved these quality improvement benefits. We sought to determine whether the use of EHRs, as presently implemented, was associated with higher quality ambulatory care throughout the United States.

The National Ambulatory Medical Care Survey (NAMCS) is administered by the Ambulatory Care Statistics Branch of the National Center for Health Statistics (NCHS) of the Centers for Disease Control and Prevention.¹² The NAMCS collects information on patient visits to non–federally funded, community, office-based physician practices throughout the United States. The NAMCS has a 3-stage sampling design with sampling based on geographic location, physician practices within a geographic location (stratified by physician specialty), and visits within individual physician practices. Patient, physician, and clinical information is collected at each selected visit and is recorded on patient record forms by participating physicians, office staff, or US Census Bureau representatives. Patient information includes demographics and insurance status.¹³ Race and ethnicity are classified by the person filling out the patient record form, according to an office's usual practice for collecting such information. Physician information includes self-identified specialty, geographic region, and whether the practice is in a rural area. Clinical characteristics include up to 3 reasons for the visit (coded using the NCHS-specific Reason for Visit Classification), 3 diagnoses (coded using the International Classification of Diseases, Ninth Revision, Clinical Modification[ICD-9-CM]), 8 medications, and the corresponding National Drug Code Directory Class¹⁴ number for each medicine. Listed medications include prescription and nonprescription medications, which the physician prescribed or provided at the visit or prior to the visit and which the physician expected the patient to continue taking.

The NAMCS collected 25 288 patient records from 1407 physician practices in 2003 and 25 286 patient records from 1121 practices in 2004.¹³ The participation rate of contacted physician practices in the NAMCS was 67% in 2003 and 65% in 2004. Quality control was performed using a 2-way independent verification procedure for 10% of the sample records. Coding errors for various items ranged from 0.0% to 1.1%.

In 2003 and 2004, the NAMCS included a question in the intake survey that asked, “Does your practice use electronic medical records (not including billing records)?” We use the term electronic health record because it is more widely used throughout the medical literature and connotes the maintenance of health, not just the treatment of illness.¹⁵ The NCHS has reported that this question was clear to respondents in pretesting.¹⁶ Responses to this question were blank in 0.25% of estimated visits and “don't know” in 0.45% of estimated visits. We considered blank responses or “don't know” as not using an EHR. Information about specific EHR capabilities (eg, clinical decision support) was not available.

The NCHS weights each visit to allow extrapolation to national estimates for all aspects of the survey. The visit weight accounts for selection probability, nonresponse adjustment, and other adjustments to reflect the universe of ambulatory visits in the United States.¹³ The NCHS institutional review board approved the protocols for the NAMCS, including a waiver of the requirement for informed consent of participating patients.

We performed a retrospective, cross-sectional analysis of ambulatory visits in the NAMCS from 2003 and 2004. We began by modifying, expanding, and updating the analyses by Burt and Sisk¹⁶ and Burt and Hing,¹⁷ who examined the use of computerized clinical support systems and EHRs in practices throughout the United States from 2001 to 2003. We measured the number and percentage of ambulatory visits in which clinicians used an EHR. We examined the associations between EHR use and patient demographics, physician specialty, and office characteristics. We examined patient race and ethnicity to identify potential differences in access to practices using EHRs. Information about individual physician demographic characteristics, such as age and sex, were not available in the publicly available NAMCS. The NCHS has previously reported that no physician-level characteristics were associated with EHR use.¹⁶

We examined a set of 23 quality indicators, which have been previously described for use in the NAMCS by Ma and Stafford,¹⁸ modified slightly to more clearly define comorbidities.¹⁹ The quality indicators were constructed using NAMCS reason for visit codes, medication codes, and ICD-9-CM codes. Detailed criteria and methods for the construction of the quality indicators have been previously described.¹⁸ Briefly, the quality indicators were developed in accordance with the Institute of Medicine's criteria²⁰ of clinical importance, scientific soundness, and feasibility for indicator selection as well as criteria specific to the limitations of the data source. The 23 indicators fall into 5 categories: medical management of common diseases (10 indicators), recommended antibiotic use (3 indicators), preventive counseling (5 indicators), screening tests (4 indicators), and potentially inappropriate prescribing in elderly patients (1 indicator).

The performance on quality indicators was the percentage of applicable visits receiving recommended care. Visits with exclusionary criteria were excluded from the numerator and denominator. Exclusions were identified using reason for visit codes, medication codes, and ICD-9-CM codes.

Our primary analysis included visits to physicians of any specialty. Because many of the quality indicators concern the practice of primary care physicians and cardiovascular specialists, we performed a secondary analysis limited to visits to physicians with a specialty of family medicine, general medicine, internal medicine, pediatrics, and cardiovascular diseases.

We calculated standard errors for all results as recommended by the NCHS using SUDAAN software (Research Triangle Institute International, Research Triangle Park, North Carolina), which accounts for the complex, clustered sampling design of the NAMCS.¹³ The unit of our analysis was the visit. All statistical tests were based on estimates that had less than 30% relative standard error (ie, the standard error divided by the estimate expressed as a percentage of the estimate) and were based on 30 cases or more in the sample data. According to the NCHS, estimates with greater than a 30% relative standard error or based on fewer than 30 sample cases may be unreliable.

Five quality indicators had a relative standard error that was too large or a sample size that was too small to be considered reliable. These 5 quality indicators were angiotensin-converting enzyme inhibitor use for patients with congestive heart failure, selected antibiotic use for women with urinary tract infections, avoiding antibiotic use for upper respiratory tract infections, diet counseling to adolescents, and exercise counseling to adolescents. Inhaled corticosteroid use for children and for adults were reported separately in a previous article.¹⁸ To increase the sample size, we combined the number of children and adults with asthma receiving inhaled corticosteroids into a single indicator for this analysis. This resulted in 17 reportable quality indicators.

We evaluated categorical variables with the χ² test. We also performed multivariable logistic regression modeling including variables associated with EHR use at P < .10 and variables previously hypothesized to be associated with the ambulatory quality indicators (age, sex, and race¹⁸). All analyses were performed with SAS statistical software (version 9.1; SAS Institute, Cary, North Carolina), and SAS-callable SUDAAN software (version 9.0.1; Research Triangle Institute). All P values are 2-tailed, and P < .05 was considered significant.

During 2003 and 2004, there were 1.8 billion ambulatory visits (95% confidence interval [CI], 1.7-2.0 billion visits) in the United States to non–federally funded, community, office-based physician practices. Electronic health records were used in 18% of visits (95% CI, 15%-22%). They were used in 16% of visits (95% CI, 13%-21%) in 2003 and 20% of visits (95% CI, 17%-24%) in 2004 (P = .14).

Electronic health record use was not associated with patient age, sex, race, ethnicity, or insurance status (Table 1) and did not differ by physician specialty or office type. The use of EHRs was less common at visits to solo practices (13%) vs nonsolo practices (21%; P = .01) and in practices in which the physicians were the owners (16%) vs those at which the physician was an employee, contractor, or had another employment status (25%; P = .003). Similarly, EHR use was less common in visits to practices owned by a physician or physician group (17%) or other practice ownership (19%) vs those owned by health maintenance organizations (60%) and other health care corporations (36%; P = .003).

Among the 17 quality indicators, EHR use was associated with better performance on 2, similar performance on 14, and worse performance on 1 (Table 2). Physicians avoided prescribing benzodiazepines to patients with depression at higher rates at visits associated with EHR use (91%) vs without EHR use (84%; P = .01). Physicians avoided routine urinalyses at routine general medical visits more commonly at visits with EHR use (94%) than those without EHR use (91%; P = .003). Physicians prescribed hydroxymethyl glutaryl coenzyme A reductase inhibitors (statins) less frequently to patients with hyperlipidemia at visits associated with EHR use (33%) vs without EHR use (47%; P = .01). In multivariable modeling, the results were unchanged except that EHR use was no longer associated with significantly less benzodiazepine prescribing to patients with depression (odds ratio, 0.63; 95% CI, 0.33-1.18).

In a secondary analysis, limiting the analysis to visits to physicians with a specialty of primary care or cardiovascular disease (n = 18 798 patient records representing 1.0 billion visits [95% CI, 0.9-1.1 billion visits]), there were no changes in the results except for the smoking cessation counseling indicator. Primary care and cardiovascular disease physicians provided smoking cessation counseling to adult smokers at general medical examination visits (n = 1252 patient records representing 64 million estimated visits) more frequently at visits associated with EHR use (39%; 95% CI, 29-49) vs without EHR use (25%; 95% CI, 21-30; P = .03).

In a nationally representative survey, we found no consistent association between EHR use and the quality of ambulatory care. Our analysis updated and expanded the analyses by Burt and Sisk¹⁶ and Burt and Hing¹⁷ who examined the use of computerized clinical support systems and EHRs in office practices throughout the United States from 2001 to 2003, but they did not examine the association between EHR use and the quality of ambulatory care. In our primary analysis, we found an association between EHR use and better quality for 2 of 17 quality indicators: avoiding benzodiazepine prescribing to patients with depression and avoiding urinalysis testing during general medical examinations. In a secondary analysis, limited to primary care and cardiovascular physicians, we found that smoking cessation counseling rates were higher at visits associated with EHR use.

Surprisingly, for 1 indicator, statin prescribing to patients with hyperlipidemia, we found that EHR use was associated with worse quality. This contradicts the findings of a previous study.²¹ The difference we found persisted in multivariable modeling and, in supplementary analyses, was not explained by differential coding of hyperlipidemia or contraindications to statins in practices using EHRs. This finding, which is a cause for concern, may be a result of statistical chance or unmeasured confounding and should certainly be evaluated in future studies.

Electronic health records that include clinical decision support have been efficacious in improving quality in previous studies, as supported by the recent Agency for Healthcare Research and Quality systematic review.^7,9-11 There are several possible explanations why the efficacy of EHRs has not translated into practice. First, unlike the EHRs at benchmark institutions, the types of EHRs that have been widely disseminated may be more rudimentary, lacking clinical decision support, and not focused on quality improvement. A recent report from the NCHS found that only about 40% of physicians who reported using an EHR in 2005 had all 4 of the minimally necessary features of a “complete” EHR system (electronic prescription ordering, test ordering, results, and physician clinical notes²²). Second, physicians may not be using decision support even if it is available within their EHR; in most US settings today there are few incentives to do so. Third, institutions involved in developing and evaluating EHRs might have other attributes that allowed EHRs to be successful in changing quality, such as improved implementation and support practices. Fourth, studies of the efficacy of HIT and EHRs may have involved intensive focus on a restricted set of outcomes that do not easily translate into real-world clinical practice with its competing priorities.

Beyond the lack of association between EHR use and quality, it is worth noting that the performance on most indicators was suboptimal regardless of whether an EHR was used. There was substantial room for improvement, which increases the expectation that EHRs should be associated with better quality. Indeed, advanced EHRs with integrated clinical decision support almost certainly represent part of the solution.³ Tools within EHRs that provide registry functions and support team care are also needed. Financial incentives also may be useful. In the United Kingdom, where EHR use in general practices is nearly universal and approximately 20% of general practice income is based on quality performance, there was dramatic improvement in quality across a large number of measures.²³ Further improvement in health care quality will likely require a broader reinvention of health care, including organizational change, a focus on disease prevention, and the greater involvement of patients in their own care.^24-27

This analysis has several limitations that should be considered. First, despite the large total number of patient records and estimated visits, the sample size was actually small for some of the quality indicators. This prevented us from reporting on several quality indicators and limited our ability to detect potential differences for other quality indicators that had wide CIs. Despite this, there does not seem to be a consistent trend even in the point estimates of the nonsignificant quality indicators. Second, the NAMCS is dependent on physicians, office staff, and census workers for the accurate coding of medications, diagnoses, and other components of the survey.

Third, the NAMCS is a cross-sectional survey, so one cannot assume causality. For example, it is possible that clinics with poorer quality have recently implemented EHRs in an effort to improve quality. Also, because the NAMCS is cross-sectional, it may be more difficult to demonstrate improvements in quality, especially in regard to screening and prevention. A longitudinal data source may be better for showing an association between EHR use and improved ambulatory quality for these types of measures. Fourth, although the results were unchanged using multivariable modeling, the NAMCS may not include important negative confounders that obscured a true relationship between EHR use and improved ambulatory quality. However, in considering unmeasured confounding, it seems more likely that there would have been unmeasured positive confounders (eg, clinics with adequate resources to implement EHRs and quality improvement strategies to keep quality high), yet we found no consistent association between EHR use and ambulatory quality. Fifth, the definition of an EHR itself is a potential limitation. Although the NCHS tested this term for understandability, clinics could report they are using an EHR simply for prescribing functionality or simply as a note-keeping function. Even though about 80% of EHRs in medical group practices have drug interaction warnings and about 65% contain clinical guidelines and protocols,²⁸ we have no information on how many EHRs in the present analysis had clinical decision support that would be applicable to the quality indicators we examined. In the coming years, analysis of additional NAMCS data, which should contain more granular information about EHR capabilities, may shed light on associations between specific EHR capabilities and ambulatory quality. Despite these limitations, the NAMCS is nationally representative and is the best source of data to determine if the use of EHRs, as presently implemented in the United States, is associated with higher quality ambulatory care.

In summary, although HIT and EHRs can improve quality, we found that EHR use was generally not associated with improved quality of ambulatory care. Our findings are not a refutation of previous studies. Rather, they suggest that as EHR use broadens, one should not assume an automatic diffusion of improved quality of care. In selecting an EHR, physician practices should carefully consider the inclusion of clinical decision support to facilitate quality care for individuals as well as the availability of tools, like quality reporting and registry functions, to facilitate quality care for populations. Policy makers should consider steps to increase the likelihood that further diffusion of EHR has the desired effect of improving quality of care. Clinicians, health system leaders, and researchers should continuously measure the quality of medical care as EHR use expands.

Correspondence: Jeffrey A. Linder, MD, MPH, Division of General Medicine, Brigham and Women's Hospital, 1620 Tremont St, BC-3-2X, Boston, MA 02120 (jlinder@partners.org).

Accepted for Publication: March 8, 2007.

Author Contributions: Dr Linder had full access to all of the data in the study, which are publicly available through the National Center for Health Statistics. Dr Linder performed and takes responsibility for the accuracy of the data analysis. Study concept and design: Linder, Ma, Bates, Middleton, and Stafford. Acquisition of data: Linder. Analysis and interpretation of data: Linder, Ma, Bates, Middleton, and Stafford. Drafting of the manuscript: Linder and Bates. Critical revision of the manuscript for important intellectual content: Linder, Ma, Bates, Middleton, and Stafford. Statistical analysis: Linder, Ma, Bates, and Stafford. Obtained funding: Linder. Administrative, technical, and material support: Linder, Bates, and Middleton. Study supervision: Linder, Bates, and Middleton.

Financial Disclosure: Dr Bates is a coinventor of patent No. 6029138 held by Brigham and Women's Hospital on the use of decision support software for medical management, licensed to the Medicalis Corp. He holds a minority equity position in Medicalis, a privately held company that develops web-based decision support for radiology test ordering, and serves as a consultant to Medicalis. He is on the clinical advisory board for Zynx Inc, which develops evidence-based algorithms.

Funding/Support: Dr Linder is supported by a Career Development Award (K08 HS014563) from the Agency for Healthcare Research and Quality. Components of this study were also supported by a research grant from Agency for Healthcare Research and Quality (R01 HS11313).

Role of the Sponsors: No sponsor or funding source had a role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript.

Previous Presentation: This work was presented in part at the 29th Annual Meeting of the Society of General Internal Medicine; April 28, 2006; Los Angeles, California.