National Trends in the Safety Performance of Electronic Health Record Systems From 2009 to 2018 | Clinical Pharmacy and Pharmacology | JAMA Network Open | JAMA Network
[Skip to Navigation]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
Bates  DW, Singh  H.  Two decades since to err is human: an assessment of progress and emerging priorities in patient safety.   Health Aff (Millwood). 2018;37(11):1736-1743. doi:10.1377/hlthaff.2018.0738 PubMedGoogle ScholarCrossref
US Department of Health and Human Services, Office of the National Coordinator for Health Information Technology. 2018 Report to Congress: annual update on the adoption of a nationwide system for the electronic use and exchange of health information. Accessed April 29, 2020.
Committee on Patient Safety and Health Information Technology, Institute of Medicine.  Health IT and Patient Safety: Building Safer Systems for Better Care. National Academies Press; 2012.
Kilbridge  PM, Welebob  EM, Classen  DC.  Development of the Leapfrog methodology for evaluating hospital implemented inpatient computerized physician order entry systems.   Qual Saf Health Care. 2006;15(2):81-84. doi:10.1136/qshc.2005.014969 PubMedGoogle ScholarCrossref
Metzger  J, Welebob  E, Bates  DW, Lipsitz  S, Classen  DC.  Mixed results in the safety performance of computerized physician order entry.   Health Aff (Millwood). 2010;29(4):655-663. doi:10.1377/hlthaff.2010.0160 PubMedGoogle ScholarCrossref
Classen  DC, Avery  AJ, Bates  DW.  Evaluation and certification of computerized provider order entry systems.   J Am Med Inform Assoc. 2007;14(1):48-55. doi:10.1197/jamia.M2248 PubMedGoogle ScholarCrossref
Leung  AA, Keohane  C, Lipsitz  S,  et al.  Relationship between medication event rates and the Leapfrog computerized physician order entry evaluation tool.   J Am Med Inform Assoc. 2013;20(e1):e85-e90. doi:10.1136/amiajnl-2012-001549 PubMedGoogle ScholarCrossref
Chaparro  JD, Classen  DC, Danforth  M, Stockwell  DC, Longhurst  CA.  National trends in safety performance of electronic health record systems in children’s hospitals.   J Am Med Inform Assoc. 2017;24(2):268-274. doi:10.1093/jamia/ocw134PubMedGoogle Scholar
Holmgren  AJ, Co  Z, Newmark  L, Danforth  M, Classen  D, Bates  D.  Assessing the safety of electronic health records: a national longitudinal study of medication-related decision support.   BMJ Qual Saf. 2020;29(1):52-59. doi:10.1136/bmjqs-2019-009609PubMedGoogle ScholarCrossref
Leapfrog Group. Computerized physician order entry. Accessed April 28, 2020.
Jha  AK, Orav  EJ, Ridgway  AB, Zheng  J, Epstein  AM.  Does the Leapfrog program help identify high-quality hospitals?   Jt Comm J Qual Patient Saf. 2008;34(6):318-325. doi:10.1016/S1553-7250(08)34040-9 PubMedGoogle Scholar
The Leapfrog Group. The Leapfrog Hospital Survey overview. Accessed April 28, 2020.
American Hospital Association. AHA annual survey database. Accessed April 28, 2020.
Jha  AK, DesRoches  CM, Campbell  EG,  et al.  Use of electronic health records in US hospitals.   N Engl J Med. 2009;360(16):1628-1638. doi:10.1056/NEJMsa0900592 PubMedGoogle ScholarCrossref
Roberts  JS, Coale  JG, Redman  RR.  A history of the Joint Commission on Accreditation of Hospitals.   JAMA. 1987;258(7):936-940. doi:10.1001/jama.1987.03400070074038 PubMedGoogle ScholarCrossref
Werner  RM, Bradlow  ET.  Relationship between Medicare’s hospital compare performance measures and mortality rates.   JAMA. 2006;296(22):2694-2702. PubMedGoogle ScholarCrossref
Clancy  CM.  CMS’s hospital-acquired condition lists link hospital payment, patient safety.   Am J Med Qual. 2009;24(2):166-168. doi:10.1177/1062860608331241 PubMedGoogle ScholarCrossref
Holmgren  AJ, Adler-Milstein  J, McCullough  J.  Are all certified EHRs created equal: assessing the relationship between EHR vendor and hospital meaningful use performance.   J Am Med Inform Assoc. 2018;25(6):654-660. doi:10.1093/jamia/ocx135 PubMedGoogle ScholarCrossref
Smith  SN, Reichert  HA, Ameling  JM, Meddings  J.  Dissecting Leapfrog: how well do Leapfrog safe practices scores correlate with Hospital compare ratings and penalties, and how much do they matter?   Med Care. 2017;55(6):606-614. doi:10.1097/MLR.0000000000000716 PubMedGoogle ScholarCrossref
Adler-Milstein  J, Holmgren  AJ, Kralovec  P, Worzala  C, Searcy  T, Patel  V.  Electronic health record adoption in US hospitals: the emergence of a digital “advanced use” divide.   J Am Med Inform Assoc. 2017;24(6):1142-1148. doi:10.1093/jamia/ocx080 PubMedGoogle ScholarCrossref
Kaushal  R, Jha  AK, Franz  C,  et al; Brigham and Women’s Hospital CPOE Working Group.  Return on investment for a computerized physician order entry system.   J Am Med Inform Assoc. 2006;13(3):261-266. doi:10.1197/jamia.M1984 PubMedGoogle ScholarCrossref
Gelles  D. Boeing 737 Max: What’s Happened After the 2 Deadly Crashes. New York Times. October 28, 2019. Accessed December 9, 2019.
Longhurst  CA, Davis  T, Maneker  A,  et al; Arch Collaborative.  Local investment in training drives electronic health record user satisfaction.   Appl Clin Inform. 2019;10(2):331-335. doi:10.1055/s-0039-1688753 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    1 Comment for this article
    What is safety?
    Chad Todd |
    A profitable exercise is asking “What is safety?”. Ponder upon from Gene I. Rochlin one of the original researchers for high-reliability organization theory wrote a paper in 1999 called “Safe operation as a social construct.” Safety is something a system does, not something a system has (e.g. this includes EHR systems).

    “However intersubjective, socially constructed and collective these descriptors may be, they are central to understanding the role of agency in providing safety as a positive effect that is centrally important in understanding the reliable operation of complex, potentially hazardous socio-technical systems. To the extent that regulators, systems
    designers, and analysts continue to focus their attention on the avoidance of error and the control of risk, and to seek objective and positivistic indicators of performance, safety becomes marginalized as a residual property. This not only neglects the importance of expressed and perceived safety as a constitutive property of safe operation, but may actually interfere with the means and processes by which it is created and maintained. It is therefore important in practice, as well as in theory, to encourage continued and expanding research and enquiry into safety as an expression of myth and ritual, of agency as well as structure, and not just as a measurable, statistical property or organization, a reporting system for work team errors, or a way of expressing the avoidance of consequential accidents.”
    Original Investigation
    Health Informatics
    May 29, 2020

    National Trends in the Safety Performance of Electronic Health Record Systems From 2009 to 2018

    Author Affiliations
    • 1Division of Clinical Epidemiology, University of Utah School of Medicine, Salt Lake City
    • 2Harvard Business School, Boston, Massachusetts
    • 3Department of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
    • 4Clinical and Quality Analysis, Partners Healthcare, Somerville, Massachusetts
    • 5The Leapfrog Group, Washington, DC
    • 6Harvard Medical School, Boston, Massachusetts
    JAMA Netw Open. 2020;3(5):e205547. doi:10.1001/jamanetworkopen.2020.5547
    Key Points español 中文 (chinese)

    Question  How did safety performance of electronic health record systems (EHRs) change in the US from 2009 to 2018?

    Findings  In this case series using 8657 hospital-year observations from adult hospitals nationwide that used the National Quality Forum Health IT Safety Measure, a computerized physician order entry and EHR safety test, from 2009 to 2018, mean scores on the overall test increased from 53.9% in 2009 to 65.6% in 2018. There was considerable variation in test performance by hospital and EHR vendor.

    Meaning  These findings suggest that, despite broad adoption and optimization of EHR systems in hospitals, wide variation in the safety performance of operational EHR systems remains across a large sample of hospitals and EHR vendors, and serious safety vulnerabilities persist in these operational EHRs.


    Importance  Despite the broad adoption of electronic health record (EHR) systems across the continuum of care, safety problems persist.

    Objective  To measure the safety performance of operational EHRs in hospitals across the country during a 10-year period.

    Design, Setting, and Participants  This case series included all US adult hospitals nationwide that used the National Quality Forum Health IT Safety Measure EHR computerized physician order entry safety test administered by the Leapfrog Group between 2009 and 2018. Data were analyzed from July 1, 2018 to December 1, 2019.

    Exposure  The Health IT Safety Measure test, which uses simulated medication orders that have either injured or killed patients previously to evaluate how well hospital EHRs could identify medication errors with potential for patient harm.

    Main Outcomes and Measures  Descriptive statistics for performance on the assessment test over time were calculated at the overall test score level, type of decision support category level, and EHR vendor level.

    Results  Among 8657 hospital-years observed during the study, mean (SD) scores on the overall test increased from 53.9% (18.3%) in 2009 to 65.6% (15.4%) in 2018. Mean (SD) hospital score for the categories representing basic clinical decision support increased from 69.8% (20.8%) in 2009 to 85.6% (14.9%) in 2018. For the categories representing advanced clinical decision support, the mean (SD) score increased from 29.6% (22.4%) in 2009 to 46.1% (21.6%) in 2018. There was considerable variation in test performance by EHR.

    Conclusions and Relevance  These findings suggest that despite broad adoption and optimization of EHR systems in hospitals, wide variation in the safety performance of operational EHR systems remains across a large sample of hospitals and EHR vendors. Hospitals using some EHR vendors had significantly higher test scores. Overall, substantial safety risk persists in current hospital EHR systems.