Association of an Electronic Health Record Add-on App for Neonatal Bilirubin Management With Physician Efficiency and Care Quality | Electronic Health Records | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure.  App Screenshots for Bilirubin Analysis
App Screenshots for Bilirubin Analysis

Screenshots shown for a patient with hyperbilirubinemia (A) and projected hyperbilirubinemia (B). 1, link to open the app in the EHR sidebar; 2, bilirubin levels over time (blue line); 3, patient-specific phototherapy threshold according to American Academy of Pediatrics guideline (green line); 4, inpatient phototherapy (yellow area); 5, patient-specific risk factors; 6, mother’s laboratory information; 7, other neurotoxicity risk factors; 8, guideline-based care recommendations (in blue box); and 9, predicted risk of rebound hyperbilirubinemia following phototherapy (in green box). AAP indicates American Academy of Pediatrics; InPt, inpatient; Neg, negative; OutPt, outpatient; Pos, positive; TSB, total serum bilirubin. Used with permission from Epic Systems Corp.

Table 1.  Randomized Task-Timing Study: Time Required to Complete Bilirubin Management Tasks
Randomized Task-Timing Study: Time Required to Complete Bilirubin Management Tasks
Table 2.  App Use Study: Number of Uses and Estimated Time Savings per Year in 2018a
App Use Study: Number of Uses and Estimated Time Savings per Year in 2018a
Table 3.  Retrospective Pre-Post Intervention Study: Patient Characteristics and Outcomes
Retrospective Pre-Post Intervention Study: Patient Characteristics and Outcomes
Table 4.  Usability Survey Study: SUS Scoresa
Usability Survey Study: SUS Scoresa
1.
Schulte  F, Fry  E. Death by a thousand clicks: where electronic health records went wrong. Kaiser Health News. https://khn.org/news/death-by-a-thousand-clicks/. Published March 18, 2019. Accessed June 21, 2019.
2.
Sinsky  C, Colligan  L, Li  L,  et al.  Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties.  Ann Intern Med. 2016;165(11):753-760. doi:10.7326/M16-0961PubMedGoogle ScholarCrossref
3.
Chaiyachati  KH, Shea  JA, Asch  DA,  et al.  Assessment of inpatient time allocation among first-year internal medicine residents using time-motion observations.  JAMA Intern Med. 2019;179(6):760-767. doi:10.1001/jamainternmed.2019.0095PubMedGoogle ScholarCrossref
4.
Gardner  RL, Cooper  E, Haskell  J,  et al.  Physician stress and burnout: the impact of health information technology.  J Am Med Inform Assoc. 2019;26(2):106-114. doi:10.1093/jamia/ocy145PubMedGoogle ScholarCrossref
5.
Mandl  KD, Mandel  JC, Kohane  IS.  Driving innovation in health systems through an apps-based information economy.  Cell Syst. 2015;1(1):8-13. doi:10.1016/j.cels.2015.05.001PubMedGoogle ScholarCrossref
6.
Mandl  KD, Kohane  IS.  No small change for the health information economy.  N Engl J Med. 2009;360(13):1278-1281. doi:10.1056/NEJMp0900411PubMedGoogle ScholarCrossref
7.
Mandel  JC, Kreda  DA, Mandl  KD, Kohane  IS, Ramoni  RB.  SMART on FHIR: a standards-based, interoperable apps platform for electronic health records.  J Am Med Inform Assoc. 2016;23(5):899-908. doi:10.1093/jamia/ocv189PubMedGoogle ScholarCrossref
8.
Health Level Seven International (HL7). SMART App Launch Framework. http://www.hl7.org/fhir/smart-app-launch/. Accessed September 23, 2019.
9.
Health Level Seven International (HL7). Fast Healthcare Interoperability Resources (FHIR) standard. https://www.hl7.org/fhir/. Accessed September 23, 2019.
10.
Twichell  SA, Rea  CJ, Melvin  P,  et al.  The effect of an electronic health record–based tool on abnormal pediatric blood pressure recognition.  Congenit Heart Dis. 2017;12(4):484-490. doi:10.1111/chd.12469PubMedGoogle ScholarCrossref
11.
American Academy of Pediatrics Subcommittee on Hyperbilirubinemia.  Management of hyperbilirubinemia in the newborn infant 35 or more weeks of gestation.  Pediatrics. 2004;114(1):297-316. doi:10.1542/peds.114.1.297PubMedGoogle ScholarCrossref
12.
Lobach  D, Sanders  GD, Bright  TJ,  et al.  Enabling health care decisionmaking through clinical decision support and knowledge management.  Evid Rep Technol Assess (Full Rep). 2012;(203):1-784.PubMedGoogle Scholar
13.
Ogrinc  G, Davies  L, Goodman  D, Batalden  P, Davidoff  F, Stevens  D.  SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process.  BMJ Qual Saf. 2016;25:986-992. doi:10.1136/bmjqs-2015-004411PubMedGoogle ScholarCrossref
14.
Longhurst  C, Turner  S, Burgos  AE.  Development of a Web-based decision support tool to increase use of neonatal hyperbilirubinemia guidelines.  Jt Comm J Qual Patient Saf. 2009;35(5):256-262. doi:10.1016/S1553-7250(09)35035-7PubMedGoogle ScholarCrossref
15.
Thornton  SN, Thompson  BS, Millar  JA, Eggert  LD, Wilcox  AB. Neonatal bilirubin management as an implementation example of interdisciplinary continuum of care tools. AMIA. Annual Symposium Proceedings. 2007;2007:726-730.
16.
SMART App Gallery. Intermountain Healthcare bilirubin chart. https://apps.smarthealthit.org/app/bilirubin-chart. Accessed July 10, 2019.
17.
US Core FHIR Profiles. (Release 1.0.1). Health Level Seven International (HL7). http://hl7.org/fhir/us/core/1.0.1/. Accessed July 10, 2019.
18.
Chang  PW, Kuzniewicz  MW, McCulloch  CE, Newman  TB.  A clinical prediction rule for rebound hyperbilirubinemia following inpatient phototherapy.  Pediatrics. 2017;139(3):e20162896. doi:10.1542/peds.2016-2896PubMedGoogle Scholar
19.
Brooke  J. SUS—a quick and dirty usability scale. In: Usability Evaluation in Industry. Vol 189. 1996:4-7. https://cui.unige.ch/isi/icle-wiki/_media/ipm:test-suschapt.pdf. Accessed October 15, 2019.
20.
Bangor  A, Kortum  P, Miller  J.  Determining what individual SUS scores mean: adding an adjective rating scale.  J Usability Stud. 2009;4(3):114-123. http://uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS_Bangor_May2009.pdf. Accessed October 15, 2019.Google Scholar
21.
Sauro  J. SUStisfied? Little-known system usability scale facts. User Exp Mag. 2011;10(3). https://uxpamagazine.org/sustified/. Accessed September 23, 2019.
22.
Office of the National Coordinator for Health Information Technology. Provider user-experience challenge. https://www.healthit.gov/techlab/ipg/node/4/submission/2131. Published 2017. Accessed July 10, 2019.
23.
Centers for Disease Control and Prevention. National Vital Statistics System—Birth Data. https://www.cdc.gov/nchs/nvss/births.htm. Accessed June 21, 2019.
24.
Levine  DM, Linder  JA, Landon  BE.  The quality of outpatient care delivered to adults in the United States, 2002 to 2013.  JAMA Intern Med. 2016;176(12):1778-1790. doi:10.1001/jamainternmed.2016.6217PubMedGoogle ScholarCrossref
25.
McGlynn  EA, Asch  SM, Adams  J,  et al.  The quality of health care delivered to adults in the United States.  N Engl J Med. 2003;348(26):2635-2645. doi:10.1056/NEJMsa022615PubMedGoogle ScholarCrossref
26.
Yarnall  KSH, Pollak  KI, Østbye  T, Krause  KM, Michener  JL.  Primary care: is there enough time for prevention?  Am J Public Health. 2003;93(4):635-641. doi:10.2105/AJPH.93.4.635PubMedGoogle ScholarCrossref
27.
Wright  MC, Borbolla  D, Waller  RG,  et al.  Critical care information display approaches and design frameworks: a systematic review and meta-analysis.  J Biomed Inform X. 2019;3:3. doi:10.1016/j.yjbinx.2019.100041PubMedGoogle Scholar
28.
Waller  RG, Wright  MC, Segall  N,  et al.  Novel displays of patient information in critical care settings: a systematic review.  J Am Med Inform Assoc. 2019;26(5):479-489. doi:10.1093/jamia/ocy193PubMedGoogle ScholarCrossref
29.
Ericsson  KA. Superior working memory in experts. In:  The Cambridge Handbook of Expertise and Expert Performance. Cambridge, UK: Cambridge University Press; 2018:696-713. doi:10.1017/9781316480748.036
30.
Camos  V.  Domain-specific versus domain-general maintenance in working memory: reconciliation within the time-based resource sharing model.  Prog Brain Res. 2017;232:135-171. doi:10.1016/bs.plm.2017.03.005Google Scholar
31.
Van de Velde  S, Heselmans  A, Delvaux  N,  et al.  A systematic review of trials evaluating success factors of interventions with computerised clinical decision support.  Implement Sci. 2018;13(1):114. doi:10.1186/s13012-018-0790-1PubMedGoogle ScholarCrossref
32.
Kawamoto  K, Houlihan  CA, Balas  EA, Lobach  DF.  Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success.  BMJ. 2005;330(7494):765. doi:10.1136/bmj.38398.500764.8FPubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Health Informatics
    November 15, 2019

    Association of an Electronic Health Record Add-on App for Neonatal Bilirubin Management With Physician Efficiency and Care Quality

    Author Affiliations
    • 1Department of Biomedical Informatics, University of Utah, Salt Lake City
    • 2Department of Pediatrics, University of Utah, Salt Lake City
    JAMA Netw Open. 2019;2(11):e1915343. doi:10.1001/jamanetworkopen.2019.15343
    Key Points español 中文 (chinese)

    Question  Is an electronic health record add-on app for neonatal bilirubin management associated with time savings for clinicians and improved quality of care?

    Findings  In this quality improvement study, an electronic health record add-on app for neonatal bilirubin management saved clinicians a mean of 66 seconds for bilirubin management tasks compared with a commonly used tool. In a retrospective pre-post analysis, the odds of clinically appropriate phototherapy orders during hospitalization increased significantly by 84%.

    Meaning  These findings suggest that well-designed electronic health record add-on apps may be associated with time savings for physicians and improvements in patient care.

    Abstract

    Importance  The usefulness of electronic health record (EHR) systems could be significantly enhanced by innovative, third-party EHR add-on apps.

    Objective  To evaluate whether an EHR add-on app for neonatal bilirubin management can save clinicians time and improve patient care.

    Design, Setting, and Participants  This quality improvement study was conducted at the University of Utah Health Well Baby nursery and outpatient clinics and consisted of 4 substudies: (1) time savings were estimated in an experimental task-timing study comparing the time required for physicians to manage newborns’ bilirubin levels with and without the add-on app, (2) app use was estimated from app logs, (3) health care use measures and guideline compliance were compared retrospectively before and after the intervention, and (4) clinician-perceived usability was measured through System Usability Scale surveys. The study took place between April 1, 2016, and September 3, 2019. Data analyses were conducted from October 30, 2018, to September 23, 2019.

    Interventions  At baseline, clinicians used a manual approach to ensure compliance with an evidence-based clinical guideline for neonatal bilirubin management. To facilitate guideline compliance, an EHR add-on app that automatically retrieves, organizes, and visualizes relevant patient data was developed. The app provides patient-specific assessments and recommendations, including the risk of rebound hyperbilirubinemia following phototherapy based on a predictive model. The add-on app was integrated with the University of Utah Health EHR on April 12, 2017.

    Main Outcomes and Measures  Clinician time savings, app use, health care use measures, guideline-compliant phototherapy ordering, and perceived usability as measured by the System Usability Scale survey. The survey is composed of 10 statements with responses ranging from 1 (strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100, with ratings described as worst imaginable (mean System Usability Scale score, 12.5), awful (20.3), poor (35.7), okay (50.9), good (71.4), excellent (85.5), and best imaginable (90.9).

    Results  In 2018, the application was used 20 516 times by clinicians for 91.84% of eligible newborns. Use of the app saved 66 seconds for bilirubin management tasks compared with a commonly used tool (95% CI, 53-79 seconds; P < .001). Following the intervention, health care use rates remained stable, while orders for clinically appropriate phototherapy during hospitalization increased for newborns with bilirubin levels above the guideline-recommended threshold (odds ratio, 1.84; 95% CI, 1.16-2.90; P = .009). Surveys indicated excellent usability (System Usability Scale score, 83.90; 95% CI, 81.49-86.31).

    Conclusions and Relevance  Well-designed EHR add-on apps may save clinicians time and improve patient care. If time-saving apps, such as the bilirubin app, were implemented widely across institutions and care domains, the potential association with improved patient care and clinician efficiency could be significant. The University of Utah Health bilirubin app is being prepared for release into EHR app stores as free-to-use software.

    Introduction

    Despite billions of dollars invested, electronic health records (EHRs) often fall short in supporting efficient, high-quality patient care.1 Ambulatory care physicians can spend 2 hours on EHR and desk work for every hour spent in direct clinical face time,2 and inpatient physicians can spend 5 hours on such tasks for every hour spent in direct patient care.3 In a statewide survey of Rhode Island physicians conducted in 2017, close to two-thirds of physicians reported that use of EHRs added to their daily frustration.4

    Through emerging technology standards, EHRs now allow the integration of add-on apps, whereby third-party app developers can deliver innovative solutions that enhance the utility of EHRs for targeted health care tasks and domains.5,6 Just as smartphones deliver a better user experience through a variety of add-on apps, the hope is that EHRs can deliver improved user experiences through such apps. Unlike EHR vendors, who may be overwhelmed with competing priorities, such as maintaining legacy functionality and providing support to thousands of customers for a wide range of clinical and administrative tasks, a third-party app developer can focus entirely on optimizing a comparatively small range of decision tasks. Apps also use a technical framework designed to facilitate substituting one app for another, whereas switching EHRs is a disruptive and costly process. Thus, app developers may be inherently incentivized to optimize patient care and the user experience in their domain of focus, whereas EHR vendors are unlikely to gain or lose a customer based on their level of support for any specific decision task. Also, unlike EHR vendors, who may have years-long cycles for feature prioritization and development, third-party app developers can adopt a much more rapid and iterative development approach, with the ability to develop and release enhancements much more quickly based on user feedback.

    The technology enabling EHR add-on apps is known as Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources (SMART on FHIR; pronounced smart on fire).7 Apps enabled by SMART on FHIR use the Health Level Seven International (HL7) SMART standard8 to enable single sign-on and integration with the EHR user interface, as well as the HL7 FHIR data interface standard9 to pull in relevant patient data automatically. Although the evidence of the influence of SMART on FHIR apps on clinical outcomes is limited, investigators at Boston Children's Hospital showed in 2017 that the introduction of a SMART on FHIR app for pediatric blood pressure visualization was associated with an increase in the recognition of abnormal blood pressure (7.1% vs 4.9%; P < .001).10

    The present study reports on an evaluation of an EHR add-on app that was implemented in 2017 at an academic medical center to support the American Academy of Pediatrics (AAP) guideline on the management of neonatal hyperbilirubinemia.11 To manage newborns’ care according to this guideline, the clinician must retrieve disparate data scattered across the medical record, synthesize the data for risk classification, and apply guideline algorithms to identify patient-specific care needs, such as the administration of phototherapy when bilirubin levels exceed risk-based treatment thresholds.11 The app was designed to support these tasks by retrieving relevant data, providing a visual data summary, and delivering guideline-based recommendations on next steps. We also designed the app to support known success factors for clinical decision support (CDS) systems, including providing CDS at the time and location of decision-making, providing recommendations rather than just assessments, integrating with the EHR, and minimizing the need for additional clinician data entry.12 The goal of this study was to evaluate our hypothesis that this EHR add-on app would save clinicians time and improve care quality.

    Methods
    Study Design

    This was a mixed-methods quality improvement study that included 4 separate substudies: (1) an experimental task-timing study to estimate time savings, (2) an observational study of app use, (3) a retrospective pre-post intervention study evaluating patient outcomes, and (4) a usability survey study. App implementation and the subsequent quantitative evaluations of app use and patient outcomes were exempted as quality improvement by the University of Utah Institutional Review Board (IRB). The timing study and survey were approved by the IRB, and participants consented verbally after reading an IRB-approved informed consent cover letter. This study report follows the Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) reporting guidelines.13

    The study was conducted at University of Utah Health (UUH), an academic health care system using the Epic EHR (EPIC Systems Corp). Inpatient care for well newborns is provided in a newborn nursery as mother-baby couplet care. Follow-up visits are conducted in 27 UUH clinics. Universal bilirubin screening was implemented in the nursery on March 31, 2016. Prior to the UUH app intervention, a stand-alone web-based tool known as BiliTool (BiliTool Inc) was used for bilirubin management.14

    Newborns born at 35 weeks’ gestation or longer and admitted to the nursery were included in the analysis. Follow-up visits at UUH clinics within 14 days from discharge were included.

    Before proceeding with the design and development of the Bili App, and in accordance with UUH’s standard operating procedures for new app development, the project team explored whether native EHR approaches could be used to meet the user needs. In particular, conventional EHR mechanisms for CDS, such as alerts, reminders, and order sets, were considered. However, it was determined that these conventional CDS mechanisms would not be able to support the desired functionality, including providing a graphical summary of relevant patient data, supporting the full complexity of the underlying clinical decision logic, and adjusting recommendations based on user input. Thus, the decision was made to proceed with app development.

    The app leverages the SMART on FHIR framework. Starting from a basic app developed by Intermountain Healthcare to graph bilirubin levels against a risk nomogram,15,16 the University of Utah ReImagine EHR team iteratively refined the app based on physician feedback. The ReImagine EHR team includes clinical informaticists with expertise in areas including software development, software architecture, standards-based interoperability, cognitive psychology, and biostatistics. Both the SMART and a core set of FHIR data standards, known as the US Core FHIR profiles,17 were natively supported by the Epic EHR, facilitating integration of the app. However, to achieve the full set of functionality requested by clinicians, custom FHIR interfaces had to be developed to pull additional data elements, including the mother’s laboratory data and outpatient phototherapy orders. Such custom FHIR interfaces can be shared across health care systems and may become unnecessary as EHR vendors increase the scope of data covered by their native FHIR data interfaces.

    The resulting app (Figure) supports neonatal bilirubin management according to the AAP guideline11 and estimates the risk of postphototherapy rebound hyperbilirubinemia.18 The app was deployed institution-wide on April 12, 2017. Beyond a brief email message to EHR users on app availability, app awareness was spread by word-of-mouth.

    Resident physicians screen for and manage hyperbilirubinemia in the nursery under the guidance of attending physicians. Prior to the introduction of the UUH app, clinicians used the BiliTool website, which is available as a link in the EHR. Clinicians would open the website and manually enter the time of birth as well as the last total bilirubin level and associated specimen collection time. The website takes these inputs and provides recommendations for all 3 potential risk levels as defined by the AAP guideline (lower, medium, and higher risk).11 BiliTool does not calculate the patient-specific risk level; as such, clinicians independently assessed the patient’s risk status using data in the EHR, with the patient’s risk level determined primarily by the patient’s gestational age and direct Coombs test results.11

    With the UUH app, clinicians launch the app through the EHR’s sidebar (Figure, A, item 1). The app then loads as any other EHR screen does. There is no need for additional login and all the information is pulled automatically from the EHR. If the patient does not have direct Coombs test results, the app prompts the clinician whether other neurotoxicity risk factors are present before providing a recommendation (Figure, A, item 7). Clinicians then review the patient data, including the patient’s bilirubin levels over time (Figure, A, item 2); the patient’s risk-specific threshold where phototherapy is indicated (Figure, A, item 3); history of phototherapy administration (Figure, A, item 4); other relevant data, including the mother’s blood type and indirect Coombs test results (Figure, A, item 6); patient-specific recommendations on next steps (Figure, A, item 8); and, if applicable, the patient’s estimated risk of rebound hyperbilirubinemia following phototherapy (Figure, A, item 9). Clinicians can also evaluate the influence of potential changes on patient risk factors, such as if the patient’s gestational age at birth was 37 weeks 6 days rather than 38 weeks 0 days (Figure, A, item 5).

    Substudies

    In a time-savings evaluation, 12 pediatric and family medicine resident physicians on service in the nursery (including 7 interns, 3-second-year residents, and 2 third-year residents) were recruited to complete bilirubin management tasks for their current patients (n = 42 patients). Each resident physician completed these bilirubin management tasks for 2 to 5 patients, with every patient randomly assigned to either Bili App or BiliTool use. Participants were asked to complete their work as they usually would for making a decision regarding the patients' care needs with regard to hyperbilirubinemia screening and management. Typically, this process involved reviewing the bilirubin levels, identifying risk factors, and making a decision about appropriate next steps, including possible additional bilirubin testing, phototherapy use, and discharge. We recorded the EHR screen as the residents completed these bilirubin management tasks. Participants were asked to act as if the observer was not present. Specifically, participants were asked to not “think aloud” or describe what they were doing while performing these tasks.

    Task time was calculated from screen recordings from the moment the patient’s medical record was opened in the EHR to the moment when residents indicated that they were finished. Time savings were estimated using generalized linear regression with generalized estimating equations to account for correlation within timings from the same residents. We included subset analyses for users with different experience levels.

    Based on EHR logs for eligible newborns born in 2018, app use was measured as the number of uses and the proportion of newborns with admissions (n = 3826) and follow-up visits (n = 1932) for whom the Bili App was used. To calculate annual time savings for UUH, we multiplied the number of uses in 2018 by the mean time savings from the task-timing study.

    The preintervention period was April 1, 2016, through March 31, 2017 (n = 3714), and the postintervention period was May 1, 2017, through April 30, 2019 (n = 7520), excluding 5 newborns with missing gestational age or sex. There were no other interventions related to bilirubin management introduced in the nursery during the study timeframe.

    Reported patient characteristics include sex, gestational age less than 38 weeks, direct Coombs test results, and performance of a serum bilirubin test. Patient characteristics before and after the intervention were compared using χ2 tests. Health care use measures were length of stay, intensive care unit admissions, urgent care visits, and readmissions. Risk-specific thresholds for phototherapy (Figure, A, item 3) were determined at 12 to 144 hours of life based on whether the gestational age was less than 38 weeks and whether a positive direct Coombs test result was documented.11 The potential existence of other risk factors was not assessed for these purposes. Phototherapy use was evaluated for 3 mutually exclusive patient subsets based on whether patients (1) had a bilirubin level that was above the phototherapy threshold (hyperbilirubinemia, n = 591), (2) had a bilirubin level that was projected from the rate of rise to be above the threshold within 24 hours (projected hyperbilirubinemia, n = 557) (Figure, B, provides an example), or (3) met neither condition (nonhyperbilirubinemia, n = 9980). For patients with hyperbilirubinemia, we evaluated phototherapy ordering rates within 4 hours of the first documented serum bilirubin level result above the AAP phototherapy threshold during hospitalization. For patients with projected hyperbilirubinemia or nonhyperbilirubinemia, we evaluated phototherapy ordering rates for any time during hospitalization.

    Generalized linear models were used for the evaluation of all measures. Gamma regression was used for length of stay and logistic regression was used for all other measures. Covariates included in the model were gestational age less than 38 weeks, positive direct Coombs test results, and, for the phototherapy-ordering measures for patients with projected hyperbilirubinemia or nonhyperbilirubinemia, the shortest distance between a bilirubin level and the phototherapy threshold. Estimated marginal means and percentages were calculated at mean covariate values. Odds ratios and fold increases were estimated by exponentiating the regression coefficients. Percentage changes in odds were calculated as the odds ratio −1, then multiplied by 100.

    In February 2019, an invitation to participate in a System Usability Scale (SUS) survey was emailed to 208 clinician users who had used the Bili App in January 2019; of these, 109 users (52.40%) participated.19 The SUS survey is composed of 10 statements with responses ranging from 1 (strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100. Bangor et al20 described the adjective ratings associated with SUS scores: worst imaginable (mean SUS score, 12.5), awful (20.3), poor (35.7), okay (50.9), good (71.4), excellent (85.5), and best imaginable (90.9).

    Statistical Analysis

    Data for analyses were retrieved from the enterprise data warehouse. All statistical analyses were performed using R, version 3.5.1 (R Foundation) and 2-tailed tests. P values <.05 were considered significant. The study took place between April 1, 2016, and September 3, 2019. Data analyses were conducted from October 30, 2018, to September 23, 2019.

    Results

    The Bili App required a mean of 35 seconds (95% CI, 27-42 seconds) to complete the bilirubin management tasks compared with 100 seconds (95% CI, 89-112 seconds) with a commonly used tool (66-second time savings, 95% CI, 53-79 seconds; P < .001) (Table 1). Interns saved 81 seconds, while second and third-year residents saved 56 seconds.

    In 2018, the app was used 20 516 times (Table 2), including 17 812 times for 91.84% of births in the inpatient setting and 2704 times in outpatient settings for 39.18% of patients with follow-up visits. The app was accessed by 77 fellows and attending physicians, 107 residents, 162 registered nurses, 136 medical students, and 45 other clinicians, such as nurse practitioners and physician assistants (Table 2). Given that the app was used 20 516 times overall, we estimated 374.36 (95% CI, 301.19-447.59) hours of time savings annually for all clinicians at UUH, including 235.12 (95% CI, 189.16-281.11) hours of time savings for residents.

    Patient characteristics and outcomes pre-post intervention are summarized in Table 3. There were no significant changes in patient characteristics and health care use measures. For patients with hyperbilirubinemia, the adjusted percentage of newborns for whom phototherapy was ordered within 4 hours of the first documented inpatient serum bilirubin test result above the AAP treatment threshold increased significantly from 74.27% to 84.12% (odds ratio [OR], 1.84; 95% CI, 1.16-2.90; P = .009). For patients with projected hyperbilirubinemia, the adjusted percentage of newborns for whom phototherapy was ordered increased significantly from 27.34% to 53.41% (OR, 3.05; 95% CI, 2.01-4.62; P < .001). For patients without hyperbilirubinemia, the adjusted percentage of newborns for whom phototherapy was ordered remained stable (OR, 0.95; 95% CI, 0.68-1.33; P = .78).

    The mean SUS score indicated excellent usability (SUS, 83.90; 95% CI, 81.49-86.31) (Table 4).20 Attending physicians rated the system at 91.05 (95% CI, 86.31-95.79), resident physicians at 86.56 (95% CI, 82.14-90.98), registered nurses at 81.63 (95% CI, 81.63 77.85-85.41), and medical students at 76.25 (95% CI, 65.25-87.25). The benchmark mean SUS score across industries is 68.21 The app was recognized with multiple awards from the Department of Health and Human Services’ Provider User-Experience Challenge for EHR-integrated apps.22

    Discussion

    Electronic health records are evolving into platforms in which third-party apps add value to their users through integrated tools targeted at specific decisions and tasks. This study may provide empirical support for this vision, in which an add-on app for neonatal bilirubin management was widely used, was associated with clinicians time savings and improved guideline-compliant care, and had high perceived usability. To our knowledge, this is one of the first studies to provide real-world, long-term data suggesting that EHR add-on apps using the emerging SMART on FHIR standard can save clinicians time, improve care, and provide a positive user experience.

    The app focuses on a narrow domain and saves only a minute per use, and yet the potential for time savings if deployed on a national scale is significant. If extrapolated to the approximately 3.4 million nonpreterm births annually in the United States,23 universal use of the app could potentially save more than 300 000 hours of clinician time every year. Given the limited extent to which EHRs currently support task-optimized data retrieval and synthesis, there are likely millions of hours of clinician time each year that could be saved through the widespread deployment of similar EHR add-on apps developed through a user-centered design process.

    Beyond the time savings, another finding was the improvement in appropriate phototherapy ordering. Studies have shown that patients often do not receive recommended care.24,25 Moreover, these challenges in care quality may be associated with the time constraints that clinicians face; Yarnall et al26 found, for example, that just satisfying US Preventive Services Task Force A and B recommendations would require 7.4 hours per working day of physician time. By making it easier to provide appropriate care, tools such as the app could engender a positive cycle of clinician time savings and improved patient care.

    Through the University of Utah ReImagine EHR initiative, we now have experience with implementing a number of SMART on FHIR apps in addition to the bilirubin app. Based on this experience, we believe that several specific features of the bilirubin app have helped to make this implementation successful. These system features include those identified in a meta-analysis of the literature as being important to CDS success: provision of CDS at the time and location of decision-making, provision of recommendations rather than just assessments, integration with the EHR, and the minimal need for additional clinician data entry.12 We believe our findings are also congruent with systematic reviews of information displays for critical care, which found that comprehensive information displays that integrate information from multiple sources (eg, laboratory test results, procedures) and present trend data graphically had the strongest association with positive clinician performance and patient outcomes, possibly owing to improved pattern recognition and situational awareness.27,28 By automating low-level cognitive tasks, such as retrieving, organizing, sorting, and graphically representing data, integrated information displays let clinicians spare valuable cognitive resources for high-level and complex cognitive tasks. For example, the significantly higher odds of phototherapy orders among patients with projected hyperbilirubinemia suggest that the app may help clinicians to more efficiently identify patients for whom phototherapy may be indicated (Figure, B). Our findings are consistent with cognitive research on working memory, which has found that cognitive load can be reduced—and user performance improved—through displays that match users’ mental models29 and automate subtasks, such as information searching, that distract and reduce working memory.30

    In addition, we believe key factors in the app’s widespread adoption are the reduction in cognitive effort and associated time savings, which can provide a relief from the burden otherwise imposed by EHRs.2,3 Because the app applies to practically every newborn, its use became almost universal in the newborn nursery. As such, the app was able to automatically provide CDS as a part of routine clinician workflow, which is a factor that has repeatedly been found to be critical to CDS success.31,32

    We are actively working to disseminate the app as a free tool through EHR app stores and hope to report on the adoption and outcome of the app’s use at other institutions. We are also developing numerous additional EHR add-on apps through the University of Utah ReImagine EHR initiative to optimize patient care and the clinicians’ EHR experience. Further research is also needed on how to best support clinical decision-making through EHR add-on apps, such as through the application of cognitive science principles and CDS best practices.

    Limitations

    This study has limitations. Although the app uses a standards-based approach that should be deployable across health care systems and EHR products, this was a single-center study, and further studies are needed to demonstrate generalizability. The pre-post intervention design could be confounded by secular trends; however, we used multivariate, generalized linear models to adjust for covariates. Also, for the health care use measures (length of stay, intensive care unit admission, urgent care visit, and readmission), we did not limit our analyses to events related to hyperbilirubinemia (eg, intensive care unit admissions due to hyperbilirubinemia). Conversely, the task-timing study was limited to inpatient residents, such that overall time savings may differ from our estimates if other types of clinicians save more or less time from use of the app. Another limitation is that local users were actively involved in the development of the intervention—a factor known to correlate with CDS success12 and thus potentially overestimate the outcome that can be expected at other institutions. However, most users of the app are residents, who rotate in and out of the nursery service on a regular basis and had no involvement in the development of the app. In addition, while EHR add-on apps are a promising solution to support clinicians in complex cognitive tasks, they are unlikely to replace more transactional EHR functionality that is tightly coupled with the EHR, such as order entry. Where possible, native EHR functionality should be improved to preserve a cohesive user experience. To this end, as EHR add-on apps grow in their prevalence, it will become increasingly important for SMART on FHIR apps and the underlying EHRs to adopt common user interface conventions.

    Conclusions

    The findings of this study suggest that well-designed EHR add-on-apps can save clinicians time, improve care, and enhance the EHR user experience by supporting complex decision tasks. Further research is needed to evaluate the generalizability of these findings in other health care domains and settings. If well-designed EHR add-on apps were widely implemented across health care organizations, the potential for improved patient care and clinician efficiency could be significant.

    Back to top
    Article Information

    Accepted for Publication: September 24, 2019.

    Published: November 15, 2019. doi:10.1001/jamanetworkopen.2019.15343

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Kawamoto K et al. JAMA Network Open.

    Corresponding Author: Kensaku Kawamoto, MD, PhD, MHS, Department of Biomedical Informatics, University of Utah, 421 Wakara Way, Ste 108, Salt Lake City, UT 84108 (kensaku.kawamoto@utah.edu).

    Author Contributions: Dr Kawamoto had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: All authors.

    Acquisition, analysis, or interpretation of data: Kukhareva, Shakib, Kramer, Shields, Weir, Del Fiol, Taft, Stipelman.

    Drafting of the manuscript: Kawamoto, Kukhareva, Shields, Weir.

    Critical revision of the manuscript for important intellectual content: Kawamoto, Kukhareva, Shakib, Kramer, Rodriguez, Warner, Weir, Del Fiol, Taft, Stipelman.

    Statistical analysis: Kukhareva, Weir.

    Obtained funding: Kawamoto.

    Administrative, technical, or material support: Rodriguez, Warner, Shields, Weir.

    Supervision: Kawamoto, Weir, Del Fiol, Stipelman.

    Conflict of Interest Disclosures: Dr Kawamoto reported receiving awards and associated cash prizes for the Bili App from the US Department of Health and Human Services (HHS) Provider User-Experience Challenge during the conduct of the study; grants and personal fees from Hitachi, personal fees from McKesson InterQual, Premier, Klesis Healthcare, Research Triangle Institute, the Mayo Clinic, Vanderbilt University, University of Washington, University of California at San Francisco, and the US Office of the National Coordinator for Health IT (via Enterprise Science and Computing, JBS International, A+ Government Solutions, Hausam Consulting, and Security Risk Solutions) outside the submitted work. All authors reported receiving awards and associated cash prizes for the Bili App from the HHS Provider User-Experience Challenge during the conduct of the study. Mr Warner reported receiving personal fees from Phast-Services during the conduct of the study. No other disclosures were reported.

    Funding/Support: This work was funded by the University of Utah.

    Role of the Funder/Sponsor: The University of Utah had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Information: The app was recognized with multiple awards and associated cash prizes from the Department of Health and Human Services’ Provider User-Experience Challenge for electronic health records-integrated apps.

    References
    1.
    Schulte  F, Fry  E. Death by a thousand clicks: where electronic health records went wrong. Kaiser Health News. https://khn.org/news/death-by-a-thousand-clicks/. Published March 18, 2019. Accessed June 21, 2019.
    2.
    Sinsky  C, Colligan  L, Li  L,  et al.  Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties.  Ann Intern Med. 2016;165(11):753-760. doi:10.7326/M16-0961PubMedGoogle ScholarCrossref
    3.
    Chaiyachati  KH, Shea  JA, Asch  DA,  et al.  Assessment of inpatient time allocation among first-year internal medicine residents using time-motion observations.  JAMA Intern Med. 2019;179(6):760-767. doi:10.1001/jamainternmed.2019.0095PubMedGoogle ScholarCrossref
    4.
    Gardner  RL, Cooper  E, Haskell  J,  et al.  Physician stress and burnout: the impact of health information technology.  J Am Med Inform Assoc. 2019;26(2):106-114. doi:10.1093/jamia/ocy145PubMedGoogle ScholarCrossref
    5.
    Mandl  KD, Mandel  JC, Kohane  IS.  Driving innovation in health systems through an apps-based information economy.  Cell Syst. 2015;1(1):8-13. doi:10.1016/j.cels.2015.05.001PubMedGoogle ScholarCrossref
    6.
    Mandl  KD, Kohane  IS.  No small change for the health information economy.  N Engl J Med. 2009;360(13):1278-1281. doi:10.1056/NEJMp0900411PubMedGoogle ScholarCrossref
    7.
    Mandel  JC, Kreda  DA, Mandl  KD, Kohane  IS, Ramoni  RB.  SMART on FHIR: a standards-based, interoperable apps platform for electronic health records.  J Am Med Inform Assoc. 2016;23(5):899-908. doi:10.1093/jamia/ocv189PubMedGoogle ScholarCrossref
    8.
    Health Level Seven International (HL7). SMART App Launch Framework. http://www.hl7.org/fhir/smart-app-launch/. Accessed September 23, 2019.
    9.
    Health Level Seven International (HL7). Fast Healthcare Interoperability Resources (FHIR) standard. https://www.hl7.org/fhir/. Accessed September 23, 2019.
    10.
    Twichell  SA, Rea  CJ, Melvin  P,  et al.  The effect of an electronic health record–based tool on abnormal pediatric blood pressure recognition.  Congenit Heart Dis. 2017;12(4):484-490. doi:10.1111/chd.12469PubMedGoogle ScholarCrossref
    11.
    American Academy of Pediatrics Subcommittee on Hyperbilirubinemia.  Management of hyperbilirubinemia in the newborn infant 35 or more weeks of gestation.  Pediatrics. 2004;114(1):297-316. doi:10.1542/peds.114.1.297PubMedGoogle ScholarCrossref
    12.
    Lobach  D, Sanders  GD, Bright  TJ,  et al.  Enabling health care decisionmaking through clinical decision support and knowledge management.  Evid Rep Technol Assess (Full Rep). 2012;(203):1-784.PubMedGoogle Scholar
    13.
    Ogrinc  G, Davies  L, Goodman  D, Batalden  P, Davidoff  F, Stevens  D.  SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process.  BMJ Qual Saf. 2016;25:986-992. doi:10.1136/bmjqs-2015-004411PubMedGoogle ScholarCrossref
    14.
    Longhurst  C, Turner  S, Burgos  AE.  Development of a Web-based decision support tool to increase use of neonatal hyperbilirubinemia guidelines.  Jt Comm J Qual Patient Saf. 2009;35(5):256-262. doi:10.1016/S1553-7250(09)35035-7PubMedGoogle ScholarCrossref
    15.
    Thornton  SN, Thompson  BS, Millar  JA, Eggert  LD, Wilcox  AB. Neonatal bilirubin management as an implementation example of interdisciplinary continuum of care tools. AMIA. Annual Symposium Proceedings. 2007;2007:726-730.
    16.
    SMART App Gallery. Intermountain Healthcare bilirubin chart. https://apps.smarthealthit.org/app/bilirubin-chart. Accessed July 10, 2019.
    17.
    US Core FHIR Profiles. (Release 1.0.1). Health Level Seven International (HL7). http://hl7.org/fhir/us/core/1.0.1/. Accessed July 10, 2019.
    18.
    Chang  PW, Kuzniewicz  MW, McCulloch  CE, Newman  TB.  A clinical prediction rule for rebound hyperbilirubinemia following inpatient phototherapy.  Pediatrics. 2017;139(3):e20162896. doi:10.1542/peds.2016-2896PubMedGoogle Scholar
    19.
    Brooke  J. SUS—a quick and dirty usability scale. In: Usability Evaluation in Industry. Vol 189. 1996:4-7. https://cui.unige.ch/isi/icle-wiki/_media/ipm:test-suschapt.pdf. Accessed October 15, 2019.
    20.
    Bangor  A, Kortum  P, Miller  J.  Determining what individual SUS scores mean: adding an adjective rating scale.  J Usability Stud. 2009;4(3):114-123. http://uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS_Bangor_May2009.pdf. Accessed October 15, 2019.Google Scholar
    21.
    Sauro  J. SUStisfied? Little-known system usability scale facts. User Exp Mag. 2011;10(3). https://uxpamagazine.org/sustified/. Accessed September 23, 2019.
    22.
    Office of the National Coordinator for Health Information Technology. Provider user-experience challenge. https://www.healthit.gov/techlab/ipg/node/4/submission/2131. Published 2017. Accessed July 10, 2019.
    23.
    Centers for Disease Control and Prevention. National Vital Statistics System—Birth Data. https://www.cdc.gov/nchs/nvss/births.htm. Accessed June 21, 2019.
    24.
    Levine  DM, Linder  JA, Landon  BE.  The quality of outpatient care delivered to adults in the United States, 2002 to 2013.  JAMA Intern Med. 2016;176(12):1778-1790. doi:10.1001/jamainternmed.2016.6217PubMedGoogle ScholarCrossref
    25.
    McGlynn  EA, Asch  SM, Adams  J,  et al.  The quality of health care delivered to adults in the United States.  N Engl J Med. 2003;348(26):2635-2645. doi:10.1056/NEJMsa022615PubMedGoogle ScholarCrossref
    26.
    Yarnall  KSH, Pollak  KI, Østbye  T, Krause  KM, Michener  JL.  Primary care: is there enough time for prevention?  Am J Public Health. 2003;93(4):635-641. doi:10.2105/AJPH.93.4.635PubMedGoogle ScholarCrossref
    27.
    Wright  MC, Borbolla  D, Waller  RG,  et al.  Critical care information display approaches and design frameworks: a systematic review and meta-analysis.  J Biomed Inform X. 2019;3:3. doi:10.1016/j.yjbinx.2019.100041PubMedGoogle Scholar
    28.
    Waller  RG, Wright  MC, Segall  N,  et al.  Novel displays of patient information in critical care settings: a systematic review.  J Am Med Inform Assoc. 2019;26(5):479-489. doi:10.1093/jamia/ocy193PubMedGoogle ScholarCrossref
    29.
    Ericsson  KA. Superior working memory in experts. In:  The Cambridge Handbook of Expertise and Expert Performance. Cambridge, UK: Cambridge University Press; 2018:696-713. doi:10.1017/9781316480748.036
    30.
    Camos  V.  Domain-specific versus domain-general maintenance in working memory: reconciliation within the time-based resource sharing model.  Prog Brain Res. 2017;232:135-171. doi:10.1016/bs.plm.2017.03.005Google Scholar
    31.
    Van de Velde  S, Heselmans  A, Delvaux  N,  et al.  A systematic review of trials evaluating success factors of interventions with computerised clinical decision support.  Implement Sci. 2018;13(1):114. doi:10.1186/s13012-018-0790-1PubMedGoogle ScholarCrossref
    32.
    Kawamoto  K, Houlihan  CA, Balas  EA, Lobach  DF.  Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success.  BMJ. 2005;330(7494):765. doi:10.1136/bmj.38398.500764.8FPubMedGoogle ScholarCrossref
    ×