Association of an Electronic Health Record Add-on App for Neonatal Bilirubin Management With Physician Efficiency and Care Quality

IMPORTANCE The usefulness of electronic health record (EHR) systems could be significantly enhanced by innovative, third-party EHR add-on apps. OBJECTIVE To evaluate whether an EHR add-on app for neonatal bilirubin management can save clinicians time and improve patient care. DESIGN, SETTING, AND PARTICIPANTS This quality improvement study was conducted at the University of Utah Health Well Baby nursery and outpatient clinics and consisted of 4 substudies: (1) time savings were estimated in an experimental task-timing study comparing the time required for physicians to manage newborns’ bilirubin levels with and without the add-on app, (2) app use was estimated from app logs, (3) health care use measures and guideline compliance were compared retrospectively before and after the intervention, and (4) clinician-perceived usability was measured through System Usability Scale surveys. The study took place between April 1, 2016, and September 3, 2019. Data analyses were conducted from October 30, 2018, to September 23, 2019. INTERVENTIONS At baseline, clinicians used a manual approach to ensure compliance with an evidence-based clinical guideline for neonatal bilirubin management. To facilitate guideline compliance, an EHR add-on app that automatically retrieves, organizes, and visualizes relevant patient data was developed. The app provides patient-specific assessments and recommendations, including the risk of rebound hyperbilirubinemia following phototherapy based on a predictive model. The add-on app was integrated with the University of Utah Health EHR on April 12, 2017. MAIN OUTCOMES AND MEASURES Clinician time savings, app use, health care use measures, guideline-compliant phototherapy ordering, and perceived usability as measured by the System Usability Scale survey. The survey is composed of 10 statements with responses ranging from 1 (strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100, with ratings described as worst imaginable (mean System Usability Scale score, 12.5), awful (20.3), poor (35.7), okay (50.9), good (71.4), excellent (85.5), and best imaginable (90.9). RESULTS In 2018, the application was used 20 516 times by clinicians for 91.84% of eligible newborns. Use of the app saved 66 seconds for bilirubin management tasks compared with a commonly used tool (95% CI, 53-79 seconds; P < .001). Following the intervention, health care use rates remained stable, while orders for clinically appropriate phototherapy during hospitalization increased for newborns with bilirubin levels above the guideline-recommended threshold (odds ratio, 1.84; 95% CI, 1.16-2.90; P = .009). Surveys indicated excellent usability (System Usability Scale score, 83.90; 95% CI, 81.49-86.31). (continued) Key Points Question Is an electronic health record add-on app for neonatal bilirubin management associated with time savings for clinicians and improved quality of care? Findings In this quality improvement study, an electronic health record add-on app for neonatal bilirubin management saved clinicians a mean of 66 seconds for bilirubin management tasks compared with a commonly used tool. In a retrospective pre-post analysis, the odds of clinically appropriate phototherapy orders during hospitalization increased significantly by 84%. Meaning These findings suggest that well-designed electronic health record add-on apps may be associated with time savings for physicians and improvements in patient care. Author affiliations and article information are listed at the end of this article. Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2019;2(11):e1915343. doi:10.1001/jamanetworkopen.2019.15343 (Reprinted) November 15, 2019 1/11 Downloaded From: https://jamanetwork.com/ by a Non-Human Traffic (NHT) User on 09/29/2021 Abstract (continued)continued) CONCLUSIONS AND RELEVANCE Well-designed EHR add-on apps may save clinicians time and improve patient care. If time-saving apps, such as the bilirubin app, were implemented widely across institutions and care domains, the potential association with improved patient care and clinician efficiency could be significant. The University of Utah Health bilirubin app is being prepared for release into EHR app stores as free-to-use software. JAMA Network Open. 2019;2(11):e1915343. doi:10.1001/jamanetworkopen.2019.15343


Introduction
Despite billions of dollars invested, electronic health records (EHRs) often fall short in supporting efficient, high-quality patient care. 1 Ambulatory care physicians can spend 2 hours on EHR and desk work for every hour spent in direct clinical face time, 2 and inpatient physicians can spend 5 hours on such tasks for every hour spent in direct patient care. 3 In a statewide survey of Rhode Island physicians conducted in 2017, close to two-thirds of physicians reported that use of EHRs added to their daily frustration. 4 Through emerging technology standards, EHRs now allow the integration of add-on apps, whereby third-party app developers can deliver innovative solutions that enhance the utility of EHRs for targeted health care tasks and domains. 5,6 Just as smartphones deliver a better user experience through a variety of add-on apps, the hope is that EHRs can deliver improved user experiences through such apps. Unlike EHR vendors, who may be overwhelmed with competing priorities, such as maintaining legacy functionality and providing support to thousands of customers for a wide range of clinical and administrative tasks, a third-party app developer can focus entirely on optimizing a comparatively small range of decision tasks. Apps also use a technical framework designed to facilitate substituting one app for another, whereas switching EHRs is a disruptive and costly process.
Thus, app developers may be inherently incentivized to optimize patient care and the user experience in their domain of focus, whereas EHR vendors are unlikely to gain or lose a customer based on their level of support for any specific decision task. Also, unlike EHR vendors, who may have years-long cycles for feature prioritization and development, third-party app developers can adopt a much more rapid and iterative development approach, with the ability to develop and release enhancements much more quickly based on user feedback.
The technology enabling EHR add-on apps is known as Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources (SMART on FHIR; pronounced smart on fire). 7 Apps enabled by SMART on FHIR use the Health Level Seven International (HL7) SMART standard 8 to enable single sign-on and integration with the EHR user interface, as well as the HL7 FHIR data interface standard 9 to pull in relevant patient data automatically. Although the evidence of the influence of SMART on FHIR apps on clinical outcomes is limited, investigators at Boston Children's Hospital showed in 2017 that the introduction of a SMART on FHIR app for pediatric blood pressure visualization was associated with an increase in the recognition of abnormal blood pressure (7.1% vs 4.9%; P < .001). 10 The present study reports on an evaluation of an EHR add-on app that was implemented in 2017 at an academic medical center to support the American Academy of Pediatrics (AAP) guideline on the management of neonatal hyperbilirubinemia. 11 To manage newborns' care according to this guideline, the clinician must retrieve disparate data scattered across the medical record, synthesize the data for risk classification, and apply guideline algorithms to identify patient-specific care needs, such as the administration of phototherapy when bilirubin levels exceed risk-based treatment thresholds. 11 The app was designed to support these tasks by retrieving relevant data, providing a visual data summary, and delivering guideline-based recommendations on next steps. We also designed the app to support known success factors for clinical decision support (CDS) systems, including providing CDS at the time and location of decision-making, providing recommendations rather than just assessments, integrating with the EHR, and minimizing the need for additional clinician data entry. 12 The goal of this study was to evaluate our hypothesis that this EHR add-on app would save clinicians time and improve care quality.

Study Design
This was a mixed-methods quality improvement study that included 4 separate substudies: (1)  The resulting app ( Figure) supports neonatal bilirubin management according to the AAP guideline 11 and estimates the risk of postphototherapy rebound hyperbilirubinemia. 18 The app was   Clinicians can also evaluate the influence of potential changes on patient risk factors, such as if the patient's gestational age at birth was 37 weeks 6 days rather than 38 weeks 0 days (Figure, A, item 5).

Substudies
In a time-savings evaluation, 12 pediatric and family medicine resident physicians on service in the nursery (including 7 interns, 3-second-year residents, and 2 third-year residents) were recruited to complete bilirubin management tasks for their current patients (n = 42 patients). Each resident physician completed these bilirubin management tasks for 2 to 5 patients, with every patient randomly assigned to either Bili App or BiliTool use. Participants were asked to complete their work as they usually would for making a decision regarding the patients' care needs with regard to hyperbilirubinemia screening and management. Typically, this process involved reviewing the bilirubin levels, identifying risk factors, and making a decision about appropriate next steps, including possible additional bilirubin testing, phototherapy use, and discharge. We recorded the EHR screen as the residents completed these bilirubin management tasks. Participants were asked to act as if the observer was not present. Specifically, participants were asked to not "think aloud" or describe what they were doing while performing these tasks.
Task time was calculated from screen recordings from the moment the patient's medical record was opened in the EHR to the moment when residents indicated that they were finished. Time savings were estimated using generalized linear regression with generalized estimating equations to account for correlation within timings from the same residents. We included subset analyses for users with different experience levels.  Odds ratios and fold increases were estimated by exponentiating the regression coefficients.

JAMA Network Open | Health Informatics
Percentage changes in odds were calculated as the odds ratio −1, then multiplied by 100.
In February 2019, an invitation to participate in a System Usability Scale (SUS) survey was emailed to 208 clinician users who had used the Bili App in January 2019; of these, 109 users (52.40%) participated. 19 The SUS survey is composed of 10 statements with responses ranging from 1 (strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100.

Statistical Analysis
Data for analyses were retrieved from the enterprise data warehouse. All statistical analyses were performed using R, version 3.5.

Discussion
Electronic health records are evolving into platforms in which third-party apps add value to their users through integrated tools targeted at specific decisions and tasks. This study may provide empirical support for this vision, in which an add-on app for neonatal bilirubin management was widely used, was associated with clinicians time savings and improved guideline-compliant care, and had high perceived usability. To our knowledge, this is one of the first studies to provide real-world, long-term data suggesting that EHR add-on apps using the emerging SMART on FHIR standard can making, provision of recommendations rather than just assessments, integration with the EHR, and the minimal need for additional clinician data entry. 12 We believe our findings are also congruent with systematic reviews of information displays for critical care, which found that comprehensive information displays that integrate information from multiple sources (eg, laboratory test results, procedures) and present trend data graphically had the strongest association with positive clinician performance and patient outcomes, possibly owing to improved pattern recognition and situational awareness. 27,28 By automating low-level cognitive tasks, such as retrieving, organizing, sorting, and graphically representing data, integrated information displays let clinicians spare valuable cognitive resources for high-level and complex cognitive tasks. For example, the significantly higher odds of phototherapy orders among patients with projected hyperbilirubinemia suggest that the app may help clinicians to more efficiently identify patients for whom phototherapy may be indicated (Figure, B). Our findings are consistent with cognitive research on working memory, which has found that cognitive load can be reduced-and user performance improved-through displays that match users' mental models 29 and automate subtasks, such as information searching, that distract and reduce working memory. 30 In addition, we believe key factors in the app's widespread adoption are the reduction in cognitive effort and associated time savings, which can provide a relief from the burden otherwise imposed by EHRs. 2,3 Because the app applies to practically every newborn, its use became almost universal in the newborn nursery. As such, the app was able to automatically provide CDS as a part of routine clinician workflow, which is a factor that has repeatedly been found to be critical to CDS success. 31,32 We are actively working to disseminate the app as a free tool through EHR app stores and hope to report on the adoption and outcome of the app's use at other institutions. We are also developing numerous additional EHR add-on apps through the University of Utah ReImagine EHR initiative to optimize patient care and the clinicians' EHR experience. Further research is also needed on how to best support clinical decision-making through EHR add-on apps, such as through the application of cognitive science principles and CDS best practices.

Limitations
This study has limitations. Although the app uses a standards-based approach that should be deployable across health care systems and EHR products, this was a single-center study, and further studies are needed to demonstrate generalizability. The pre-post intervention design could be confounded by secular trends; however, we used multivariate, generalized linear models to adjust for covariates. Also, for the health care use measures (length of stay, intensive care unit admission, urgent care visit, and readmission), we did not limit our analyses to events related to hyperbilirubinemia (eg, intensive care unit admissions due to hyperbilirubinemia). Conversely, the task-timing study was limited to inpatient residents, such that overall time savings may differ from our estimates if other types of clinicians save more or less time from use of the app. Another limitation is that local users were actively involved in the development of the intervention-a factor known to correlate with CDS success 12 and thus potentially overestimate the outcome that can be expected at other institutions. However, most users of the app are residents, who rotate in and out of the nursery service on a regular basis and had no involvement in the development of the app. In addition, while EHR add-on apps are a promising solution to support clinicians in complex cognitive tasks, they are unlikely to replace more transactional EHR functionality that is tightly coupled with the EHR, such as order entry. Where possible, native EHR functionality should be improved to preserve a cohesive user experience. To this end, as EHR add-on apps grow in their prevalence, it will become increasingly important for SMART on FHIR apps and the underlying EHRs to adopt common user interface conventions.

Conclusions
The findings of this study suggest that well-designed EHR add-on-apps can save clinicians time, improve care, and enhance the EHR user experience by supporting complex decision tasks. Further research is needed to evaluate the generalizability of these findings in other health care domains and settings. If well-designed EHR add-on apps were widely implemented across health care organizations, the potential for improved patient care and clinician efficiency could be significant.