Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic

Sepsis early warning systems aim to assist clinicians in recognizing and treating sepsis. Historically, these early warning systems have relied on simple clinical rules, such as systemic inflammatory response syndrome criteria, to identify patients with possible sepsis. To date, sepsis early warning systems have not been shown to reliably improve patient outcomes,¹ and artificial intelligence (AI) systems such as the widely implemented Epic Sepsis Model (ESM) are beginning to replace them.

Concerns have arisen recently regarding the potential for sepsis models to cause alert fatigue.^2-4 Between 3 and 4 weeks after its first COVID-19 hospitalization, the University of Michigan paused ESM-generated alerts in April 2020 after nursing reports of overalerting.⁵

This increase in alerting could have resulted from dataset shift, a phenomenon in which model performance deteriorates as a result of changes in the case mix (eg, COVID-19).⁵ However, even accurate alerts can be disruptive in the presence of resource constraints. For example, more than a quarter of hospitalized patients in the first wave of the COVID-19 pandemic in 2020 required a transfer to an intensive care unit or died.⁶

We quantified the number of alerts generated by the ESM at 24 hospitals in the months before and during the COVID-19 pandemic to (1) evaluate the extent to which nursing reports of sepsis overalerting were linked to the alert volume and (2) examine the variation in alert volume across US hospitals.

This descriptive study was approved by the institutional review boards at the University of Michigan, Washington University, and Mass General Brigham and was considered to be nonregulated at the New York University Grossman School of Medicine. The need for consent was waived because the research involved no more than minimal risk to participants, the research could not be carried out practicably without the waiver, and the waiver would not adversely affect the rights and welfare of the participants.

ESM scores were calculated prospectively from 24 hospitals across 4 geographically diverse health systems (University of Michigan in Ann Arbor, Michigan; New York University Langone Health in New York, New York; Mass General Brigham in Boston, Massachusetts; and BJC HealthCare in St Louis, Missouri) from November 3, 2019, to April 25, 2020. These scores were aligned to the hospitals’ first case of COVID-19. We compared the total hospital census, the proportion of patients generating sepsis alerts, and the frequency of sepsis alerts using data before and during the COVID-19 pandemic, with an alerting threshold of 6 and a maximum of 1 alert per patient per day. Further details are provided in the eMethods in the Supplement.

In the 3 weeks before and after the first case of COVID-19 in each US health system in this study, the proportion of patients generating sepsis alerts per day more than doubled from 9% (953 of 10 159) to 21% (1363 of 6634), respectively (Table). However, the total hospital census declined by 35% (from 10 159 to 6634). The total number of alerts per day increased by 43% (from 953 to 1363) despite the lower hospital census.

Larger hospitals generally experienced an increase in the proportion of patients generating sepsis alerts, whereas the change in the alerting proportion was more heterogeneous across smaller hospitals (Figure). The pausing of sepsis alerts at the University of Michigan corresponded with the increase in alerting (Figure).

Although the increase in the proportion of patients generating sepsis alerts in this study can be explained by the cancellation of elective surgeries and a higher average acuity among the remaining hospitalized patients, the 43% increase in total alerts illustrates the increased alerting burden imposed by COVID-19 on a sepsis model.

Our study was limited in that we did not evaluate the model’s accuracy. However, even if the alerts were accurate, many existing sepsis workflows are built around bacterial sepsis and thus may not be entirely appropriate in the context of COVID-19.

Being able to rapidly assess and disable AI-based alerts is a responsibility faced by health systems using AI to support clinical care. Given the susceptibility of AI-based systems to changes in alerting patterns, clinical AI governance within health systems may play a role in monitoring and supporting deployed AI systems.

Accepted for Publication: September 26, 2021.

Published: November 19, 2021. doi:10.1001/jamanetworkopen.2021.35286

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Wong A et al. JAMA Network Open.

Corresponding Author: Karandeep Singh, MD, MMSc, Department of Learning Health Sciences, University of Michigan Medical School, 1161H NIB, 300 N Ingalls St, Ann Arbor, MI 48109 (kdpsingh@umich.edu).

Author Contributions: Dr Singh had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Dr Wong and Ms Cao contributed equally to this work.

Concept and design: Wong, Cao, Lyons, Dutta, Ötleş, Singh.

Acquisition, analysis, or interpretation of data: Wong, Cao, Lyons, Dutta, Major, Singh.

Drafting of the manuscript: Wong, Cao, Major.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Wong, Cao, Lyons, Singh.

Administrative, technical, or material support: Wong, Ötleş.

Supervision: Dutta, Singh.

Conflict of Interest Disclosures: Dr Lyons reported receiving grants from the National Institutes of Health National Center for Advancing Translational Sciences and the Doris Duke Charitable Foundation and the Big Ideas Award from BJC HealthCare and Washington University. Mr Ötleş reported having a patent pending for the University of Michigan for an artificial intelligence–based approach for the dynamic prediction of the injured patient health state. Dr Singh reported receiving grants from Teva Pharmaceuticals and Blue Cross Blue Shield of Michigan. No other disclosures were reported.

Funding/Support: This study was supported by the University of Michigan Precision Health (Ms Cao), grant KL2TR002346 from the National Institutes of Health National Center for Advancing Translational Sciences (Dr Lyons), the Doris Duke Charitable Foundation Fund to Retain Clinical Scientists (Dr Lyons), and grant T32GM007863 from the National Institutes of Health National Institute of General Medical Sciences (to Mr Ötleş).

Role of the Funder/Sponsor: The University of Michigan Precision Health, the National Institutes of Health, and the Doris Duke Charitable Foundation had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.