Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic | Critical Care Medicine | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure.  Comparison of Total Hospital Census and Proportion of Patients Generating Sepsis Alerts at Each Hospital, Aligned by Hospitals’ First Case of COVID-19
Comparison of Total Hospital Census and Proportion of Patients Generating Sepsis Alerts at Each Hospital, Aligned by Hospitals’ First Case of COVID-19

Each hospital is depicted by a line; darker lines indicate higher hospital bed capacity. The thick black line represents the average across all hospitals. Calculations are averaged for each calendar week; day 0 represents the calendar week of the first case of COVID-19, beginning with day 0 and ending at day 6. The vertical dashed brown line depicts day 0, and the vertical dashed orange line depicts the day the Epic Sepsis Model was paused at the University of Michigan. Further details are provided in the eMethods in the Supplement.

Table.  Comparison of Total Hospital Census, Total Number of Alerts per Day, and Proportion of Patients Generating Alerts Before and After Hospitals’ First Case of COVID-19a
Comparison of Total Hospital Census, Total Number of Alerts per Day, and Proportion of Patients Generating Alerts Before and After Hospitals’ First Case of COVID-19a
1.
Downing  NL, Rolnick  J, Poole  SF,  et al.  Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation.   BMJ Qual Saf. 2019;28(9):762-768. doi:10.1136/bmjqs-2018-008765 PubMedGoogle ScholarCrossref
2.
Wong  A, Otles  E, Donnelly  JP,  et al.  External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients.   JAMA Intern Med. 2021;181(8):1065-1070. doi:10.1001/jamainternmed.2021.2626 PubMedGoogle ScholarCrossref
3.
Ginestra  JC, Giannini  HM, Schweickert  WD,  et al.  Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock.   Crit Care Med. 2019;47(11):1477-1484. doi:10.1097/CCM.0000000000003803 PubMedGoogle ScholarCrossref
4.
Eilish  MC, Watkins  EA. Repairing innovation: a study of integrating AI in clinical care. Data & Society Research Institute. Published September 2020. Accessed August 5, 2021. https://datasociety.net/wp-content/uploads/2020/09/Repairing-Innovation-DataSociety-20200930-1.pdf
5.
Finlayson  SG, Subbaswamy  A, Singh  K,  et al.  The clinician and dataset shift in artificial intelligence.   N Engl J Med. 2021;385(3):283-286. doi:10.1056/NEJMc2104626 PubMedGoogle ScholarCrossref
6.
Singh  K, Valley  TS, Tang  S,  et al  Evaluating a widely implemented proprietary deterioration index model among hospitalized patients with COVID-19.   Ann Am Thorac Soc. 2021;18(7):1129-1137. doi:10.1513/AnnalsATS.202006-698OCGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 2,690
    Citations 0
    Research Letter
    Health Informatics
    November 19, 2021

    Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic

    Author Affiliations
    • 1Department of Internal Medicine, University of Michigan Medical School, Ann Arbor
    • 2Department of Computational Medicine and Bioinformatics, University of Michigan Medical School, Ann Arbor
    • 3John T. Milliken Department of Medicine, Washington University School of Medicine, St Louis, Missouri
    • 4Department of Emergency Medicine, Massachusetts General Hospital, Boston
    • 5Department of Population Health, New York University Grossman School of Medicine, New York
    • 6Medical Scientist Training Program, University of Michigan, Ann Arbor
    • 7Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor
    • 8Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor
    • 9School of Information, University of Michigan, Ann Arbor
    JAMA Netw Open. 2021;4(11):e2135286. doi:10.1001/jamanetworkopen.2021.35286
    Introduction

    Sepsis early warning systems aim to assist clinicians in recognizing and treating sepsis. Historically, these early warning systems have relied on simple clinical rules, such as systemic inflammatory response syndrome criteria, to identify patients with possible sepsis. To date, sepsis early warning systems have not been shown to reliably improve patient outcomes,1 and artificial intelligence (AI) systems such as the widely implemented Epic Sepsis Model (ESM) are beginning to replace them.

    Concerns have arisen recently regarding the potential for sepsis models to cause alert fatigue.2-4 Between 3 and 4 weeks after its first COVID-19 hospitalization, the University of Michigan paused ESM-generated alerts in April 2020 after nursing reports of overalerting.5

    This increase in alerting could have resulted from dataset shift, a phenomenon in which model performance deteriorates as a result of changes in the case mix (eg, COVID-19).5 However, even accurate alerts can be disruptive in the presence of resource constraints. For example, more than a quarter of hospitalized patients in the first wave of the COVID-19 pandemic in 2020 required a transfer to an intensive care unit or died.6

    We quantified the number of alerts generated by the ESM at 24 hospitals in the months before and during the COVID-19 pandemic to (1) evaluate the extent to which nursing reports of sepsis overalerting were linked to the alert volume and (2) examine the variation in alert volume across US hospitals.

    Methods

    This descriptive study was approved by the institutional review boards at the University of Michigan, Washington University, and Mass General Brigham and was considered to be nonregulated at the New York University Grossman School of Medicine. The need for consent was waived because the research involved no more than minimal risk to participants, the research could not be carried out practicably without the waiver, and the waiver would not adversely affect the rights and welfare of the participants.

    ESM scores were calculated prospectively from 24 hospitals across 4 geographically diverse health systems (University of Michigan in Ann Arbor, Michigan; New York University Langone Health in New York, New York; Mass General Brigham in Boston, Massachusetts; and BJC HealthCare in St Louis, Missouri) from November 3, 2019, to April 25, 2020. These scores were aligned to the hospitals’ first case of COVID-19. We compared the total hospital census, the proportion of patients generating sepsis alerts, and the frequency of sepsis alerts using data before and during the COVID-19 pandemic, with an alerting threshold of 6 and a maximum of 1 alert per patient per day. Further details are provided in the eMethods in the Supplement.

    Results

    In the 3 weeks before and after the first case of COVID-19 in each US health system in this study, the proportion of patients generating sepsis alerts per day more than doubled from 9% (953 of 10 159) to 21% (1363 of 6634), respectively (Table). However, the total hospital census declined by 35% (from 10 159 to 6634). The total number of alerts per day increased by 43% (from 953 to 1363) despite the lower hospital census.

    Larger hospitals generally experienced an increase in the proportion of patients generating sepsis alerts, whereas the change in the alerting proportion was more heterogeneous across smaller hospitals (Figure). The pausing of sepsis alerts at the University of Michigan corresponded with the increase in alerting (Figure).

    Discussion

    Although the increase in the proportion of patients generating sepsis alerts in this study can be explained by the cancellation of elective surgeries and a higher average acuity among the remaining hospitalized patients, the 43% increase in total alerts illustrates the increased alerting burden imposed by COVID-19 on a sepsis model.

    Our study was limited in that we did not evaluate the model’s accuracy. However, even if the alerts were accurate, many existing sepsis workflows are built around bacterial sepsis and thus may not be entirely appropriate in the context of COVID-19.

    Being able to rapidly assess and disable AI-based alerts is a responsibility faced by health systems using AI to support clinical care. Given the susceptibility of AI-based systems to changes in alerting patterns, clinical AI governance within health systems may play a role in monitoring and supporting deployed AI systems.

    Back to top
    Article Information

    Accepted for Publication: September 26, 2021.

    Published: November 19, 2021. doi:10.1001/jamanetworkopen.2021.35286

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Wong A et al. JAMA Network Open.

    Corresponding Author: Karandeep Singh, MD, MMSc, Department of Learning Health Sciences, University of Michigan Medical School, 1161H NIB, 300 N Ingalls St, Ann Arbor, MI 48109 (kdpsingh@umich.edu).

    Author Contributions: Dr Singh had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Dr Wong and Ms Cao contributed equally to this work.

    Concept and design: Wong, Cao, Lyons, Dutta, Ötleş, Singh.

    Acquisition, analysis, or interpretation of data: Wong, Cao, Lyons, Dutta, Major, Singh.

    Drafting of the manuscript: Wong, Cao, Major.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Wong, Cao, Lyons, Singh.

    Administrative, technical, or material support: Wong, Ötleş.

    Supervision: Dutta, Singh.

    Conflict of Interest Disclosures: Dr Lyons reported receiving grants from the National Institutes of Health National Center for Advancing Translational Sciences and the Doris Duke Charitable Foundation and the Big Ideas Award from BJC HealthCare and Washington University. Mr Ötleş reported having a patent pending for the University of Michigan for an artificial intelligence–based approach for the dynamic prediction of the injured patient health state. Dr Singh reported receiving grants from Teva Pharmaceuticals and Blue Cross Blue Shield of Michigan. No other disclosures were reported.

    Funding/Support: This study was supported by the University of Michigan Precision Health (Ms Cao), grant KL2TR002346 from the National Institutes of Health National Center for Advancing Translational Sciences (Dr Lyons), the Doris Duke Charitable Foundation Fund to Retain Clinical Scientists (Dr Lyons), and grant T32GM007863 from the National Institutes of Health National Institute of General Medical Sciences (to Mr Ötleş).

    Role of the Funder/Sponsor: The University of Michigan Precision Health, the National Institutes of Health, and the Doris Duke Charitable Foundation had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    References
    1.
    Downing  NL, Rolnick  J, Poole  SF,  et al.  Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation.   BMJ Qual Saf. 2019;28(9):762-768. doi:10.1136/bmjqs-2018-008765 PubMedGoogle ScholarCrossref
    2.
    Wong  A, Otles  E, Donnelly  JP,  et al.  External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients.   JAMA Intern Med. 2021;181(8):1065-1070. doi:10.1001/jamainternmed.2021.2626 PubMedGoogle ScholarCrossref
    3.
    Ginestra  JC, Giannini  HM, Schweickert  WD,  et al.  Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock.   Crit Care Med. 2019;47(11):1477-1484. doi:10.1097/CCM.0000000000003803 PubMedGoogle ScholarCrossref
    4.
    Eilish  MC, Watkins  EA. Repairing innovation: a study of integrating AI in clinical care. Data & Society Research Institute. Published September 2020. Accessed August 5, 2021. https://datasociety.net/wp-content/uploads/2020/09/Repairing-Innovation-DataSociety-20200930-1.pdf
    5.
    Finlayson  SG, Subbaswamy  A, Singh  K,  et al.  The clinician and dataset shift in artificial intelligence.   N Engl J Med. 2021;385(3):283-286. doi:10.1056/NEJMc2104626 PubMedGoogle ScholarCrossref
    6.
    Singh  K, Valley  TS, Tang  S,  et al  Evaluating a widely implemented proprietary deterioration index model among hospitalized patients with COVID-19.   Ann Am Thorac Soc. 2021;18(7):1129-1137. doi:10.1513/AnnalsATS.202006-698OCGoogle Scholar
    ×