[Skip to Navigation]
Original Investigation
June 21, 2021

External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients

Author Affiliations
  • 1Department of Internal Medicine, University of Michigan Medical School, Ann Arbor
  • 2Medical Scientist Training Program, University of Michigan Medical School, Ann Arbor
  • 3Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor
  • 4Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor
  • 5School of Public Health, University of Michigan, Ann Arbor
  • 6Department of Quality, Michigan Medicine, Ann Arbor
  • 7Health Information Technology and Services, Michigan Medicine, Ann Arbor
  • 8Nursing Informatics, Michigan Medicine, Ann Arbor
JAMA Intern Med. 2021;181(8):1065-1070. doi:10.1001/jamainternmed.2021.2626
Key Points

Question  How accurately does the Epic Sepsis Model, a proprietary sepsis prediction model implemented at hundreds of US hospitals, predict the onset of sepsis?

Findings  In this cohort study of 27 697 patients undergoing 38 455 hospitalizations, sepsis occurred in 7% of the hosptalizations. The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer.

Meaning  This study suggests that the Epic Sepsis Model poorly predicts sepsis; its widespread adoption despite poor performance raises fundamental concerns about sepsis management on a national level.

Abstract

Importance  The Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM’s ability to identify patients with sepsis has not been adequately evaluated despite widespread use.

Objective  To externally validate the ESM in the prediction of sepsis and evaluate its potential clinical value compared with usual care.

Design, Setting, and Participants  This retrospective cohort study was conducted among 27 697 patients aged 18 years or older admitted to Michigan Medicine, the academic health system of the University of Michigan, Ann Arbor, with 38 455 hospitalizations between December 6, 2018, and October 20, 2019.

Exposure  The ESM score, calculated every 15 minutes.

Main Outcomes and Measures  Sepsis, as defined by a composite of (1) the Centers for Disease Control and Prevention surveillance criteria and (2) International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another. Model discrimination was assessed using the area under the receiver operating characteristic curve at the hospitalization level and with prediction horizons of 4, 8, 12, and 24 hours. Model calibration was evaluated with calibration plots. The potential clinical benefit associated with the ESM was assessed by evaluating the added benefit of the ESM score compared with contemporary clinical practice (based on timely administration of antibiotics). Alert fatigue was evaluated by comparing the clinical value of different alerting strategies.

Results  We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.

Conclusions and Relevance  This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.

Add or change institution
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    1 Comment for this article
    EXPAND ALL
    The Epic Sepsis Model: More Evaluation Needed
    Prem Thomas, MD | Yale New Haven Health
    The external validation of the proprietary Epic Sepsis Model undertaken by Wong et al[1] is commendable for its systematic approach to model performance and benefit. Further articles and vigorous discussion on the benefits of this and other models is needed. The article reports the model's diagnostic performance and measures of potential benefit and burden. Let us consider each category.

    In evaluating diagnostic models, establishing a rigorous gold standard defining the disease cohort lays the cornerstone for inference. Rhee et al[2,3] wrestled with the problem of accurate measurement of sepsis incidence and prevalence in hospitalizations. They provided the most exhaustive
    evaluation to date, assessing strategies based on International Classification of Diseases codes (explicit and explicit/implicit methods) versus a clinical surveillance definition using electronic health record data. Wong uses the Rhee clinical surveillance criteria but then introduces a second unevaluated criterion, similar to the explicit/implicit code strategy. The clinical surveillance definition had a 70% sensitivity, 98% specificity, and a 70% positive predictive value. The code strategy had lower specificity and a positive predictive value of just 31%. This second criterion weakens Wong's assessment, because it introduces the possibility of a significant number of false positives in the gold standard sepsis cohort. This may skew the report of model performance in unpredictable ways.

    Potential benefit was reported as the percent of patients who received timely antibiotics. Of the 2552 hospitalizations with sepsis, a full 34% failed timely antibiotics under usual care. The authors did not construct an actual alert but measured a theoretical benefit based on patients identified by a score ≥ 6, arrived at by committee. Thresholds involve a tug-of-war between sensitivity and specificity. According to the article's plots, a threshold of 3 would dramatically increase the number of patients identified which would increase the number who would have benefitted because they would otherwise fail antibiotics. An analysis of antibiotic benefit at a few thresholds would have been helpful.

    Theoretical alert burden was measured based only on a score trigger. Yet the implementation of actual alerts involves additional trigger criteria, such as user roles, chart actions, and time lockouts based on response to alerts. For example, an appropriate provider response such as "patient already under treatment" could lock further firing for a few days. While the hospitalization level burden provides a good baseline, the other horizons may not reflect reality.

    May the discussion continue!


    References

    1. Wong A, Otles E, Donnelly JP, et al. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Intern Med. Published online June 21, 2021. doi:10.1001/jamainternmed.2021.2626

    2. Rhee C, Dantes R, Epstein L, et al. Incidence and Trends of Sepsis in US Hospitals Using Clinical vs Claims Data, 2009-2014. JAMA. 2017;318(13):1241-1249. doi:10.1001/jama.2017.13836

    3. Rudd KE, Delaney A, Finfer S. Counting Sepsis, an Imprecise but Improving Science. JAMA. 2017;318(13):1228. doi:10.1001/jama.2017.13697
    CONFLICT OF INTEREST: None Reported
    READ MORE
    ×