[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
Purchase Options:
[Skip to Content Landing]
Figure 1.
Distribution of Patients by SOFA Score, SIRS Criteria, and qSOFA Score on ICU Admission Among Patients With Suspected Infection (N = 184 875)
Distribution of Patients by SOFA Score, SIRS Criteria, and qSOFA Score on ICU Admission Among Patients With Suspected Infection (N = 184 875)

ICU indicates intensive care unit; qSOFA, quick Sequential [Sepsis-related] Organ Function Assessment; SIRS, systemic inflammatory response syndrome; SOFA, Sequential [Sepsis-related] Organ Function Assessment. Number of patients included in the analysis were 183 331 for SOFA, 183 078 for qSOFA, and 182974 for SIRS criteria. Y-axis scale shown in blue indicates the range from 0% to 12%.

Figure 2.
Mortality by SOFA Score, SIRS Criteria, and qSOFA Score on ICU Admission Among Patients With Suspected Infection (N = 184 875)
Mortality by SOFA Score, SIRS Criteria, and qSOFA Score on ICU Admission Among Patients With Suspected Infection (N = 184 875)

ICU indicates intensive care unit; qSOFA, quick Sequential [Sepsis-related] Organ Function Assessment; SIRS, systemic inflammatory response syndrome; SOFA, Sequential [Sepsis-related] Organ Function Assessment. Number of patients included in the analysis were 183 331 for SOFA, 183 078 for qSOFA, and 182 974 for SIRS criteria. Error bars indicate 99% CIs. Y-axis scale shown in blue indicates the range from 0% to 35%.

Figure 3.
Area Under the Receiver Operating Characteristic Curves (AUROCs) for Discriminatory Capacity for In-Hospital Mortality or Composite Outcomes of In-Hospital Mortality or ICU Length of Stay ≥3 Days for SIRS Criteria, qSOFA Score, and SOFA Score (Increase in Score) on ICU Admission
Area Under the Receiver Operating Characteristic Curves (AUROCs) for Discriminatory Capacity for In-Hospital Mortality or Composite Outcomes of In-Hospital Mortality or ICU Length of Stay ≥3 Days for SIRS Criteria, qSOFA Score, and SOFA Score (Increase in Score) on ICU Admission

ICU indicates intensive care unit; qSOFA, quick Sequential [Sepsis-related] Organ Function Assessment; SIRS, systemic inflammatory response syndrome; SOFA, Sequential [Sepsis-related] Organ Function Assessment. Panel A, crude AUROCs: SOFA, 0.753 (99% CI, 0.750-0.757); SIRS, 0.589 (99% CI, 0.585-0.593); qSOFA, 0.607 (99% CI, 0.603-0.611). Panel B, AUROCs: SOFA, 0.815 (99% CI, 0.811-0.818); SIRS, 0.0.755 (0.99% CI, 0.752-0.759); qSOFA, 0.763 (99% CI, 0.758-0.765). Panel C, crude AUROCs: SOFA, 0.736 (99% CI, 0.733-0.739); SIRS, 0.609 (99% CI, 0.606-0.612); qSOFA, 0.606 (99% CI, 0.602-0.609). Panel D, AUROCs: SOFA, 0.761 (99% CI, 0.758-0.764); SIRS, 0.691 (99% CI, 0.688-0.694); qSOFA, 0.69 (99% CI, 0.687-0.693).

aAfter adjustment for baseline risk of death.

bAfter adjustment for baseline risk.

Figure 4.
Odds Ratios for In-Hospital Mortality and for In-Hospital Mortality or ICU Length of Stay ≥3 Days (Log Scale) Comparing Encounters With ≥2 Criteria vs <2 Criterion on qSOFA, SIRS Criteria, and SOFA for Each Decile of Baseline Risk in ICU Patients With Suspected Infection (N = 184 875)
Odds Ratios for In-Hospital Mortality and for In-Hospital Mortality or ICU Length of Stay ≥3 Days (Log Scale) Comparing Encounters With ≥2 Criteria vs <2 Criterion on qSOFA, SIRS Criteria, and SOFA for Each Decile of Baseline Risk in ICU Patients With Suspected Infection (N = 184 875)

ICU indicates intensive care unit; qSOFA, quick Sequential [Sepsis-related] Organ Function Assessment; SIRS, systemic inflammatory response syndrome; SOFA, Sequential [Sepsis-related] Organ Function Assessment. Data referenced against the model of baseline risk of in-hospital mortality (Panel A) and baseline risk of in-hospital mortality or ICU length of stay ≥3 days (Panel B) determined for the cohort, based on variables independent of the scoring systems (data available in eTables 1-2 in the Supplement). Number of patients included in the analysis: SOFA, 183 331); qSOFA, 183 078); SIRS, 182 974). Error bars indicate the 99% CIs. Panel A, Interpretive example: the x-axis divides the cohort into deciles of baseline mortality risk, determined by all available information at the time of ICU admission including factors relating to the ICU (size, type, location, and admission source), admission time (month, day, and hour) and patient (age, sex, comorbidities, pregnancy, diabetes, indigenous status, and treatment limitations). For a middle-aged woman with no comorbidities (decile 5) admitted to the ICU with pneumonia, her chance of dying in the hospital is 4.55 (99% CI, 3.24-6.38) times greater if she has 2 or more SOFA points compared with less than 2 SOFA points. Alternatively, her risk of dying in-hospital increases 2.53 (99% CI, 2.04-3.15) times if she has 2 or more SIRS criteria compared with less than 2 SIRS criteria, and 2.10 (99% CI, 1.87-2.37) times if she has 2 or more qSOFA points compared with less than 2 qSOFA points. Panel B, Interpretive example: the x-axis divides the cohort into deciles of baseline risk of in-hospital mortality or ICU length of stay ≥3 days, determined by all available information at the time of ICU admission including factors relating to the ICU (size, type, location, and admission source), admission time (month, day, and hour) and patient (age, sex, comorbidities, pregnancy, diabetes, indigenous status, and treatment limitations). For a middle-aged woman with no comorbidities (decile 5) admitted to the ICU with pneumonia, her chance of dying in the hospital is 4.53 (99% CI, 3.88-5.28) times greater if she has 2 or more SOFA points compared with less than 2 SOFA points. Alternatively, her risk of dying in-hospital increases 2.51 (99% CI, 2.24-2.83) times if she has 2 or more SIRS criteria compared with less than 2 SIRS criteria, and 1.90 (99% CI, 1.76-2.05) times if she has 2 or more qSOFA points compared with less than 2 qSOFA points.

Table 1.  
Demographic, Physiological, Illness Severity, Diagnostic and Outcome Data Among Critically Ill Patients Admitted With Infection in the ANZICS Adult Patient Database (2000-2015)
Demographic, Physiological, Illness Severity, Diagnostic and Outcome Data Among Critically Ill Patients Admitted With Infection in the ANZICS Adult Patient Database (2000-2015)
Table 2.  
Crude and Adjusted AUROCs for Discrimination Characteristics of SOFA, SIRS Criteria, and qSOFA on ICU Admission Among Patients With Infection in the ANZICS Adult Patient Database Cohort (N = 184 875)
Crude and Adjusted AUROCs for Discrimination Characteristics of SOFA, SIRS Criteria, and qSOFA on ICU Admission Among Patients With Infection in the ANZICS Adult Patient Database Cohort (N = 184 875)
Table 3.  
Crude Mortality Data Per Scoring System Among Patients Admitted to an ICU With Infection in the ANZICS Adult Patient Database Cohort, Reported by Decile of Baseline Risk
Crude Mortality Data Per Scoring System Among Patients Admitted to an ICU With Infection in the ANZICS Adult Patient Database Cohort, Reported by Decile of Baseline Risk
1.
Bone  RC, Balk  RA, Cerra  FB,  et al; The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine.  Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis.  Chest. 1992;101(6):1644-1655.PubMedGoogle ScholarCrossref
2.
Levy  MM, Fink  MP, Marshall  JC,  et al; SCCM/ESICM/ACCP/ATS/SIS.  2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference.  Crit Care Med. 2003;31(4):1250-1256.PubMedGoogle ScholarCrossref
3.
Shankar-Hari  M, Phillips  GS, Levy  ML,  et al; Sepsis Definitions Task Force.  Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).  JAMA. 2016;315(8):775-787.PubMedGoogle ScholarCrossref
4.
Singer  M, Deutschman  CS, Seymour  CW,  et al.  The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).  JAMA. 2016;315(8):801-810.PubMedGoogle ScholarCrossref
5.
Fleischmann  C, Scherag  A, Adhikari  NKJ,  et al; International Forum of Acute Care Trialists.  Assessment of global incidence and mortality of hospital-treated sepsis: current estimates and limitations.  Am J Respir Crit Care Med. 2016;193(3):259-272.PubMedGoogle ScholarCrossref
6.
Iwashyna  TJ, Ely  EW, Smith  DM, Langa  KM.  Long-term cognitive impairment and functional disability among survivors of severe sepsis.  JAMA. 2010;304(16):1787-1794.PubMedGoogle ScholarCrossref
7.
Davis  JS, He  V, Anstey  NM, Condon  JR.  Long term outcomes following hospital admission for sepsis using relative survival analysis: a prospective cohort study of 1,092 patients with 5 year follow up.  PLoS One. 2014;9(12):e112224. PubMedGoogle ScholarCrossref
8.
Duke  GJ, Barker  A, Rasekaba  T, Hutchinson  A, Santamaria  JD.  Development and validation of the critical care outcome prediction equation, version 4.  Crit Care Resusc. 2013;15(3):191-197.PubMedGoogle Scholar
9.
Seymour  CW, Liu  VX, Iwashyna  TJ,  et al.  Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).  JAMA. 2016;315(8):762-774.PubMedGoogle ScholarCrossref
10.
Kaukonen  K-M, Bailey  M, Pilcher  D, Cooper  DJ, Bellomo  R.  Systemic inflammatory response syndrome criteria in defining severe sepsis.  N Engl J Med. 2015;372(17):1629-1638.PubMedGoogle ScholarCrossref
11.
Knaus  WA, Wagner  DP, Draper  EA,  et al.  The APACHE III prognostic system: risk prediction of hospital mortality for critically ill hospitalized adults.  Chest. 1991;100(6):1619-1636.PubMedGoogle ScholarCrossref
12.
Kaukonen  K-M, Bailey  M, Suzuki  S, Pilcher  D, Bellomo  R.  Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012.  JAMA. 2014;311(13):1308-1316.PubMedGoogle ScholarCrossref
13.
Pilcher  D, Paul  E, Bailey  M, Huckson  S.  The Australian and New Zealand Risk of Death (ANZROD) model: getting mortality prediction right for intensive care units.  Crit Care Resusc. 2014;16(1):3-4.PubMedGoogle Scholar
14.
Paul  E, Bailey  M, Kasza  J, Pilcher  D.  The ANZROD model: better benchmarking of ICU outcomes and detection of outliers.  Crit Care Resusc. 2016;18(1):25-36.PubMedGoogle Scholar
15.
Vincent  JL, Moreno  R, Takala  J,  et al.  The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure: on behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine.  Intensive Care Med. 1996;22(7):707-710.PubMedGoogle ScholarCrossref
16.
Rivera-Fernández  R, Nap  R, Vázquez-Mata  G, Reis Miranda  D.  Analysis of physiologic alterations in intensive care unit patients and their relationship with mortality.  J Crit Care. 2007;22(2):120-128.PubMedGoogle ScholarCrossref
17.
Peake  SL, Delaney  A, Bailey  M,  et al; ARISE Investigators; ANZICS Clinical Trials Group.  Goal-directed resuscitation for patients with early septic shock.  N Engl J Med. 2014;371(16):1496-1506.PubMedGoogle ScholarCrossref
18.
Castegren  M, Jonasson  M, Castegren  S, Lipcsey  M, Sjölin  J.  Initial levels of organ failure, microbial findings and mortality in intensive care-treated primary, secondary and tertiary sepsis.  Crit Care Resusc. 2015;17(3):174-181.PubMedGoogle Scholar
19.
Albur  M, Hamilton  F, MacGowan  AP.  Early warning score: a dynamic marker of severity and prognosis in patients with gram-negative bacteraemia and sepsis.  Ann Clin Microbiol Antimicrob. 2016;15(April):23.PubMedGoogle ScholarCrossref
20.
Jung  B, Daurat  A, De Jong  A,  et al.  Rapid response team and hospital mortality in hospitalized patients.  Intensive Care Med. 2016;42(4):494-504.PubMedGoogle ScholarCrossref
21.
Ou  L, Chen  J, Burrell  T,  et al.  Incidence and mortality of postoperative sepsis in New South Wales, Australia, 2002-2009.  Crit Care Resusc. 2016;18(1):9-16.PubMedGoogle Scholar
Original Investigation
Caring for the Critically Ill Patient
January 17, 2017

Prognostic Accuracy of the SOFA Score, SIRS Criteria, and qSOFA Score for In-Hospital Mortality Among Adults With Suspected Infection Admitted to the Intensive Care Unit

Author Affiliations
  • 1Department of Intensive Care and Hyperbaric Medicine, Alfred Hospital, Prahran, Melbourne, Australia
  • 2Discipline of Surgery, School of Medicine, University of Adelaide, Adelaide, Australia
  • 3Australian and New Zealand Intensive Care Research Centre, School of Public Health and Preventive Medicine, Monash University, Alfred Hospital, Prahran, Melbourne, Australia
  • 4Department of Infectious Diseases, Alfred Hospital, Prahran, Melbourne, Australia
  • 5Department of Intensive Care, Royal Melbourne Hospital, Parkville, Melbourne, Australia
  • 6University of Melbourne, Parkville, Melbourne, Australia
  • 7Intensive Care Unit, Austin Hospital, Heidelberg, Melbourne, Australia
  • 8Centre for Outcome and Resource Evaluation, Australian and New Zealand Intensive Care Society, Melbourne, Australia
JAMA. 2017;317(3):290-300. doi:10.1001/jama.2016.20328
Key Points

Question  Does an increase of 2 or more points in Sequential [Sepsis-related] Organ Failure Assessment (SOFA) score have greater prognostic accuracy in patients who are critically ill with suspected infection than 2 or more systemic inflammatory response syndrome (SIRS) criteria or quick SOFA (qSOFA) score points?

Findings  In this retrospective cohort analysis that included 184 875 adults, SOFA (area under the receiver operating characteristic curve [AUROC], 0.753) demonstrated significantly greater discrimination for in-hospital mortality than SIRS criteria (AUROC, 0.589) or qSOFA (AUROC, 0.607).

Meaning  Among patients admitted to the intensive care unit with suspected infection, defining sepsis by an increase in SOFA score provided greater prognostic accuracy for in-hospital mortality than either SIRS criteria or qSOFA.

Abstract

Importance  The Sepsis-3 Criteria emphasized the value of a change of 2 or more points in the Sequential [Sepsis-related] Organ Failure Assessment (SOFA) score, introduced quick SOFA (qSOFA), and removed the systemic inflammatory response syndrome (SIRS) criteria from the sepsis definition.

Objective  Externally validate and assess the discriminatory capacities of an increase in SOFA score by 2 or more points, 2 or more SIRS criteria, or a qSOFA score of 2 or more points for outcomes among patients who are critically ill with suspected infection.

Design, Setting, and Participants  Retrospective cohort analysis of 184 875 patients with an infection-related primary admission diagnosis in 182 Australian and New Zealand intensive care units (ICUs) from 2000 through 2015.

Exposures  SOFA, qSOFA, and SIRS criteria applied to data collected within 24 hours of ICU admission.

Main Outcomes and Measures  The primary outcome was in-hospital mortality. In-hospital mortality or ICU length of stay (LOS) of 3 days or more was a composite secondary outcome. Discrimination was assessed using the area under the receiver operating characteristic curve (AUROC). Adjusted analyses were performed using a model of baseline risk determined using variables independent of the scoring systems.

Results  Among 184 875 patients (mean age, 62.9 years [SD, 17.4]; women, 82 540 [44.6%]; most common diagnosis bacterial pneumonia, 32 634 [17.7%]), a total of 34 578 patients (18.7%) died in the hospital, and 102 976 patients (55.7%) died or experienced an ICU LOS of 3 days or more. SOFA score increased by 2 or more points in 90.1%; 86.7% manifested 2 or more SIRS criteria, and 54.4% had a qSOFA score of 2 or more points. SOFA demonstrated significantly greater discrimination for in-hospital mortality (crude AUROC, 0.753 [99% CI, 0.750-0.757]) than SIRS criteria (crude AUROC, 0.589 [99% CI, 0.585-0.593]) or qSOFA (crude AUROC, 0.607 [99% CI, 0.603-0.611]). Incremental improvements were 0.164 (99% CI, 0.159-0.169) for SOFA vs SIRS criteria and 0.146 (99% CI, 0.142-0.151) for SOFA vs qSOFA (P <.001). SOFA (AUROC, 0.736 [99% CI, 0.733-0.739]) outperformed the other scores for the secondary end point (SIRS criteria: AUROC, 0.609 [99% CI, 0.606-0.612]; qSOFA: AUROC, 0.606 [99% CI, 0.602-0.609]). Incremental improvements were 0.127 (99% CI, 0.123-0.131) for SOFA vs SIRS criteria and 0.131 (99% CI, 0.127-0.134) for SOFA vs qSOFA (P <.001). Findings were consistent for both outcomes in multiple sensitivity analyses.

Conclusions and Relevance  Among adults with suspected infection admitted to an ICU, an increase in SOFA score of 2 or more had greater prognostic accuracy for in-hospital mortality than SIRS criteria or the qSOFA score. These findings suggest that SIRS criteria and qSOFA may have limited utility for predicting mortality in an ICU setting.

Introduction

Sepsis remains difficult to define14 but represents a significant burden of disease. A recent meta-analysis estimated its annual global incidence at 31.5 million cases, with 19.4 million cases of severe sepsis, resulting in 5.3 million deaths.5 It has been recognized that survival following sepsis is associated with long-term physical, cognitive, and psychosocial morbidity,6 and an increased mortality rate up to 2 years after an event.7

Accurate diagnostic criteria and consensus definitions have an important role in adult intensive care medicine, providing tools for research, benchmarking, performance monitoring, and accreditation.8 Seymour and colleagues9 published data concerning the validity of a 2 or more–point change in the Sequential [Sepsis-related] Organ Failure Assessment (SOFA) score as a means of identifying sepsis among patients who are critically ill with suspected infection, assuming a SOFA of 0 for patients not known to have preexisting organ dysfunction. In addition, the concept of the quick SOFA (qSOFA) score was introduced as a possible predictive tool among encounters with suspected infection outside the intensive care unit (ICU). These data were drawn from North American cohorts and a single German cohort9 and have not been validated externally.

The primary aims of this study were to (1) assess the effect of an increase in SOFA score of 2 or more points, 2 or more systemic inflammatory response syndrome (SIRS) criteria, and a qSOFA score of 2 or more points measured within the first 24 hours of admission in discriminating in-hospital mortality or prolonged length of stay among patients with suspected infection admitted to ICUs throughout Australia and New Zealand and (2) to validate the use of qSOFA in this setting.1,2,10

Methods

The Alfred Hospital Human Research Ethics Committee in Melbourne, Australia, approved this study with a waiver of informed consent.

Study Design and Population

A retrospective cohort study was performed using young and older adult (aged ≥17 years) admissions with suspected infection in the Australian and New Zealand Intensive Care Society (ANZICS) Adult Patient Database. All patient records between 2000 and 2015 were screened for study inclusion. Repeat ICU admissions from the same hospital episode and patients transferred to another ICU facility whose eventual outcome was unknown were excluded. Infection or suspected infection was then inferred from infection-related ICU admission diagnoses according to the ANZICS modification of the Acute Physiology and Chronic Health Evaluation (APACHE) III scoring system,11 as used in previous work.10,12

Data Extraction

The following data were extracted: demographic information, indigenous status, ICU and hospital length of stay, vital status at ICU and hospital discharge, physiological and laboratory variables collected for calculation of APACHE II and III scores, and the Australian and New Zealand Risk of Death (ANZROD) prediction model.13,14 Serum lactate levels were not available. SOFA scores (range, 0 [best] to 24 [worst] points),4,15 SIRS status (range, 0 [best] to 4 [worst] criteria),1 and qSOFA scores (range, 0 [best] to 3 [worst] points)4,9 were calculated using physiological and laboratory parameters recorded from within the first 24 hours of ICU admission. Standard criteria were applied, with a threshold of 2 or more points being used with each scoring system. In keeping with the Sepsis-3 consensus statement,4 a Glasgow Coma Scale Score (range, 3 [worst] to 15 [best] points) of less than 15 was used for estimation of the qSOFA score. In the primary analysis, a baseline SOFA score of 0 was assumed for all patients.4 Where individual components of SIRS criteria, SOFA, and qSOFA were missing, no contribution was made to the total score (eg, equivalent to assigning 0 or imputing a normal value). Where all components were unknown, the patient was assigned a missing score, and excluded from the primary analysis. In subsequent sensitivity analyses, 2 additional approaches were employed: first, patients with known chronic renal, liver, or respiratory dysfunction (defined per APACHE II and III comorbidity codes13) were excluded, and second, a baseline SOFA score of 4 was assigned for chronic renal or hepatic dysfunction and a baseline score of 2 for chronic respiratory impairment.

Outcomes

In accordance with Seymour et al,9 the primary study outcome was in-hospital mortality with a composite secondary outcome of in-hospital mortality or an ICU length of stay of 3 days or longer.

Statistical Analysis

All analyses were performed using SAS (SAS Institute), version 9.4. Group comparisons were conducted using χ2 tests for equal proportions, t tests for normally distributed data, and Wilcoxon rank sum tests otherwise. Discriminatory power was determined by comparing the area under the receiver operating characteristic curve (AUROC) for each score individually (unadjusted analysis) and in conjunction with a baseline risk model (adjusted analysis). Specific AUROC (99% CI) values were generated, along with incremental improvement between the scores (99% CI).

Baseline risk models for in-hospital mortality and in-hospital mortality or ICU stay of 3 or more days were created using hierarchical logistic regression with patients nested within sites and sites treated as a random variable. Both models were constructed using all available information at the time of ICU admission including factors relating to the ICU (size, type, location, and admission source), admission time (month, day, and hour) and patient (age, sex, comorbidities, pregnancy, diabetes, indigenous status, and treatment limitations) (eTables 1-2 in the Supplement). Predictive validity was determined by dividing baseline risk into deciles and comparing outcomes among patients with an increase of 2 or more points (SOFA) or qSOFA and SIRS criteria of 2 or more against patients with an increase of less than 2 for SOFA score or qSOFA and SIRS criteria of less than 2 at each decile of risk.

Sensitivity analyses were performed for ventilation status, multiple imputation to account for missing data, qSOFA calculated using Glasgow Coma Scale score less than 14 for altered mentation, the presence of chronic comorbidities, admission source, and improved baseline models that sequentially include diagnosis and physiological variables (separate to those used to calculate each of the candidate scores) collected over the first 24 hours in the ICU. Agreement between the SOFA score, SIRS criteria, and qSOFA score was determined using Cronbach α with 99% CIs generated with bootstrapping (100 samples). To increase the robustness of the analysis, a 2-sided P value of .01 was used to indicate statistical significance. Further details of the statistical analysis can be found in the eAppendix of the Supplement.

Results
Study Population

Data pertaining to 1 499 753 adult admissions were recorded in the ANZICS Adult Patient Database for the period of 2000-2015, drawn from 182 ICUs across Australia and New Zealand. Following restriction to completed index ICU admissions and selection of infection-related diagnoses,10,12 a final cohort of 184 875 cases were identified (eFigure 1 and eTable 3 in the Supplement). As shown in Table 1 (also in eTable 3 of the Supplement), mean age was 62.9 years (SD, 17.4), 55.4% (n = 102 335) were male, and 44.6% (n = 82 540) were female. Bacterial pneumonia was listed as the most common diagnosis among 32 634 patients (17.7%). There were 34 578 patients (18.7%) who died in the hospital. The secondary outcome of in-hospital mortality or length of stay in the ICU of 3 days or more occurred in 102 976 patients (55.7%).

SOFA, SIRS, qSOFA, and Study Outcomes

Of the study cohort, 165 103 patients (90.1%) had an increase of SOFA score from baseline of 2 or more; 158 710 patients (86.7%) manifested 2 or more SIRS criteria, and 99 611 patients (54.4%) had a qSOFA score of 2 or more. The distributions of each score and their relationship with in-hospital mortality are presented in Figure 1 and Figure 2.

The SOFA score could not be calculated in 1544 patients (0.8%); SIRS status, 1901 patients (1.0%); qSOFA score, 1797 patients (1.0%). In-hospital mortality was 28.6%, 27.2% and 27.2% for each of these groups, respectively, for which complete data were missing (eFigure 2 in the Supplement).

Score Discrimination

Discrimination of in-hospital mortality (Table 2) was significantly higher using SOFA (AUROC, 0.753 [99% CI, 0.750-0.757]) than either SIRS criteria (AUROC, 0.589 [99% CI, 0.585-0.593]) or qSOFA (AUROC, 0.607 [99% CI, 0.603-0.611]) with all incremental differences being statistically significant (between-group difference: SOFA vs qSOFA, 0.146 [99% CI, 0.142-0.151]; SOFA vs SIRS criteria, 0.164 [99% CI, 0.159-0.169]; qSOFA vs SIRS criteria, 0.018 [99% CI, 0.013-0.023]; all P < .001) (Figure 3A). Similarly, when considered in conjunction with baseline prediction of mortality, SOFA (AUROC, 0.815 [99% CI, 0.811-0.818]) outperformed both SIRS criteria (AUROC, 0.755 [99% CI, 0.752-0.759]) and qSOFA (AUROC, 0.762 [99% CI, 0.758-0.765]) for discrimination of hospital mortality (Figure 3B), with all incremental differences being statistically significant (between-group difference: SOFA vs SIRS criteria, 0.047 [99% CI, 0.044-0.049]; SOFA vs qSOFA, 0.042 [99% CI, 0.040-0.044]; qSOFA vs SIRS criteria, 0.005 [99% CI, 0.003-0.006]); all P < .001).

The superior discriminatory performance of SOFA over both SIRS criteria and qSOFA was further maintained when considering the secondary outcome of hospital mortality or ICU length of stay of 3 or more days when considered in isolation or in conjunction with the baseline prediction (Table 2, Figure 3C, and Figure 3D).

Additional data showing calibration plots for observed and expected outcomes are demonstrated in eFigure 3 and 4 of the Supplement. Full baseline risk models for in-hospital mortality and for the secondary outcome of hospital mortality or ICU length of stay of 3 or more days are also provided in eTables 1 and 2 of the Supplement.

Incremental Increase in Mortality Associated With 2 or More Points or Criteria for Each Measurement

The in-hospital mortality of patients who had an increase from baseline in SOFA score of 2 or more was 20.2% (33 365 of 165 103 patients) vs 4.4% (793 of 18 228 patients), P < .001 (between-group difference, 15.9% [99% CI, 15.4%-16.3%]); for those who manifested 2 or more SIRS criteria, mortality was 19.9% (31 648 of 158 710 patients) vs 9.8% (2387 of 24 264 patients), P < .001 (between-group difference, 10.1% [99% CI, 9.5%-10.7%]); and for those who had a qSOFA score of 2 or more, mortality was 22.8% (22 758 of 99 611 patients) vs 13.6% (11 332 of 83 457 patients), P < .001 (between-group difference, 9.3% [99% CI, 8.8%-9.7%]); when compared with those with less than 2 points or criteria on each respective score.

Patients with an increase from baseline in SOFA score of 2 or more had a greater incremental increase in mortality across all deciles of baseline risk, than patients with 2 or more SIRS criteria, or those with a qSOFA score of 2 or more (Table 3 and Figure 4A). The incremental increase in mortality when a patient had an increase in SOFA score of 2 or more ranged from an odds ratio (OR) of 7.46 (99% CI, 4.01-13.89) among those in the lowest baseline risk decile to an OR of 2.70 (99% CI, 2.14-3.42) among those in the highest decile.

This analysis was repeated with the composite secondary outcome (Table 3 and Figure 4B). Those with an increase of SOFA score of 2 or more had a significantly greater incremental increase in risk of the composite outcome of in-hospital mortality or length of stay in the ICU of 3 or more days, at all deciles of baseline risk, than patients who manifested 2 or more SIRS criteria or those with a qSOFA score of 2 or more points. The incremental increase in risk of the composite secondary outcome when a patient had an increase in SOFA score of 2 or more varied from an OR of 3.59 (99% CI, 3.15-4.10) among those in the lowest baseline risk decile to an OR of 4.13 (99% CI, 3.38-5.03) among those in the highest decile.

Sensitivity Analyses

A number of sensitivity analyses were performed, wherein the AUROC over and above the baseline level for the primary outcome (using each scoring system) was calculated in a range of scenarios. These included (1) mechanically ventilated vs nonventilated patients, (2) using multiple imputation to account for missing data, (3) using a Glasgow Coma Scale score less than 14 vs less than 15 for calculation of qSOFA, (4) having patients with chronic organ dysfunction either excluded or assigned a given baseline SOFA score, (5) differing the ICU admission source, and (6) using improved baseline models. In each case, SOFA demonstrated superior discrimination compared with SIRS criteria or qSOFA (eTable 4 in the Supplement).

Discussion
Key Findings

In comparison with SIRS criteria or a qSOFA score of 2 or more, an increase in SOFA score of 2 or more within 24 hours of ICU admission demonstrated superior prognostic accuracy for in-hospital mortality and the composite outcome of in-hospital mortality or ICU length of stay of 3 days or more among patients with suspected infection admitted to Australian and New Zealand ICUs.

Relationship With Previous Studies

This study examined and compared the discriminatory capacity of SOFA, SIRS criteria, and qSOFA in a large ICU population of patients outside of the cohort used to develop the new definition of sepsis. Kaukonen and colleagues10 have previously demonstrated that SIRS criteria are an imperfect predictor of ICU mortality, inappropriately excluding otherwise similar patients with infection, organ failure, and increased mortality. Furthermore, use of SIRS criteria failed to define a transition point in risk of death, despite adjustment for baseline characteristics.10

Conversely, SOFA scores have been demonstrated to be a useful predictor of ICU mortality.15 Rivera-Fernández and colleagues16 demonstrated that 28-day mortality was related to mean and maximum daily SOFA scores in a cohort of patients who were critically ill with an AUROC of 0.95.16 The relationship between SOFA scores and risk of death has been confirmed in a variety of subgroups14,1618 including sepsis.9,15 In the sepsis consensus definition, the ability of SOFA, SIRS criteria, and qSOFA scores to discriminate in-hospital mortality or prolonged ICU length of stay was determined in a cohort of both ICU and non-ICU encounters with suspected infection.9 SOFA demonstrated significantly greater capacity compared with qSOFA and SIRS criteria in the ICU cohort.9 This study demonstrated findings consistent with these reports.

The mean age (63 years) in this study was similar to that reported in the ICU validation cohort for the definition of sepsis by Seymour et al,9 as was the proportion of males (approximately 55%). In-hospital mortality in this study was marginally higher (19% in this study vs 16% in Seymour et al), although roughly equal fractions manifested an increase in SOFA score of 2 or more (90% in this study vs 91% in Seymour et al). Importantly, this cohort was more than 11 times larger than the total number of ICU encounters used by Seymour et al,9 and was derived from multiple ICUs dispersed among 2 countries. The adjusted AUROCs for in-hospital mortality for SOFA score, SIRS criteria, and qSOFA score in this ICU cohort were substantially higher than previously reported,9 which was, in part, likely a reflection of the strength of the baseline model used in analysis.

Study Implications

Using a large binational database independent of the sepsis consensus development cohorts, this study confirmed that an increase in SOFA score of 2 or more points within the first 24 hours of ICU admission had superior prognostic accuracy for mortality or ICU length of stay of 3 days or more compared with SIRS criteria or qSOFA. It established that the use of an increase in SOFA score of 2 or more points to define sepsis was an appropriate data-based starting point, and would likely have broad external validity in ICU patients from developed countries. This study’s findings confirmed that the SIRS criteria provided no additional predictive use for mortality or prolonged ICU length of stay beyond that achieved with SOFA. Finally, this study confirmed that, as already suggested in the consensus statement, the qSOFA score had little additional predictive value over the SIRS criteria among patients admitted to the ICU with suspected infection, and that among these patients, the use of qSOFA may be low.

Strengths

This study had a number of strengths. It was based on one of the largest complete ICU data sets, with a cohort of 184 875 infection-related admissions over a 16-year period, encompassing a wide geographical area. Moreover, due to the use of a prospective, quality-surveillance data collection process, these results were unlikely to be biased. This study used the primary diagnosis leading to ICU admission for patient selection recorded in the ANZICS Adult Patient Database. This use of a clinical diagnosis, rather than administrative coding data increased the generalizability and applicability of the study’s findings. The reported patients had similar characteristics, sex, age, and outcomes to those used by Seymour et al,9 for which they provide an important external validation. This study was further strengthened by the use of a reproducible identification process for infection,10,12 and the use of the same methodology as in the consensus paper.9 This study considered in-hospital mortality as a primary end point, but also measured a composite of in-hospital mortality or ICU length of stay of 3 days or more (as applied to the consensus definition), thus assessing the discrimination for both of these outcomes for the 3 measures. Finally, these data had excellent internal validity, as evidenced by an in-hospital mortality rate (18.7%) that is highly consistent with a recent multicenter study exploring early goal-directed therapy for early septic shock conducted in Australia and New Zealand.17

Limitations

The data used for this analysis were not primarily collected for such study purposes. SOFA, SIRS criteria, and qSOFA could only be studied for the first 24 hours in ICU. Biochemical and physiological values could have come from any time within the first 24 hours of ICU admission and, as a result, could not be more accurately linked to the timing of the diagnosis of infection. The SOFA score used should be considered a modification of the original because the cardiovascular component was estimated without knowledge of inotrope or vasopressor dose. The incidence of nosocomial infections and of infections in patients admitted with another diagnosis were unknown.18 However, although this study could not comment specifically on the applicability of these scores to patients who develop infections later during their ICU stay, the consistency of findings across multiple sensitivity analyses suggests that these scores might have similar performance in all ICU patients. A population at risk for sepsis could not be studied to assess the diagnostic value of all 3 measures. Finally, this study did not examine the non-ICU population. As a result, although the advantage of the qSOFA score is its brevity and use of clinical variables only, its applicability to patients with suspected infection in the rest of the hospital (on the wards or in the Emergency Department) could not be determined by this study.9,1921

Conclusions

Among adults with suspected infection admitted to an ICU, an increase in SOFA score of 2 or more points had greater prognostic accuracy for in-hospital mortality than SIRS criteria or qSOFA. These findings suggest that SIRS and qSOFA may have limited use for predicting mortality in an ICU setting.

Section Editor: Derek C. Angus, MD, MPH, Associate Editor, JAMA (angusdc@upmc.edu).
Back to top
Article Information

Corresponding Author: David V. Pilcher, MBBS, FRACP, FCICM, Department of Intensive Care and Hyperbaric Medicine, Alfred Hospital, Commercial Road, Prahran, Melbourne VIC 3181, Australia (d.pilcher@alfred.org.au).

Author Contributions: Drs Pilcher and Bailey had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Raith, McGloughlin, MacIsaac, Bellomo, Pilcher.
Acquisition, analysis, or interpretation of data: Raith, Udy, Bailey, Bellomo, Pilcher.
Drafting of the manuscript: Raith, Udy, McGloughlin, Bellomo, Pilcher.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Bailey, Pilcher.
Administrative, technical, or material support: Raith, Udy, Bellomo, Pilcher.
Supervision: Udy, McGloughlin, Bellomo, Pilcher.
No additional contributions: MacIsaac.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Udy receives salary support from Monash University as an Alfred ICU Monash University practitioner fellow and receives support from the National Health and Medical Research Council of Australia (Early Career Fellowship). Dr Bailey receives salary support from the School of Public Health, Monash University. Dr Pilcher receives salary support from Monash University as an Alfred ICU Monash University practitioner fellow.

Additional Information: This work was performed at the Centre for Outcome and Resource Evaluation of the Australian and New Zealand Intensive Care Society (ANZICS), Melbourne, Australia, and at the Department of Intensive Care and Hyperbaric Medicine of Alfred Hospital in Prahran, Melbourne, Australia. We thank ANZICS Centre for Outcomes and Resource Evaluation (CORE) in completing this research and the work of data collectors and clinicians who contributed to the CORE registries throughout Australia and New Zealand. The ANZICS CORE registry database is funded by local jurisdictions. No one received compensation for their contribution.

References
1.
Bone  RC, Balk  RA, Cerra  FB,  et al; The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine.  Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis.  Chest. 1992;101(6):1644-1655.PubMedGoogle ScholarCrossref
2.
Levy  MM, Fink  MP, Marshall  JC,  et al; SCCM/ESICM/ACCP/ATS/SIS.  2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference.  Crit Care Med. 2003;31(4):1250-1256.PubMedGoogle ScholarCrossref
3.
Shankar-Hari  M, Phillips  GS, Levy  ML,  et al; Sepsis Definitions Task Force.  Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).  JAMA. 2016;315(8):775-787.PubMedGoogle ScholarCrossref
4.
Singer  M, Deutschman  CS, Seymour  CW,  et al.  The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).  JAMA. 2016;315(8):801-810.PubMedGoogle ScholarCrossref
5.
Fleischmann  C, Scherag  A, Adhikari  NKJ,  et al; International Forum of Acute Care Trialists.  Assessment of global incidence and mortality of hospital-treated sepsis: current estimates and limitations.  Am J Respir Crit Care Med. 2016;193(3):259-272.PubMedGoogle ScholarCrossref
6.
Iwashyna  TJ, Ely  EW, Smith  DM, Langa  KM.  Long-term cognitive impairment and functional disability among survivors of severe sepsis.  JAMA. 2010;304(16):1787-1794.PubMedGoogle ScholarCrossref
7.
Davis  JS, He  V, Anstey  NM, Condon  JR.  Long term outcomes following hospital admission for sepsis using relative survival analysis: a prospective cohort study of 1,092 patients with 5 year follow up.  PLoS One. 2014;9(12):e112224. PubMedGoogle ScholarCrossref
8.
Duke  GJ, Barker  A, Rasekaba  T, Hutchinson  A, Santamaria  JD.  Development and validation of the critical care outcome prediction equation, version 4.  Crit Care Resusc. 2013;15(3):191-197.PubMedGoogle Scholar
9.
Seymour  CW, Liu  VX, Iwashyna  TJ,  et al.  Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).  JAMA. 2016;315(8):762-774.PubMedGoogle ScholarCrossref
10.
Kaukonen  K-M, Bailey  M, Pilcher  D, Cooper  DJ, Bellomo  R.  Systemic inflammatory response syndrome criteria in defining severe sepsis.  N Engl J Med. 2015;372(17):1629-1638.PubMedGoogle ScholarCrossref
11.
Knaus  WA, Wagner  DP, Draper  EA,  et al.  The APACHE III prognostic system: risk prediction of hospital mortality for critically ill hospitalized adults.  Chest. 1991;100(6):1619-1636.PubMedGoogle ScholarCrossref
12.
Kaukonen  K-M, Bailey  M, Suzuki  S, Pilcher  D, Bellomo  R.  Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012.  JAMA. 2014;311(13):1308-1316.PubMedGoogle ScholarCrossref
13.
Pilcher  D, Paul  E, Bailey  M, Huckson  S.  The Australian and New Zealand Risk of Death (ANZROD) model: getting mortality prediction right for intensive care units.  Crit Care Resusc. 2014;16(1):3-4.PubMedGoogle Scholar
14.
Paul  E, Bailey  M, Kasza  J, Pilcher  D.  The ANZROD model: better benchmarking of ICU outcomes and detection of outliers.  Crit Care Resusc. 2016;18(1):25-36.PubMedGoogle Scholar
15.
Vincent  JL, Moreno  R, Takala  J,  et al.  The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure: on behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine.  Intensive Care Med. 1996;22(7):707-710.PubMedGoogle ScholarCrossref
16.
Rivera-Fernández  R, Nap  R, Vázquez-Mata  G, Reis Miranda  D.  Analysis of physiologic alterations in intensive care unit patients and their relationship with mortality.  J Crit Care. 2007;22(2):120-128.PubMedGoogle ScholarCrossref
17.
Peake  SL, Delaney  A, Bailey  M,  et al; ARISE Investigators; ANZICS Clinical Trials Group.  Goal-directed resuscitation for patients with early septic shock.  N Engl J Med. 2014;371(16):1496-1506.PubMedGoogle ScholarCrossref
18.
Castegren  M, Jonasson  M, Castegren  S, Lipcsey  M, Sjölin  J.  Initial levels of organ failure, microbial findings and mortality in intensive care-treated primary, secondary and tertiary sepsis.  Crit Care Resusc. 2015;17(3):174-181.PubMedGoogle Scholar
19.
Albur  M, Hamilton  F, MacGowan  AP.  Early warning score: a dynamic marker of severity and prognosis in patients with gram-negative bacteraemia and sepsis.  Ann Clin Microbiol Antimicrob. 2016;15(April):23.PubMedGoogle ScholarCrossref
20.
Jung  B, Daurat  A, De Jong  A,  et al.  Rapid response team and hospital mortality in hospitalized patients.  Intensive Care Med. 2016;42(4):494-504.PubMedGoogle ScholarCrossref
21.
Ou  L, Chen  J, Burrell  T,  et al.  Incidence and mortality of postoperative sepsis in New South Wales, Australia, 2002-2009.  Crit Care Resusc. 2016;18(1):9-16.PubMedGoogle Scholar
×