Development and Validation of a Machine Learning Model to Estimate Bacterial Sepsis Among Immunocompromised Recipients of Stem Cell Transplant | Stem Cell Transplantation | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Receiver Operating Characteristic Curves, Precision Recall Curve, and Calibration Plots for High–Sepsis Risk Bacteremia for All Potential Bloodstream Infections Among Recipients of Allogeneic Hematopoietic Cell Transplant
Receiver Operating Characteristic Curves, Precision Recall Curve, and Calibration Plots for High–Sepsis Risk Bacteremia for All Potential Bloodstream Infections Among Recipients of Allogeneic Hematopoietic Cell Transplant

C- indicates clinical factor–specific; NEWS, National Early Warning Score; PPV, positive predictive value; qSOFA, quick Sequential Organ Failure Assessment; SHBSL, super hematopoietic cell transplant bacterial sepsis learner; and SIRS, Systemic Inflammatory Response Syndrome. High–sepsis risk bacteremia was defined as gram-negative, Staphylococcus aureus, or Streptococcus bacteremia.

Figure 2.  Receiver Operating Characteristic Curves for High-Risk Bacteremia Among Inpatient and Outpatient Identified Potential Bloodstream Infections and for 10-Day and 28-Day Mortality Among All Potential Bloodstream Infections
Receiver Operating Characteristic Curves for High-Risk Bacteremia Among Inpatient and Outpatient Identified Potential Bloodstream Infections and for 10-Day and 28-Day Mortality Among All Potential Bloodstream Infections

C- indicates clinical factor–specific; NEWS, National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment; SHBSL, super hematopoietic cell transplant bacterial sepsis learner; and SIRs, Systemic Inflammatory Response Syndrome. High–sepsis risk bacteremia was defined as gram-negative, Staphylococcus aureus, or Streptococcus bacteremia; 10-day and 28-day mortality, death within 10 or 28 days of culture collection.

Table 1.  Baseline Characteristics of Participants Stratified by Occurrence of High-Risk Bacteremia
Baseline Characteristics of Participants Stratified by Occurrence of High-Risk Bacteremia
Table 2.  Discrimination and Predictive Accuracy of Sepsis Tools for High–Sepsis Risk Bacteremia Among Recipients of Allogeneic Hematopoietic Cell Transplant With Potential Bloodstream Infections
Discrimination and Predictive Accuracy of Sepsis Tools for High–Sepsis Risk Bacteremia Among Recipients of Allogeneic Hematopoietic Cell Transplant With Potential Bloodstream Infections
Table 3.  Prognostic Value of Sepsis Prognosis Tools for High–Sepsis Risk Bacteremia Stratified by Patient Location at Time of Culture Collection and for Short-term Mortality Among Recipients of Allogeneic Hematopoietic Cell Transplant With Potential Bloodstream Infections
Prognostic Value of Sepsis Prognosis Tools for High–Sepsis Risk Bacteremia Stratified by Patient Location at Time of Culture Collection and for Short-term Mortality Among Recipients of Allogeneic Hematopoietic Cell Transplant With Potential Bloodstream Infections
Supplement.

eFigure 1. Bacterial Sepsis Prognosis Tool Schematic

eAppendix 1. Outcome Justification

eTable 1. Frequency of Potential Factors Screened for Model Inclusion

eFigure 2. Missingness Frequency and Pattern for Potential Model Factors

eAppendix 2. Imputation Approach

eTable 2. Reference Value Ranges for Single Reference Value Imputation

eAppendix 3. Detailed Description of Super Learner Library

eAppendix 4. Net Reclassification

eFigure 3. Flowchart of Hematopoietic Cell Transplant Recipient (HCT) Study Population and Potential Bloodstream Infection Cohort

eFigure 4. Histograms of Location of Patient and Collected Cultures by Day Since Transplant

eTable 3. Population Demographic and Transplant Factors by Randomly Assigned Modeling and Validation Dataset

eTable 4. Discrimination and Predictive Accuracy of Sepsis Prognosis Tools for High–Sepsis Risk Bacteremia

eTable 5. Categorical Reclassification Index Comparing Risk Classification of Full Predictor Tool (SHBSL) With Other Examined Tools

eFigure 5. Summary of Prediction Scores by Patient Location

eAppendix 5. Ensemble Selection Rational

eFigure 6. Cross-Validated Area Under the Curve Estimates of Super Learner and User-Supplied Algorithms

eTable 6. Cross-Validated and Bootstrapped Area Under the Curve Estimates for High–Sepsis Risk Bacteremia

eTable 7. Calibration and Observed vs Estimated High–Sepsis Risk Bacteremia Probabilities

eAppendix 6. Sensitivity Analyses

eTable 8. Predictive Ability of Examined Prognostic Tools for High–Sepsis Risk Bacteremia Among Allogeneic HCT Recipients with PBIs Under Varying Missing Data Assumptions

eTable 9. Predicative Ability of Sepsis Prognosis Tools Among Varying Racial/Ethnic Groups

eTable 10. Predicative Ability of Sepsis Prognosis Tools Under Varying Factor Measurement Collection Time Windows

eTable 11. Predictive Ability of Sepsis Prognosis Tools Under Varying Culture Collection and Potential Infection Restriction Definitions

eTable 12. Predictive Ability of Sepsis Prognosis Tools Under Varying Recent Antibiotic Definitions

eReferences

1.
Kumar  G, Ahmad  S, Taneja  A, Patel  J, Guddati  AK, Nanchal  R; Milwaukee Initiative in Critical Care Outcomes Research Group of Investigators.  Severe sepsis in hematopoietic stem cell transplant recipients.   Crit Care Med. 2015;43(2):411-421. doi:10.1097/CCM.0000000000000714PubMedGoogle ScholarCrossref
2.
Mikulska  M, Del Bono  V, Bruzzi  P,  et al.  Mortality after bloodstream infections in allogeneic haematopoietic stem cell transplant (HSCT) recipients.   Infection. 2012;40(3):271-278. doi:10.1007/s15010-011-0229-yPubMedGoogle ScholarCrossref
3.
Weber  D, Jenq  RR, Peled  JU,  et al.  Microbiota disruption induced by early use of broad-spectrum antibiotics is an independent risk factor of outcome after allogeneic stem cell transplantation.   Biol Blood Marrow Transplant. 2017;23(5):845-852. doi:10.1016/j.bbmt.2017.02.006PubMedGoogle ScholarCrossref
4.
Golob  JL, Pergam  SA, Srinivasan  S,  et al.  Stool microbiota at neutrophil recovery is predictive for severe acute graft vs host disease after hematopoietic cell transplantation.   Clin Infect Dis. 2017;65(12):1984-1991. doi:10.1093/cid/cix699PubMedGoogle ScholarCrossref
5.
Holler  E, Butzhammer  P, Schmid  K,  et al.  Metagenomic analysis of the stool microbiome in patients receiving allogeneic stem cell transplantation: loss of diversity is associated with use of systemic antibiotics and more pronounced in gastrointestinal graft-versus-host disease.   Biol Blood Marrow Transplant. 2014;20(5):640-645. doi:10.1016/j.bbmt.2014.01.030PubMedGoogle ScholarCrossref
6.
Ogimi  C, Krantz  EM, Golob  JL,  et al.  Antibiotic exposure prior to respiratory viral infection is associated with progression to lower respiratory tract disease in allogeneic hematopoietic cell transplant recipients.   Biol Blood Marrow Transplant. 2018;24(11):2293-2301. doi:10.1016/j.bbmt.2018.05.016PubMedGoogle ScholarCrossref
7.
Galloway-Peña  JR, Jenq  RR, Shelburne  SA.  Can consideration of the microbiome improve antimicrobial utilization and treatment outcomes in the oncology patient?   Clin Cancer Res. 2017;23(13):3263-3268. doi:10.1158/1078-0432.CCR-16-3173PubMedGoogle ScholarCrossref
8.
Gouliouris  T, Warne  B, Cartwright  EJP,  et al.  Duration of exposure to multiple antibiotics is associated with increased risk of VRE bacteraemia: a nested case-control study.   J Antimicrob Chemother. 2018;73(6):1692-1699. doi:10.1093/jac/dky075PubMedGoogle ScholarCrossref
9.
Dadwal  S, Eftekharai  Z, Tushondra  T, Johnson  D.  A machine-learning based prediction model for sepsis in patients undergoing hematopoietic stem cell transplantation.   Blood. 2018;132(Suppl 1):711. doi:10.1182/blood-2018-99-117002Google ScholarCrossref
10.
Kochanek  M, Schalk  E, von Bergwelt-Baildon  M,  et al.  Management of sepsis in neutropenic cancer patients: 2018 guidelines from the Infectious Diseases Working Party (AGIHO) and Intensive Care Working Party (iCHOP) of the German Society of Hematology and Medical Oncology (DGHO).   Ann Hematol. 2019;98(5):1051-1069. doi:10.1007/s00277-019-03622-0PubMedGoogle ScholarCrossref
11.
Lind  ML, Phipps  AI, Mooney  S,  et al.  Predictive value of three clinical criteria for sepsis (qSOFA, SIRS, and NEWS) with respect to short-term mortality in allogeneic hematopoietic cell transplant recipients with suspected infections.   Clin Infect Dis. 2020;ciaa214. doi:10.1093/cid/ciaa214PubMedGoogle Scholar
12.
Polley  E, LeDell  E, Kennedy  C, Lendle  S, van der Laan  M. SuperLearner: Super Learner Prediction. Accessed April 26, 2019. https://CRAN.R-project.org/package=SuperLearner
13.
Kam  HJ, Kim  HY.  Learning representations for the early detection of sepsis with deep neural networks.   Comput Biol Med. 2017;89:248-255. doi:10.1016/j.compbiomed.2017.08.015PubMedGoogle ScholarCrossref
14.
Calvert  JS, Price  DA, Chettipally  UK,  et al.  A computational approach to early sepsis detection.   Comput Biol Med. 2016;74:69-73. doi:10.1016/j.compbiomed.2016.05.003PubMedGoogle ScholarCrossref
15.
Hämäläinen  S, Kuittinen  T, Matinlauri  I, Nousiainen  T, Koivula  I, Jantunen  E.  Severe sepsis in autologous stem cell transplant recipients: microbiological aetiology, risk factors and outcome.   Scand J Infect Dis. 2009;41(1):14-20. doi:10.1080/00365540802454706PubMedGoogle ScholarCrossref
16.
Kethireddy  S, Bilgili  B, Sees  A,  et al; Cooperative Antimicrobial Therapy of Septic Shock (CATSS) Database Research Group.  Culture-negative septic shock compared with culture-positive septic shock: a retrospective cohort study.   Crit Care Med. 2018;46(4):506-512. doi:10.1097/CCM.0000000000002924PubMedGoogle ScholarCrossref
17.
Gupta  S, Sakhuja  A, Kumar  G, McGrath  E, Nanchal  RS, Kashani  KB.  Culture-negative severe sepsis: nationwide trends and outcomes.   Chest. 2016;150(6):1251-1259. doi:10.1016/j.chest.2016.08.1460PubMedGoogle ScholarCrossref
18.
Seymour  CW, Liu  VX, Iwashyna  TJ,  et al.  Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).   JAMA. 2016;315(8):762-774. doi:10.1001/jama.2016.0288PubMedGoogle ScholarCrossref
19.
Abe  R, Oda  S, Sadahiro  T,  et al.  Gram-negative bacteremia induces greater magnitude of inflammatory response than gram-positive bacteremia.   Crit Care. 2010;14(2):R27. doi:10.1186/cc8898PubMedGoogle ScholarCrossref
20.
Alexandraki  I, Palacio  C.  Gram-negative versus gram-positive bacteremia: what is more alarmin(g)?   Crit Care. 2010;14(3):161. doi:10.1186/cc9013PubMedGoogle ScholarCrossref
21.
Kumar  A, Roberts  D, Wood  KE,  et al.  Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock.   Crit Care Med. 2006;34(6):1589-1596. doi:10.1097/01.CCM.0000217961.75225.E9PubMedGoogle ScholarCrossref
22.
Healthwise Staff. Fever temperatures: accuracy and comparison. Updated February 2020. Accessed May 15, 2020. https://www.mottchildren.org/health-library/tw9223
23.
Knaus  WA, Zimmerman  JE, Wagner  DP, Draper  EA, Lawrence  DE.  APACHE-acute physiology and chronic health evaluation: a physiologically based classification system.   Crit Care Med. 1981;9(8):591-597. doi:10.1097/00003246-198108000-00008PubMedGoogle ScholarCrossref
24.
Seymour  CW, Kahn  JM, Cooke  CR, Watkins  TR, Heckbert  SR, Rea  TD.  Prediction of critical illness during out-of-hospital emergency care.   JAMA. 2010;304(7):747-754. doi:10.1001/jama.2010.1140PubMedGoogle ScholarCrossref
25.
Vincent  JL, Moreno  R, Takala  J,  et al.  The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure: on behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine.   Intensive Care Med. 1996;22(7):707-710. doi:10.1007/BF01709751PubMedGoogle ScholarCrossref
26.
van der Laan  MJ, Rose  S.  Targeted Learning: Causal Inference for Observational and Experimental Data. Springer; 2011. doi:10.1007/978-1-4419-9782-1
27.
Benitez  AE, Musinguzi  N, Bangsberg  DR,  et al.  Super learner analysis of real-time electronically monitored adherence to antiretroviral therapy under constrained optimization and comparison to non-differentiated care approaches for persons living with HIV in rural Uganda.   J Int AIDS Soc. 2020;23(3):e25467. doi:10.1002/jia2.25467PubMedGoogle Scholar
28.
Houssaïni  A, Assoumou  L, Marcelin  AG, Molina  JM, Calvez  V, Flandre  P.  Investigation of Super Learner methodology on HIV-1 Small Sample: application on Jaguar trial data.   AIDS Res Treat. 2012;2012:478467. doi:10.1155/2012/478467PubMedGoogle Scholar
29.
Pirracchio  R, Petersen  ML, Carone  M, Rigon  MR, Chevret  S, van der Laan  MJ.  Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study.   Lancet Respir Med. 2015;3(1):42-52. doi:10.1016/S2213-2600(14)70239-5PubMedGoogle ScholarCrossref
30.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.   Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2PubMedGoogle ScholarCrossref
31.
Cox  DR.  Two further applications of a model for binary regression.   Biometrika. 1958;45(3-4):562-565. doi:10.1093/biomet/45.3-4.562Google ScholarCrossref
32.
Harrell  FE  Jr. Rms: regression modeling strategies. Accessed October 23, 2020. https://CRAN.R-project.org/package=rms
33.
Vermont  J, Bosson  JL, François  P, Robert  C, Rueff  A, Demongeot  J.  Strategies for graphical threshold determination.   Comput Methods Programs Biomed. 1991;35(2):141-150. doi:10.1016/0169-2607(91)90072-2PubMedGoogle ScholarCrossref
34.
Singer  M, Deutschman  CS, Seymour  CW,  et al.  The Third International Consensus definitions for sepsis and septic shock (Sepsis-3).   JAMA. 2016;315(8):801-810. doi:10.1001/jama.2016.0287PubMedGoogle ScholarCrossref
35.
Peter  E. Fbroc: Fast algorithms to bootstrap receiver operating characteristics curves. Accessed August 30, 2019. https://CRAN.R-project.org/package=fbroc
36.
López-Ratón  M, Rodríguez-Álvarez  MX, Cadarso-Suárez  C, Gude-Sampedro  F.  OptimalCutpoints: an R package for selecting optimal cutpoints in diagnostic tests.   J Stat Softw. 2014;61(1):1-36. doi:10.18637/jss.v061.i08Google Scholar
37.
Bone  RC, Grodzin  CJ, Balk  RA.  Sepsis: a new hypothesis for pathogenesis of the disease process.   Chest. 1997;112(1):235-243. doi:10.1378/chest.112.1.235PubMedGoogle ScholarCrossref
38.
Fleuren  LM, Klausch  TLT, Zwager  CL,  et al.  Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy.   Intensive Care Med. 2020;46(3):383-400. doi:10.1007/s00134-019-05872-yPubMedGoogle ScholarCrossref
39.
Greenwell  BM, Boehmke  BC, McCarthy  AJ.  A simple and effective model-based variable importance measure.   arXiv. Preprint published online May 12, 2018. Accessed January 28, 2021. https://arxiv.org/abs/1805.04755Google Scholar
40.
Lieberman  M, Morris  J.  The precise effect of multicollinearity on classification prediction.   Multiple Linear Regression Viewpoint. 2014;40(1):5-10.Google Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 2,630
    Citations 0
    Original Investigation
    Critical Care Medicine
    April 19, 2021

    Development and Validation of a Machine Learning Model to Estimate Bacterial Sepsis Among Immunocompromised Recipients of Stem Cell Transplant

    Author Affiliations
    • 1Department of Epidemiology, University of Washington, Seattle
    • 2Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Research Center, Seattle, Washington
    • 3Harborview Injury Prevention and Research Center, Seattle, Washington
    • 4Department of Biostatistics, University of Washington, Seattle
    • 5Department of Statistics, University of Washington, Seattle
    • 6Institute for Disease Modeling, Bellevue, Washington
    • 7Information School, University of Washington, Seattle
    • 8Department of Biology, New Mexico State University, Las Cruces
    • 9Division of Allergy and Infectious Disease, Department of Medicine, University of Washington, Seattle
    • 10Clinical Research Division, Fred Hutchinson Cancer Research Center, Seattle, Washington
    • 11Antimicrobial and Outpatient Parenteral Antimicrobial Therapy Program, Seattle Cancer Care Alliance, Seattle, Washington
    • 12Division of Pulmonary, Critical Care and Sleep Medicine, University of Washington, Seattle
    • 13Oncology and Bone Marrow Transplant Intensive Care Unit, University of Washington, Seattle
    • 14Medical Intensive Care Unit, University of Washington, Seattle
    • 15Division of Medical Oncology, University of Washington, Seattle
    • 16School of Medicine, University of Washington, Seattle
    • 17Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, Washington
    JAMA Netw Open. 2021;4(4):e214514. doi:10.1001/jamanetworkopen.2021.4514
    Key Points

    Question  Can machine learning be used with electronic medical record data to improve bacterial sepsis prediction among recipients of allogeneic hematopoietic cell transplant (allo-HCT)?

    Findings  In this prognostic study including 1943 recipients of allo-HCT, the population-specific full predictor bacterial sepsis decision support tool (SHBSL) had superior prognostic performance regardless of outcome or patient location compared with the clinical factor–specific SHBSL and existing tools. Additionally, SHBSL had higher positive predictive values relative to sensitivities than existing tools.

    Meaning  These findings suggest that, if used at the time of blood culture collection, the SHBSL could provide relevant information regarding bacterial sepsis risk and antibiotic needs of recipients of allo-HCT.

    Abstract

    Importance  Sepsis disproportionately affects recipients of allogeneic hematopoietic cell transplant (allo-HCT), and timely detection is crucial. However, the atypical presentation of sepsis within this population makes detection challenging, and existing clinical sepsis tools have limited prognostic value among this high-risk population.

    Objective  To develop a full risk factor (demographic, transplant, clinical, and laboratory factors) and clinical factor–specific automated bacterial sepsis decision support tool for recipients of allo-HCT with potential bloodstream infections (PBIs).

    Design, Setting, and Participants  This prognostic study used data from adult recipients of allo-HCT transplanted at the Fred Hutchinson Cancer Research Center, Seattle, Washington, between June 2010 and June 2019 randomly divided into 70% modeling and 30% validation data sets. Tools were developed using the area under the curve (AUC) optimized SuperLearner, and their performance was compared with existing clinical sepsis tools: National Early Warning Score (NEWS), quick Sequential Organ Failure Assessment (qSOFA), and Systemic Inflammatory Response Syndrome (SIRS), using the validation data set. Data were analyzed between January and October of 2020.

    Main Outcomes and Measures  The primary outcome was high–sepsis risk bacteremia (culture confirmed gram-negative species, Staphylococcus aureus, or Streptococcus spp bacteremia), and the secondary outcomes were 10- and 28-day mortality. Tool discrimination and calibration were examined using accuracy metrics and expected vs observed probabilities.

    Results  Between June 2010 and June 2019, 1943 recipients of allo-HCT received their first transplant, and 1594 recipients (median [interquartile range] age at transplant, 54 [43-63] years; 911 [57.2%] men; 1242 individuals [77.9%] identifying as White) experienced at least 1 PBI. Of 8131 observed PBIs, 238 (2.9%) were high–sepsis risk bacteremia. Compared with high–sepsis risk bacteremia, the full decision support tool had the highest AUC (0.85; 95% CI, 0.81-0.89), followed by the clinical factor–specific tool (0.72; 95% CI, 0.66-0.78). SIRS had the highest AUC of existing tools (0.64; 95% CI, 0.57-0.71). The full decision support tool had the highest AUCs for PBIs identified in inpatient (0.82; 95% CI, 0.76-0.89) and outpatient (0.82; 95% CI, 0.75-0.89) settings and for 10-day (0.85; 95% CI, 0.79-0.91) and 28-day (0.80; 95% CI, 0.75-0.84) mortality.

    Conclusions and Relevance  These findings suggest that compared with existing tools and the clinical factor–specific tool, the full decision support tool had superior prognostic accuracy for the primary (high–sepsis risk bacteremia) and secondary (short-term mortality) outcomes in inpatient and outpatient settings. If used at the time of culture collection, the full decision support tool may inform more timely sepsis detection among recipients of allo-HCT.

    Introduction

    Sepsis disproportionately affects immunocompromised populations, including recipients of allogeneic hematopoietic cell transplant (allo-HCT).1 Early empirical broad-spectrum antibiotic use among recipients of HCT with bacterial sepsis has been associated with 3-fold reduced risk of 30-day mortality.2 However, antibiotic use is associated with increased antibiotic resistance, hospital-acquired infections, and, in recipients of HCT, may have detrimental microbiota-mediated associations with mortality.3-8 To minimize these outcomes, early and accurate sepsis detection among recipients of allo-HCT is crucial.

    However, sepsis is challenging to diagnose among recipients of allo-HCT. In individuals who are immunosuppressed, sepsis has less common presentations, manifests with subtle clinical findings, and progresses more rapidly than in immunocompetent populations.9 Further, common complications of HCT, such as engraftment syndrome, anemia, and drug adverse effects, can present with similar signs and symptoms as sepsis.9,10 Unsurprisingly, current clinical sepsis tools, which were built among patients from the general population, have limited prognostic value among recipients of allo-HCT, and their use may lead to missed sepsis events and antibiotic misuse.11

    With the goal of improving the accuracy of bacterial sepsis risk estimation at the time of clinical infection consideration (blood culture collection) in inpatient and outpatient settings, we developed 2 automated bacterial sepsis decision support tools for recipients of allo-HCT with potential bloodstream infections (PBIs) using a state-of-the-art stacked ensembling technique, the SuperLearner.12 Our tools, super HCT bacterial sepsis learner (SHBSL) and clinical factor–specific super HCT bacterial sepsis learner (C-SHBSL), have the potential to be integrated into electronic medical record (EMR) systems and to provide treatment decision support in the form of patient-specific sepsis risk probability estimates or flags for patient who are at high risk at the time of blood culture collection (eFigure 1 in the Supplement). Through retrospective evaluation, we assessed whether our population specific tools led to more accurate risk estimation among recipients of allo-HCT than commonly used existing tools.

    Methods

    This prognostic study was approved by the Fred Hutchinson Cancer Research Center (FHCRC) institutional review board. All participants provided written informed consent. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guidelines.

    Study Overview, Population, and Data Collection

    We identified adult (ie, age ≥18 years) recipients of allo-HCT who agreed to research participation and received their first transplant at the FHCRC or Seattle Cancer Care Alliance between June 2010 and June 2019. Data from the initial 100 days after transplant, a period of close patient monitoring by inpatient and outpatient transplant teams, or until patient death, if prior to day 100, were retrospectively collected.

    Data were queried from a prospectively maintained transplant database and EMRs. We defined outliers as measurements outside of biologically plausible ranges and, following a EMR review verification of a randomly selected 10% of outliers, dropped outliers prior to missing value imputation.

    Cohort Development

    We developed and evaluated the tools among all PBIs that occurred within the study. We defined PBI as any blood culture, including surveillance cultures, collected in the absence of recent (ie, past 3 days) antibiotic administration or prescription other than the center recommended neutropenia prophylaxis (ie, levofloxacin). We considered multiple cultures collected within 1 hour to be the same event (culture time defined as time of first nonsurveillance culture) and excluded follow-up cultures, defined as cultures collected continuously (within 36 hours of prior culture) for the 14 days after a culture with positive results or until distinct organisms were identified, whichever came first. Patients for whom any PBIs were identified were randomly divided into a modeling (70%) or validation (30%) group, and PBIs were assigned to the modeling or validation data set based on the patient’s grouping.

    Prophylaxis and Culture Collection Guidelines

    FHCRC guidelines during the study have been described elsewhere.11 Briefly, cultures were recommended for patients with fevers, and surveillance cultures were recommended weekly or biweekly for patients receiving high-dose glucocorticoids (ie, >0.5 mg/kg). Levofloxacin was used as the primary antibiotic prophylaxis agent during neutropenic periods while patients with neutropenic fever received empirical therapy with ceftazidime, cefepime, or meropenem.

    Outcome and Factors

    The atypical presentation of sepsis among recipients of HCT prevented us from using a symptom-based primary end point.13,14 Instead, we selected bacteremia with high risk for sepsis (ie, blood culture–confirmed gram-negative organisms, Staphylococcus aureus, or Streptococcus spp bacteremia) based on evidence that sepsis is significantly more likely among recipients of HCT with PBIs and the selected bloodstream infections than without (eAppendix 1 in the Supplement).15

    Because our end point excludes sepsis events with negative results in blood cultures,16,17 we feared our tools may capture bacteremia but not sepsis risk. To test the ability of our tools to capture sepsis risk in general, not just bacteremia risk, we examined 2 short-term mortality secondary outcomes (ie, 10-day and 28-day mortality) under the assumption that mortality was more common among patients with sepsis than without.18 However, unlike a 2016 study by Seymour et al,18 we chose against a primary mortality end point because we were unable to account for the association between mortality and antimicrobial and supportive treatment undergone by most patients in our retrospectively collected sample.19-21

    We considered bedside assessable demographic and clinical factors as potential factors in C-SHBSL. For SHBSL, we additionally included a wide array of comorbidities, transplant factors, biomarkers, vitals, and recent infection data (eTable 1 in the Supplement). Additionally, in SHBSL, we included binary indicators of measurement missingness and, to account for trends in numerical factors during the window of measurement collection, linear change-over-time factors developed using univariable linear regressions.

    Factors were standardized to a uniform unit (eg, weight in kg), and lesser- or greater-than values were converted to the first number included in the statement (eg, 35 °C for ≤35 °C). Temperatures were standardized to oral temperatures by adding 0.45 °C to axillary measurements and subtracting 0.2 °C from arterial measurements.22 Binary (0 or 1) variables were developed from categorical factors, all of which were nominal. Each factor was standardized to a mean of zero and SD of 1.

    Data for the 24 hours prior to and 2 hours after culture collection were extracted, and the measurement taken closest in time to collection was retained. Potential factors with more than 35% missingness were excluded. Missingness and change-over-time factors were included if the static measurement was included. Missing observations for frequently captured factors were filled using a single normal value imputation approach (eAppendix 2, eFigure 2, and eTable 2 in the Supplement).18,23-25

    Tool Development

    We developed our tools in the modeling data set using the Super Learner with 10-fold cross-validation, optimized with respect to area under the receiver operating characteristic curve (AUROC). The Super Learner is a statistical learning technique that uses cross-validation to construct an optimal ensemble (weighted mean) of user-supplied candidate algorithms.26 The resulting model, the Super Learner, has been proven in large samples to perform as well as the best performing ensemble of specified algorithms and shown in small samples to perform essentially as well as the best performing ensemble of specified algorithms.26-28 To optimize the resulting Super Learners, our library included a broad selection of parametric, semiparametric, and nonparametric learners and both linear and nonlinear algorithms. Additionally, our library included hyperparameter and screening algorithm variants (eAppendix 3 in the Supplement). As with the modeling and validation randomization, observations were partitioned into cross-validation folds at the patient level.

    Statistical Analysis
    Tool Calibration and Discrimination

    Using the validation data set, we compared the calibration and discrimination of our tools to 3 existing clinical sepsis tools available for use in outpatient settings: the National Early Warning Score (NEWS), quick Sequential Organ Failure Assessment (qSOFA), and Systemic Inflammatory Response Syndrome (SIRS). We assessed calibration by comparing the mean estimated vs observed outcome probabilities and examined calibration plots.29-32 To evaluate discrimination, we compared AUROCs, area under the precision and recall curve (pr-AUC), categorical net reclassification index (NRI) (eAppendix 4 in the Supplement), and predictive accuracy metrics (ie, sensitivity, specificity, negative predictive value [NPV], and positive predictive value [PPV]) for all cut points (primary analysis), and we compared qSOFA/SIRS at a cut point of 2 or greater, NEWS at cut points of 4 or greater and 7 or greater, and SHBSL/C-SHBSL (primary analysis selected optimal [Upper-Left Method33] cut points) for secondary and sensitivity analyses. Optimal cut points (upper-left) were identified for the primary and secondary analyses. To visually compare tool discrimination, we generated ROC and precision and recall curves.

    We performed secondary analyses assessing tool performance by inpatient status at time of culture collection and regarding short-term (10- and 28-day) mortality.18,34 Inpatient status was defined for each calendar day and patients were considered inpatient for the full day of hospital admission, discharge and during intensive care unit (ICU) stays. Additionally, we estimated the cross-validated AUROCs of our tools using 10-fold nested cross-validation and the bootstrapped AUROCs of existing tools using 1000 modeling and validation split variants. All analyses were performed using R statistical software version 3.6.2 (R Project for Statistical Computing) base functions and functions from the rms, fbroc, OptimalCutpoints, and SuperLearner packages.12,32,35,36 P values were 2-sided, and statistical significance was set at α = .05. Data were analyzed from between January and October of 2020.

    Sensitivity Analyses

    We performed several sensitivity analyses to examine the generalizability of our tools. Specifically, we explored whether our evaluation results were robust to varying demographic characteristic, missingness, recent antibiotic use, data collection, and culture collection and inclusion scenarios. The sensitivity analyses were performed using the validation data set and are described in eAppendix 6 in the Supplement.

    Results
    Study Population and Cohort

    Between June 2010 and June 2019, 1943 individuals received their first allo-HCT at the Seattle Cancer Care Alliance or FHCRC, and 1594 individuals (median [interquartile range] age at transplant, 54 [43-63] years; 911 [57.2%] men; 1242 individuals [77.9%] identifying as White) experienced at least 1 PBI (Table 1; eFigure 3 in the Supplement). During follow-up, these patients experienced 8131 PBIs, including 6046 PBIs (74.4%) identified in outpatient settings and 2085 PBIs (25.6%) in inpatient settings (eFigure 4 in the Supplement). Patients who developed high-risk bacteremia, compared with those who did not, were more likely to experience grade 4 acute graft-vs-host disease (19 patients [9.8%] vs 38 patients [2.7%]) and die during follow-up (33 patients [17.1%] vs 105 patients [7.5%]) (Table 1).

    Of 8131 PBIs, 238 (2.9%) were defined as high–sepsis risk bacteremia, 258 (3.2%) occurred with 28 days of death, and 74 (0.9%) occurred within 10 days of death. The modeling data set contained 5845 PBIs, of which 168 (2.9%) high–sepsis risk bacteremia. The validation data set included 2286 PBIs, of which 70 PBIs (3.1%) were high–sepsis risk bacteremia. Patients randomized to the modeling and validation groups had similar demographic and medical characteristics (eTable 3 in the Supplement). Of 35 vital measurements or biomarkers examined, 17 were discarded owing to excessive missingness (ie, >35%; eTable 1 in the Supplement).

    Tool Calibration and Discrimination

    In detecting patients with high-risk bacteremia, SHBSL had the highest AUROC (0.85; 95% CI, 0.81-0.89) and pr-AUC (0.20), and C-SHBSL had the second highest (AUROC: 0.72; 95% CI, 0.66-0.78; pr-AUC: 0.13). qSOFA had the lowest AUROC (0.51, 95% CI, 0.46-0.56) and all 3 existing tools had pr-AUCs of 0.07 (Table 2, Figure 1A and B). At optimal cut points, SHBSL had a sensitivity of 80.0% (95% CI, 69.2%-87.7%) and a specificity of 72.8% (95% CI, 69.5%-75.8%), and C-SHBSL had a sensitivity of 65.7% (95% CI, 51.9%-77.3%) and a specificity of 66.9% (95% CI, 64.0%-69.7%) (Table 2). At cut points with sensitivities of 90% or greater, SHBSL had a PPV of 7.3% or less and C-SHBSL had a PPV of 4% or less (eTable 4 in the Supplement). All tools had high NPVs (range, 96.9%-100%) (eTable 4 in the Supplement). Compared with the other tools, SHBSL had significantly better risk classification (C-SHBSL: NRI, 0.37; 95% CI, 0.20 - 0.54; P < .001; SIRS: NRI, 0.52; 95% CI, 0.34-0.69; P < .001; qSOFA: NRI, 0.31; 95% CI, 0.13-0.49; P < .001; NEWS: NRI, 0.35; 95% CI, 0.20-0.49; P < .001) (eTable 5 in the Supplement).

    SHBSL had the highest AUROC for secondary analyses. Among location stratified cohorts, SHBSL had an AUROC of 0.82 (95% CI, 0.76-0.89) among PBIs identified in inpatient settings and 0.82 (95% CI, 0.75-0.89) among PBIs identified in outpatient settings. Inpatient PBIS had higher estimated scores than outpatient PBIs (eFigure 5 in the Supplement), and the optimal cut points for SHBSL, C-SBHSL, and NEWS varied by location (Table 2). SHBSL had an AUROC of 0.85 (95% CI, 0.79-0.91) for 10-day mortality and 0.80 (95% CI, 0.75-0.84) for 28-day mortality. When compared with the primary analysis (all PBIs), existing tools had lower outpatient AUROCs but higher short-term mortality AUROCs (Table 3 and Figure 2). SHBSL had a cross-validated AUC of 0.85 (95% CI, 0.70-0.91), and C-SHBSL had a cross-validated AUC of 0.74 (95% CI, 0.59-0.86) (eFigure 6 and eTable 6 in the Supplement). SIRS had the highest bootstrapped AUC, at 0.64 (95% CI, 0.52-0.77) (eTable 6 in the Supplement).

    The mean estimated high-risk bacteremia probability of each tool matched the observed proportion in the validation data set (3.1%; eTable 7 in the Supplement) The calibration plots show that SHBSL had the best overall calibration (ie, the estimated vs observed probability line falls closest to the ideal [diagonal] calibration line) (Figure 1C).

    Sensitivity Analyses

    SHBSL had the highest AUROC in all examined scenarios and a higher pr-AUC than existing tools under all examined scenarios (eTables 8-12 in the Supplement) and C-SHBSL for all but the scenario only including non-Hispanic White individuals (eTable 9 in the Supplement). SHBSL and C-SHBSL had lower AUROCs when the surveillance cultures were removed than in the primary analysis. Under varying culture collection (eTable 11 in the Supplement), racial/ethnic makeup (eTable 9 in the Supplement), and missingness (eTable 8 in the Supplement) scenarios, SHBSL had higher AUROCs than those of existing tools, and a primary analysis selected optimal cut point sensitivity between 70.0% and 94.1% and specificity between 72.3% and 75.8%.

    Discussion

    In this prognostic study, we built 2 automated prognosis tools to provide sepsis-related treatment decision support at the time of blood culture collection among recipients of allo-HCT in inpatient or outpatient settings. We developed these tools using a state-of-the-art ensemble modeling technique (the Super Learner) and included demographic and vital factors in both tools as well as a wide array of clinical and transplant factors in SHBSL. Compared with existing clinical sepsis tools, these tools had higher estimated discrimination and calibration for high–sepsis risk bacteremia and short-term mortality among recipients of allo-HCT with PBIs.

    Regardless of cohort, sepsis proxy end point, or data scenario, SHBSL had a moderate to high AUROC, and C-SHBSL had a moderate AUROC, and existing tools had low to moderate AUROCs. In addition, SHBSL had higher PPVs for all sensitivities than existing tools and relative to all sensitivities greater than 20% compared with C-SHBSL. These results suggests that the additional data cleaning and processing required to run SHBSL instead of C-SHBSL are justified, that SHBSL can be run on complete or incomplete data, the superior observed performance of SHBSL is likely generalizable to other transplant centers, SBHSL can be run in inpatient or outpatient settings (but location-specific cut points are needed), and SHBSL would likely result in fewer false flags relative to any sensitivity than existing tools. Finally, the fact that SHBSL had superior prognostic performance irrespective of sepsis-related end point suggests that, compared with existing tools, SHBSL is better able to estimate sepsis risk, not just high-risk bacteremia, among recipients of allo-HCT with PBIs.

    We observed lower than expected sensitivities for NEWS and SIRS for short-term mortality. Previous 10-day mortality sensitivity estimates for NEWS at the 7 or greater cut point and SIRS at the 2 or greater cut point among recipients of allo-HCT were 62 percentage points higher for NEWS at the 7 or greater cut point and 46 percentage points higher for SIRS at the 2 or greater cut point than our estimates.11 This difference is likely driven by our decision to evaluate potential infections (ie, all blood cultures) instead of the commonly used suspected infections (ie, body fluid cultures collected within a specific antibiotic time epoch) as well as missingness frequency differences (suspected infection study temperature missingness, 6%; temperature missingness in this study, 35%).11,18 While examining PBIs likely contributed to our large amount of missingness and meant we included some cultures in the absence of clinical suspicion of infection (ie, surveillance cultures), we evaluated PBIs rather than suspected infections because our primary end point required blood culture confirmation, and we wanted our tools to inform antibiotic treatment use rather than estimate disease escalation among patients already receiving antibiotic treatment.34,37 To ensure our tools were not biased toward surveillance cultures, we performed a sensitivity analysis excluding surveillance cultures and found that neither C-SHBSL nor SHBSL performed meaningfully different.

    Unlike most automated sepsis estimation tools, which were built to improve early detection of clinically identified sepsis through real-time estimation, we built SBHSL and C-SHBSL to improve the accuracy of the sepsis risk estimation itself. This decision was driven by the atypical presentation of sepsis within our study population population, the desire to build tools functional in both inpatient and outpatient settings (limited continuous data), the limited accuracy of existing clinical scores, and reliance on outcomes that require cultures (absence of culture may not equal negative culture). While this prevents direct comparison between SBHSL or C-SHBSL and real-time tools, the performances of real-time tools provide a reference of what may be considered good. According to a 2020 systematic review,38 real-time tools and SHBSL had similar AUROCs. This comparison demonstrates that not only does SHBSL have superior performance compared with existing clinical tools, its performance is numerically comparable to state-of-the-art real-time estimation tools.

    Strengths and Limitations

    Our study has several strengths. First, the 100-day observation of our sample allowed us to develop bacterial sepsis prognosis tools for recipients throughout their most immunocompromised periods regardless of their inpatient status. Second, our data-driven modeling approach allowed the unique presentation of bacterial sepsis within this population to be captured. Third, we showed that our prognosis tool evaluation was robust to numerous assumptions made, including those regarding missing data and culture inclusion. Fourth, through secondary analyses using alternative end points, we were able to show that SHBSL captured risk, regardless of proxy end point, with similarly high AUROC.

    Our study also has several limitations. First, our sample is reflective of the northwestern US transplant community but may not reflect the larger international transplant population. Second, our tools rely on culture collection and are potentially biased toward our center’s practices. Third, our analyses relied on retrospective data and suffered from data collection and cleaning related biases. Such limitations are not unique to our study and stem from our inability to evaluate our tools externally or prospectively at the time of this study. Although we performed numerous sensitivity analyses testing our tools’ generalizability, future analyses should validate them among prospectively collected and external data. In addition, while it is possible to estimate factor importance from the Super Learner, the inclusion of correlated factors (eg, weight and height) prevented reliable estimation.39,40 Furthermore, we were unable to account for potential within-person correlation in our AUROC and NRI uncertainty estimates; 95% CIs accounting for this may differ slightly from those presented. Additionally, despite rapid advancement in EMR systems, EMR integration of automated tools, including ours, would be costly and, in the absence of a well-maintained and organized system, may be impossible. Furthermore, the Super Learner ensembles had lower (<1%) cross-validated AUCs than a supplied algorithm; however, we presented the ensembles as our final models because they provide the least biased options for future updates (eAppendix 5 in the Supplement).

    Conclusions

    In this prognostic study, we built 2 automated bacterial sepsis decision support tools for recipients of allo-HCT with PBIs in either inpatient or outpatient settings. When compared with existing tools and C-SHBSL, SHBSL had superior performance regardless of patient location at time of culture collection and examined sepsis-related end point. Our results suggest that, if SHBSL were to be run at the time of blood culture collection, it would provide clinically relevant information regarding bacterial sepsis risk and antibiotic initiation needs of recipients of allo-HCT.

    Back to top
    Article Information

    Accepted for Publication: February 15, 2021.

    Published: April 19, 2021. doi:10.1001/jamanetworkopen.2021.4514

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Lind ML et al. JAMA Network Open.

    Corresponding Author: Margaret L. Lind, MPH, Department of Epidemiology, University of Washington, 3980 15th Ave NE, Seattle, WA 98195 (mllind89@uw.edu).

    Author Contributions: Ms Lind had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs Pergam and Phipps are co–senior authors.

    Concept and design: Lind, Mooney, Carone, Althouse, Evans, Patel, Pergam, Phipps.

    Acquisition, analysis, or interpretation of data: Lind, Mooney, Carone, Liu, Vo, Pergam.

    Drafting of the manuscript: Lind, Mooney, Evans, Patel, Pergam.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Lind, Carone.

    Obtained funding: Lind.

    Administrative, technical, or material support: Mooney, Althouse, Pergam, Phipps.

    Supervision: Carone, Althouse, Pergam, Phipps.

    Conflict of Interest Disclosures: Ms Lind reported receiving grants from ECHO Funding Group (including Bill and Melinda Gates Foundation, US Agency for International Development, and the President’s Emergency Plan for AIDS Relief), the Swedish International Development Cooperation Agency supported by the European Union, the South African Medical Research Council, and the United Nations Population Fund outside the submitted work. Dr Evans reported serving on the steering committee of the Surviving Sepsis Campaign and as the cochair of the Adult Surviving Sepsis Campaign Guidelines. Dr Pergam reported participating in clinical trials for Chimerix, Merck, and Sanofi and grants from Global Life Technologies. No other disclosures were reported.

    Funding/Support: Ms Lind is supported by a grant from the National Center for Advancing Translational Sciences of the National Institutes of Health (grant No. TL1 TR002318) and a Ruth L. Kirschstein Predoctoral Individual National Research Service Award Diversity Award from National Heart, Lung, and Blood Institute (award No. F31HL154509). Dr Mooney is supported by a grant from the National Library of Medicine (grant No. R00LM012868). This research was also supported in part by National Cancer Institute grant No. CA-15704.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    Meeting Presentation: A preliminary subset of this study was presented in abstract form at IDWeek 2020 joint annual meeting of the Infectious Diseases Society of America, Society for Healthcare Epidemiology of America, the HIV Medical Association, the Pediatric Infectious Diseases Society, and the Society of Infectious Diseases Pharmacists; held online from October 21, 2020 to September 30, 2021.

    Additional Contributions: Ryan Basom, BS, and Howard Gary Schoch, BA (Clinical Research Data Systems, Fred Hutchinson Cancer Research Center), and John Klaassen, BA (Seattle Cancer Care Alliance), helped collect and clean the data. No direct payments were made to these nonauthor contributors for this work.

    References
    1.
    Kumar  G, Ahmad  S, Taneja  A, Patel  J, Guddati  AK, Nanchal  R; Milwaukee Initiative in Critical Care Outcomes Research Group of Investigators.  Severe sepsis in hematopoietic stem cell transplant recipients.   Crit Care Med. 2015;43(2):411-421. doi:10.1097/CCM.0000000000000714PubMedGoogle ScholarCrossref
    2.
    Mikulska  M, Del Bono  V, Bruzzi  P,  et al.  Mortality after bloodstream infections in allogeneic haematopoietic stem cell transplant (HSCT) recipients.   Infection. 2012;40(3):271-278. doi:10.1007/s15010-011-0229-yPubMedGoogle ScholarCrossref
    3.
    Weber  D, Jenq  RR, Peled  JU,  et al.  Microbiota disruption induced by early use of broad-spectrum antibiotics is an independent risk factor of outcome after allogeneic stem cell transplantation.   Biol Blood Marrow Transplant. 2017;23(5):845-852. doi:10.1016/j.bbmt.2017.02.006PubMedGoogle ScholarCrossref
    4.
    Golob  JL, Pergam  SA, Srinivasan  S,  et al.  Stool microbiota at neutrophil recovery is predictive for severe acute graft vs host disease after hematopoietic cell transplantation.   Clin Infect Dis. 2017;65(12):1984-1991. doi:10.1093/cid/cix699PubMedGoogle ScholarCrossref
    5.
    Holler  E, Butzhammer  P, Schmid  K,  et al.  Metagenomic analysis of the stool microbiome in patients receiving allogeneic stem cell transplantation: loss of diversity is associated with use of systemic antibiotics and more pronounced in gastrointestinal graft-versus-host disease.   Biol Blood Marrow Transplant. 2014;20(5):640-645. doi:10.1016/j.bbmt.2014.01.030PubMedGoogle ScholarCrossref
    6.
    Ogimi  C, Krantz  EM, Golob  JL,  et al.  Antibiotic exposure prior to respiratory viral infection is associated with progression to lower respiratory tract disease in allogeneic hematopoietic cell transplant recipients.   Biol Blood Marrow Transplant. 2018;24(11):2293-2301. doi:10.1016/j.bbmt.2018.05.016PubMedGoogle ScholarCrossref
    7.
    Galloway-Peña  JR, Jenq  RR, Shelburne  SA.  Can consideration of the microbiome improve antimicrobial utilization and treatment outcomes in the oncology patient?   Clin Cancer Res. 2017;23(13):3263-3268. doi:10.1158/1078-0432.CCR-16-3173PubMedGoogle ScholarCrossref
    8.
    Gouliouris  T, Warne  B, Cartwright  EJP,  et al.  Duration of exposure to multiple antibiotics is associated with increased risk of VRE bacteraemia: a nested case-control study.   J Antimicrob Chemother. 2018;73(6):1692-1699. doi:10.1093/jac/dky075PubMedGoogle ScholarCrossref
    9.
    Dadwal  S, Eftekharai  Z, Tushondra  T, Johnson  D.  A machine-learning based prediction model for sepsis in patients undergoing hematopoietic stem cell transplantation.   Blood. 2018;132(Suppl 1):711. doi:10.1182/blood-2018-99-117002Google ScholarCrossref
    10.
    Kochanek  M, Schalk  E, von Bergwelt-Baildon  M,  et al.  Management of sepsis in neutropenic cancer patients: 2018 guidelines from the Infectious Diseases Working Party (AGIHO) and Intensive Care Working Party (iCHOP) of the German Society of Hematology and Medical Oncology (DGHO).   Ann Hematol. 2019;98(5):1051-1069. doi:10.1007/s00277-019-03622-0PubMedGoogle ScholarCrossref
    11.
    Lind  ML, Phipps  AI, Mooney  S,  et al.  Predictive value of three clinical criteria for sepsis (qSOFA, SIRS, and NEWS) with respect to short-term mortality in allogeneic hematopoietic cell transplant recipients with suspected infections.   Clin Infect Dis. 2020;ciaa214. doi:10.1093/cid/ciaa214PubMedGoogle Scholar
    12.
    Polley  E, LeDell  E, Kennedy  C, Lendle  S, van der Laan  M. SuperLearner: Super Learner Prediction. Accessed April 26, 2019. https://CRAN.R-project.org/package=SuperLearner
    13.
    Kam  HJ, Kim  HY.  Learning representations for the early detection of sepsis with deep neural networks.   Comput Biol Med. 2017;89:248-255. doi:10.1016/j.compbiomed.2017.08.015PubMedGoogle ScholarCrossref
    14.
    Calvert  JS, Price  DA, Chettipally  UK,  et al.  A computational approach to early sepsis detection.   Comput Biol Med. 2016;74:69-73. doi:10.1016/j.compbiomed.2016.05.003PubMedGoogle ScholarCrossref
    15.
    Hämäläinen  S, Kuittinen  T, Matinlauri  I, Nousiainen  T, Koivula  I, Jantunen  E.  Severe sepsis in autologous stem cell transplant recipients: microbiological aetiology, risk factors and outcome.   Scand J Infect Dis. 2009;41(1):14-20. doi:10.1080/00365540802454706PubMedGoogle ScholarCrossref
    16.
    Kethireddy  S, Bilgili  B, Sees  A,  et al; Cooperative Antimicrobial Therapy of Septic Shock (CATSS) Database Research Group.  Culture-negative septic shock compared with culture-positive septic shock: a retrospective cohort study.   Crit Care Med. 2018;46(4):506-512. doi:10.1097/CCM.0000000000002924PubMedGoogle ScholarCrossref
    17.
    Gupta  S, Sakhuja  A, Kumar  G, McGrath  E, Nanchal  RS, Kashani  KB.  Culture-negative severe sepsis: nationwide trends and outcomes.   Chest. 2016;150(6):1251-1259. doi:10.1016/j.chest.2016.08.1460PubMedGoogle ScholarCrossref
    18.
    Seymour  CW, Liu  VX, Iwashyna  TJ,  et al.  Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).   JAMA. 2016;315(8):762-774. doi:10.1001/jama.2016.0288PubMedGoogle ScholarCrossref
    19.
    Abe  R, Oda  S, Sadahiro  T,  et al.  Gram-negative bacteremia induces greater magnitude of inflammatory response than gram-positive bacteremia.   Crit Care. 2010;14(2):R27. doi:10.1186/cc8898PubMedGoogle ScholarCrossref
    20.
    Alexandraki  I, Palacio  C.  Gram-negative versus gram-positive bacteremia: what is more alarmin(g)?   Crit Care. 2010;14(3):161. doi:10.1186/cc9013PubMedGoogle ScholarCrossref
    21.
    Kumar  A, Roberts  D, Wood  KE,  et al.  Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock.   Crit Care Med. 2006;34(6):1589-1596. doi:10.1097/01.CCM.0000217961.75225.E9PubMedGoogle ScholarCrossref
    22.
    Healthwise Staff. Fever temperatures: accuracy and comparison. Updated February 2020. Accessed May 15, 2020. https://www.mottchildren.org/health-library/tw9223
    23.
    Knaus  WA, Zimmerman  JE, Wagner  DP, Draper  EA, Lawrence  DE.  APACHE-acute physiology and chronic health evaluation: a physiologically based classification system.   Crit Care Med. 1981;9(8):591-597. doi:10.1097/00003246-198108000-00008PubMedGoogle ScholarCrossref
    24.
    Seymour  CW, Kahn  JM, Cooke  CR, Watkins  TR, Heckbert  SR, Rea  TD.  Prediction of critical illness during out-of-hospital emergency care.   JAMA. 2010;304(7):747-754. doi:10.1001/jama.2010.1140PubMedGoogle ScholarCrossref
    25.
    Vincent  JL, Moreno  R, Takala  J,  et al.  The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure: on behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine.   Intensive Care Med. 1996;22(7):707-710. doi:10.1007/BF01709751PubMedGoogle ScholarCrossref
    26.
    van der Laan  MJ, Rose  S.  Targeted Learning: Causal Inference for Observational and Experimental Data. Springer; 2011. doi:10.1007/978-1-4419-9782-1
    27.
    Benitez  AE, Musinguzi  N, Bangsberg  DR,  et al.  Super learner analysis of real-time electronically monitored adherence to antiretroviral therapy under constrained optimization and comparison to non-differentiated care approaches for persons living with HIV in rural Uganda.   J Int AIDS Soc. 2020;23(3):e25467. doi:10.1002/jia2.25467PubMedGoogle Scholar
    28.
    Houssaïni  A, Assoumou  L, Marcelin  AG, Molina  JM, Calvez  V, Flandre  P.  Investigation of Super Learner methodology on HIV-1 Small Sample: application on Jaguar trial data.   AIDS Res Treat. 2012;2012:478467. doi:10.1155/2012/478467PubMedGoogle Scholar
    29.
    Pirracchio  R, Petersen  ML, Carone  M, Rigon  MR, Chevret  S, van der Laan  MJ.  Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study.   Lancet Respir Med. 2015;3(1):42-52. doi:10.1016/S2213-2600(14)70239-5PubMedGoogle ScholarCrossref
    30.
    Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.   Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2PubMedGoogle ScholarCrossref
    31.
    Cox  DR.  Two further applications of a model for binary regression.   Biometrika. 1958;45(3-4):562-565. doi:10.1093/biomet/45.3-4.562Google ScholarCrossref
    32.
    Harrell  FE  Jr. Rms: regression modeling strategies. Accessed October 23, 2020. https://CRAN.R-project.org/package=rms
    33.
    Vermont  J, Bosson  JL, François  P, Robert  C, Rueff  A, Demongeot  J.  Strategies for graphical threshold determination.   Comput Methods Programs Biomed. 1991;35(2):141-150. doi:10.1016/0169-2607(91)90072-2PubMedGoogle ScholarCrossref
    34.
    Singer  M, Deutschman  CS, Seymour  CW,  et al.  The Third International Consensus definitions for sepsis and septic shock (Sepsis-3).   JAMA. 2016;315(8):801-810. doi:10.1001/jama.2016.0287PubMedGoogle ScholarCrossref
    35.
    Peter  E. Fbroc: Fast algorithms to bootstrap receiver operating characteristics curves. Accessed August 30, 2019. https://CRAN.R-project.org/package=fbroc
    36.
    López-Ratón  M, Rodríguez-Álvarez  MX, Cadarso-Suárez  C, Gude-Sampedro  F.  OptimalCutpoints: an R package for selecting optimal cutpoints in diagnostic tests.   J Stat Softw. 2014;61(1):1-36. doi:10.18637/jss.v061.i08Google Scholar
    37.
    Bone  RC, Grodzin  CJ, Balk  RA.  Sepsis: a new hypothesis for pathogenesis of the disease process.   Chest. 1997;112(1):235-243. doi:10.1378/chest.112.1.235PubMedGoogle ScholarCrossref
    38.
    Fleuren  LM, Klausch  TLT, Zwager  CL,  et al.  Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy.   Intensive Care Med. 2020;46(3):383-400. doi:10.1007/s00134-019-05872-yPubMedGoogle ScholarCrossref
    39.
    Greenwell  BM, Boehmke  BC, McCarthy  AJ.  A simple and effective model-based variable importance measure.   arXiv. Preprint published online May 12, 2018. Accessed January 28, 2021. https://arxiv.org/abs/1805.04755Google Scholar
    40.
    Lieberman  M, Morris  J.  The precise effect of multicollinearity on classification prediction.   Multiple Linear Regression Viewpoint. 2014;40(1):5-10.Google Scholar
    ×