Development, Validation, and Clinical Utility Assessment of a Prognostic Score for 1-Year Unplanned Rehospitalization or Death of Adult Sepsis Survivors | Critical Care Medicine | JAMA Network Open | JAMA Network
[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.  Cumulative Incidence Plots for Unplanned Rehospitalization, Infection-Related Rehospitalization, and Mortality by Rehospitalization Status
Cumulative Incidence Plots for Unplanned Rehospitalization, Infection-Related Rehospitalization, and Mortality by Rehospitalization Status

A, The cumulative all-cause unplanned rehospitalization rates (blue line) at 30, 90, 180, and 365 days were 16.9%, 28.4%, 36.9%, and 46.9%, respectively. The cumulative infection-related rehospitalization rates (orange line) at 30, 90, 180, and 365 days were 7.7%, 14.1%, 19.4%, and 26.7%, respectively (see eTable 3 in the Supplement for further information). B, The cumulative all-cause mortality rates among patients with 1 or more rehospitalizations (blue line) at 30, 90, 180, and 365 days were 17.4%, 26.2%, 32.8%, and 40.5%, respectively. These are reported using Cox proportional hazards regression models.

Figure 2.  Description and Assessment of Clinical Prediction Model in Derivation Cohort
Description and Assessment of Clinical Prediction Model in Derivation Cohort

A, Distribution of prognostic score by strata. B, Cumulative incidence of unplanned rehospitalization or death at 1 year after hospital discharge by prognostic score strata. C, Sensitivity and specificity of the score at different cutoff points. D, Calibration plot compares the observed (dotted line) vs predicted (solid line) risk of outcome events of unplanned rehospitalizations or death in the first year after hospital discharge. The distribution of outcome over the deciles is indicated by the bar graphs at the bottom, with patients with outcomes represented above the line and those without outcomes represented below the line. AUROC indicates area under the receiver operating characteristic curve.

Figure 3.  Clinical Usefulness With Decision Curve Analysis (DCA) of Prognostic Score
Clinical Usefulness With Decision Curve Analysis (DCA) of Prognostic Score

A, The x-axis of DCA graphs refers to the threshold probability, which is the risk of 1-year unplanned rehospitalization or death between 0 and 1, above which the clinician or patient would choose treatment. The y-axis shows the net benefit in units of the benefit associated with correctly identifying 1 unplanned rehospitalization or death. Decision curves are shown for scores of 5 or more, 7 or more, and 11 or more. B, Proportions by score strata for rehospitalization, for deaths in those who experience 1 or more rehospitalization episodes within that strata, and deaths without rehospitalization during the 365-day follow-up period.

Table 1.  Clinical and Outcome Characteristics of the At-Risk Sepsis Survivors in the Derivation Cohort and the Validation Cohort
Clinical and Outcome Characteristics of the At-Risk Sepsis Survivors in the Derivation Cohort and the Validation Cohort
Table 2.  Predictors in the Final Model for Calculating the Prognostic Score in the Derivation Cohorta
Predictors in the Final Model for Calculating the Prognostic Score in the Derivation Cohorta
1.
Singer  M, Deutschman  CS, Seymour  CW,  et al.  The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).   JAMA. 2016;315(8):801-810. doi:10.1001/jama.2016.0287 PubMedGoogle ScholarCrossref
2.
Shankar-Hari  M, Harrison  DA, Rowan  KM.  Differences in impact of definitional elements on mortality precludes international comparisons of sepsis epidemiology—a cohort study illustrating the need for standardized reporting.   Crit Care Med. 2016;44(12):2223-2230. doi:10.1097/CCM.0000000000001876 PubMedGoogle ScholarCrossref
3.
Shankar-Hari  M, Harrison  DA, Rubenfeld  GD, Rowan  K.  Epidemiology of sepsis and septic shock in critical care units: comparison between Sepsis-2 and Sepsis-3 populations using a national critical care database.   Br J Anaesth. 2017;119(4):626-636. doi:10.1093/bja/aex234 PubMedGoogle ScholarCrossref
4.
Kaukonen  KM, Bailey  M, Suzuki  S, Pilcher  D, Bellomo  R.  Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012.   JAMA. 2014;311(13):1308-1316. doi:10.1001/jama.2014.2637 PubMedGoogle ScholarCrossref
5.
Prescott  HC, Angus  DC.  Enhancing recovery from sepsis: a review.   JAMA. 2018;319(1):62-75. doi:10.1001/jama.2017.17687 PubMedGoogle ScholarCrossref
6.
Shankar-Hari  M, Saha  R, Wilson  J,  et al.  Rate and risk factors for rehospitalisation in sepsis survivors: systematic review and meta-analysis.   Intensive Care Med. 2020;46(4):619-636. doi:10.1007/s00134-019-05908-3 PubMedGoogle ScholarCrossref
7.
Shankar-Hari  M, Ambler  M, Mahalingasivam  V, Jones  A, Rowan  K, Rubenfeld  GD.  Evidence for a causal link between sepsis and long-term mortality: a systematic review of epidemiologic studies.   Crit Care. 2016;20(1):101. doi:10.1186/s13054-016-1276-7 PubMedGoogle ScholarCrossref
8.
Shankar-Hari  M, Harrison  DA, Ferrando-Vivas  P, Rubenfeld  GD, Rowan  K.  Risk factors at index hospitalization associated with longer-term mortality in adult sepsis survivors.   JAMA Netw Open. 2019;2(5):e194900. doi:10.1001/jamanetworkopen.2019.4900 PubMedGoogle Scholar
9.
Norman  BC, Cooke  CR, Ely  EW, Graves  JA.  Sepsis-associated 30-day risk-standardized readmissions: analysis of a nationwide Medicare sample.   Crit Care Med. 2017;45(7):1130-1137. doi:10.1097/CCM.0000000000002476 PubMedGoogle ScholarCrossref
10.
Donnelly  JP, Hohmann  SF, Wang  HE.  Unplanned readmissions after hospitalization for severe sepsis at academic medical center-affiliated hospitals.   Crit Care Med. 2015;43(9):1916-1927. doi:10.1097/CCM.0000000000001147 PubMedGoogle ScholarCrossref
11.
Chang  DW, Tseng  CH, Shapiro  MF.  Rehospitalizations following sepsis: common and costly.   Crit Care Med. 2015;43(10):2085-2093. doi:10.1097/CCM.0000000000001159 PubMedGoogle ScholarCrossref
12.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement.   Ann Intern Med. 2015;162(1):55-63. doi:10.7326/M14-0697 PubMedGoogle ScholarCrossref
13.
Fitzgerald  M, Saville  BR, Lewis  RJ.  Decision curve analysis.   JAMA. 2015;313(4):409-410. doi:10.1001/jama.2015.37 PubMedGoogle ScholarCrossref
14.
Vickers  AJ, Elkin  EB.  Decision curve analysis: a novel method for evaluating prediction models.   Med Decis Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361 PubMedGoogle ScholarCrossref
15.
Young  JD, Goldfrad  C, Rowan  K.  Development and testing of a hierarchical method to code the reason for admission to intensive care units: the ICNARC Coding Method: Intensive Care National Audit & Research Centre.   Br J Anaesth. 2001;87(4):543-548. doi:10.1093/bja/87.4.543 PubMedGoogle ScholarCrossref
17.
Healthcare Cost and Utilization Project. Clinical Classifications Software (CCS) for ICD-9-CM. Accessed October 31, 2019. https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp
18.
Prescott  HC, Langa  KM, Iwashyna  TJ.  Readmission diagnoses after hospitalization for severe sepsis and other acute medical conditions.   JAMA. 2015;313(10):1055-1057. doi:10.1001/jama.2015.1410 PubMedGoogle ScholarCrossref
19.
Liu  V, Lei  X, Prescott  HC, Kipnis  P, Iwashyna  TJ, Escobar  GJ.  Hospital readmission and healthcare utilization following sepsis in community settings.   J Hosp Med. 2014;9(8):502-507. doi:10.1002/jhm.2197 PubMedGoogle ScholarCrossref
20.
Quach  S, Hennessy  DA, Faris  P, Fong  A, Quan  H, Doig  C.  A comparison between the APACHE II and Charlson Index Score for predicting hospital mortality in critically ill patients.   BMC Health Serv Res. 2009;9:129. doi:10.1186/1472-6963-9-129 PubMedGoogle ScholarCrossref
21.
Information Standards Board for Health and Social Care. Critical care minimum dataset full specification critical care minimum dataset version 8.0. Published March 2010. Accessed July 5, 2019. https://webarchive.nationalarchives.gov.uk/+/http://www.isb.nhs.uk/documents/isb-0153/amd-81-2010/0153812010spec.pdf
22.
Retter  A, Wyncoll  D, Pearse  R,  et al; British Committee for Standards in Haematology.  Guidelines on the management of anaemia and red cell transfusion in adult critically ill patients.   Br J Haematol. 2013;160(4):445-464. doi:10.1111/bjh.12143 PubMedGoogle ScholarCrossref
23.
Shankar-Hari  M, Phillips  GS, Levy  ML,  et al; Sepsis Definitions Task Force.  Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).   JAMA. 2016;315(8):775-787. doi:10.1001/jama.2016.0289 PubMedGoogle ScholarCrossref
24.
Thompson  CG, Kim  RS, Aloe  AM, Becker  BJ.  Extracting the variance inflation factor and other multicollinearity diagnostics from typical regression results.   Basic Appl Soc Psych. 2017;39(2):81-90. doi:10.1080/01973533.2016.1277529 Google ScholarCrossref
25.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.   Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2 PubMedGoogle ScholarCrossref
26.
Efron  B.  Estimating the error rate of a prediction rule—improvement on cross-validation.   J Am Stat Assoc. 1983;78(382):316-331. doi:10.1080/01621459.1983.10477973 Google ScholarCrossref
27.
Steyerberg  EW, Harrell  FE  Jr, Borsboom  GJ, Eijkemans  MJ, Vergouwe  Y, Habbema  JD.  Internal validation of predictive models: efficiency of some procedures for logistic regression analysis.   J Clin Epidemiol. 2001;54(8):774-781. doi:10.1016/S0895-4356(01)00341-9 PubMedGoogle ScholarCrossref
28.
Seaman  S, Galati  J, Jackson  D, Carlin  J.  What is meant by “missing at random”?   Stat Sci. 2013;28(2):257-268. doi:10.1214/13-STS415 Google ScholarCrossref
29.
Braun  L, Riedel  AA, Cooper  LM.  Severe sepsis in managed care: analysis of incidence, one-year mortality, and associated costs of care.   J Manag Care Pharm. 2004;10(6):521-530. doi:10.18553/jmcp.2004.10.6.521 PubMedGoogle Scholar
30.
Prescott  HC, Langa  KM, Liu  V, Escobar  GJ, Iwashyna  TJ.  Increased 1-year healthcare use in survivors of severe sepsis.   Am J Respir Crit Care Med. 2014;190(1):62-69. doi:10.1164/rccm.201403-0471OC PubMedGoogle ScholarCrossref
31.
Yende  S, Kellum  JA, Talisa  VB,  et al.  Long-term host immune response trajectories among hospitalized patients with sepsis.   JAMA Netw Open. 2019;2(8):e198686. doi:10.1001/jamanetworkopen.2019.8686 PubMedGoogle Scholar
32.
Taylor  SP, Chou  SH, Sierra  MF,  et al.  Association between adherence to recommended care and outcomes for adult survivors of sepsis.   Ann Am Thorac Soc. 2020;17(1):89-97. doi:10.1513/AnnalsATS.201907-514OC PubMedGoogle ScholarCrossref
33.
Deb  P, Murtaugh  CM, Bowles  KH,  et al.  Does early follow-up improve the outcomes of sepsis survivors discharged to home health care?   Med Care. 2019;57(8):633-640. doi:10.1097/MLR.0000000000001152 PubMedGoogle ScholarCrossref
34.
Walter  LC, Brand  RJ, Counsell  SR,  et al.  Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization.   JAMA. 2001;285(23):2987-2994. doi:10.1001/jama.285.23.2987 PubMedGoogle ScholarCrossref
35.
Lee  SJ, Lindquist  K, Segal  MR, Covinsky  KE.  Development and validation of a prognostic index for 4-year mortality in older adults.   JAMA. 2006;295(7):801-808. doi:10.1001/jama.295.7.801 PubMedGoogle ScholarCrossref
36.
Sun  A, Netzer  G, Small  DS,  et al.  Association between index hospitalization and hospital readmission in sepsis survivors.   Crit Care Med. 2016;44(3):478-487. doi:10.1097/CCM.0000000000001464 PubMedGoogle ScholarCrossref
37.
Jones  TK, Fuchs  BD, Small  DS,  et al.  Post-acute care use and hospital readmission after sepsis.   Ann Am Thorac Soc. 2015;12(6):904-913. doi:10.1513/AnnalsATS.201411-504OC PubMedGoogle ScholarCrossref
38.
Dietz  BW, Jones  TK, Small  DS, Gaieski  DF, Mikkelsen  ME.  The relationship between index hospitalizations, sepsis, and death or transition to hospice care during 30-day hospital readmissions.   Med Care. 2017;55(4):362-370. doi:10.1097/MLR.0000000000000669PubMedGoogle ScholarCrossref
39.
Goodwin  AJ, Rice  DA, Simpson  KN, Ford  DW.  Frequency, cost, and risk factors of readmissions among severe sepsis survivors.   Crit Care Med. 2015;43(4):738-746. doi:10.1097/CCM.0000000000000859 PubMedGoogle ScholarCrossref
40.
Hua  M, Gong  MN, Brady  J, Wunsch  H.  Early and late unplanned rehospitalizations for survivors of critical illness.   Crit Care Med. 2015;43(2):430-438. doi:10.1097/CCM.0000000000000717 PubMedGoogle ScholarCrossref
41.
Weinreich  M, Nguyen  OK, Wang  D,  et al.  Predicting the risk of readmission in pneumonia. a systematic review of model performance.   Ann Am Thorac Soc. 2016;13(9):1607-1614. doi:10.1513/AnnalsATS.201602-135SR PubMedGoogle ScholarCrossref
42.
Pivodic  L, Pardon  K, Morin  L,  et al; EURO IMPACT.  Place of death in the population dying from diseases indicative of palliative care need: a cross-national population-level study in 14 countries.   J Epidemiol Community Health. 2016;70(1):17-24. doi:10.1136/jech-2014-205365 PubMedGoogle ScholarCrossref
43.
Bekelman  JE, Halpern  SD, Blankart  CR,  et al; International Consortium for End-of-Life Research (ICELR).  Comparison of site of death, health care utilization, and hospital expenditures for patients dying with cancer in 7 developed countries.   JAMA. 2016;315(3):272-283. doi:10.1001/jama.2015.18603 PubMedGoogle ScholarCrossref
44.
Shankar-Hari  M, Rubenfeld  GD.  Population enrichment for critical care trials: phenotypes and differential outcomes.   Curr Opin Crit Care. 2019;25(5):489-497. doi:10.1097/MCC.0000000000000641 PubMedGoogle ScholarCrossref
45.
Schmidt  K, Worrack  S, Von Korff  M,  et al; SMOOTH Study Group.  Effect of a primary care management intervention on mental health–related quality of life among survivors of sepsis: a randomized clinical trial.   JAMA. 2016;315(24):2703-2711. doi:10.1001/jama.2016.7207 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 3,023
    Citations 0
    Original Investigation
    Critical Care Medicine
    September 14, 2020

    Development, Validation, and Clinical Utility Assessment of a Prognostic Score for 1-Year Unplanned Rehospitalization or Death of Adult Sepsis Survivors

    Author Affiliations
    • 1Guy’s and St Thomas’ NHS Foundation Trust, ICU Support Offices, St Thomas’ Hospital, London, United Kingdom
    • 2School of Immunology & Microbial Sciences, King’s College London, London, United Kingdom
    • 3Interdepartmental Division of Critical Care Medicine, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
    • 4Associate Editor, JAMA Network Open
    • 5Intensive Care National Audit & Research Centre, Napier House, London, United Kingdom
    JAMA Netw Open. 2020;3(9):e2013580. doi:10.1001/jamanetworkopen.2020.13580
    Key Points

    Question  Could a prognostic score for unplanned rehospitalizations or death in the first year after hospital discharge of adult sepsis survivors be developed using index sepsis illness characteristics as predictors?

    Findings  In this cohort study of 94 748 patients in adult general critical care units in England, unplanned rehospitalization or death in the first year after hospital discharge occurred for 51% of patients in the derivation cohort and 53% of patients in the validation cohort. The prognostic score is calculated using 8 predictors: previous hospitalizations in the preceding year, age, socioeconomic status, preadmission dependence, number of comorbidities, admission type, site of infection, and admission blood hemoglobin level.

    Meaning  This score provides clinically useful information for prognosis discussions and planning follow-up care for sepsis survivors.

    Abstract

    Importance  The longer-term risk of rehospitalizations and death of adult sepsis survivors is associated with index sepsis illness characteristics.

    Objective  To derive and validate a parsimonious prognostic score for unplanned rehospitalizations or death in the first year after hospital discharge of adult sepsis survivors.

    Design, Setting, and Participants  This cohort study used data from the Intensive Care National Audit & Research Centre Case Mix Programme database on adult sepsis survivors identified from consecutive critical care admissions to 192 adult general critical care units in England, United Kingdom, between April 1, 2009, and March 31, 2014 (94 748 patients in the derivation cohort), and between April 1, 2014, and March 31, 2015 (24 669 patients in the validation cohort). Statistical analysis was performed from July 5 to October 31, 2019. Generic characteristics (age, sex, race/ethnicity, 2015 Index of Multiple Deprivation [IMD2015] in England quintiles, preadmission dependence, previous hospitalizations in the year preceding index sepsis admission, comorbidity, admission type, Acute Physiology and Chronic Health Evaluation II physiology score, hospital length of stay, worst blood lactate and blood hemoglobin concentrations, and type of hospital) and sepsis-specific characteristics (site of infection, numbers of organ dysfunctions, and organ support) at the index sepsis admission were used as predictors.

    Main Outcomes and Measures  Prognostic score derived and validated using multivariable logistic regression for the outcome of unplanned rehospitalization or death in the first year after hospital discharge of adult sepsis survivors, as well as clinical usefulness assessed using decision curve analysis. Prognostic score validation was performed for internal validation with bootstrapping and temporal cohort external validation.

    Results  This cohort study included 94 748 patients (51 164 men [54.0%]; mean [SD] age, 61.3 [17.0] years) in the derivation cohort and 24 669 patients (13 255 men [53.7%]; mean [SD] age, 62.1 [16.8%]) in the validation cohort. Unplanned rehospitalization or death in the first year after hospital discharge occurred for 48 594 patients (51.3%) in the derivation cohort and 13 129 patients (53.2%) in the validation cohort. Eight independent predictors were identified and weighted to generate a prognostic score for every patient: previous hospitalizations, age in 10-year increments, IMD2015 in England quintiles, preadmission dependence, comorbidities, admission type, blood hemoglobin level, and site of infection. The total prognostic score ranged from 0 to 22 points, with lower scores indicating a lower risk of the outcome. The derivation and validation cohorts had similar rates of prognostic scores of 0 to 4 points (5088 of 16 684 patients [30.5%] and 471 of 1725 patients [27.3%]) and prognostic scores of 11 points or more (15 732 of 21 641 patients [72.7%] and 5753 of 7952 patients [72.3%]). The area under the receiver operating characteristic curve for the prognostic score was 0.675 (95% CI, 0.672-0.679). The decision curve analysis highlighted an optimal score cutoff of 7 points or more.

    Conclusions and Relevance  The prognostic score reported in this study uses 8 internationally feasible predictors measured during the index sepsis admission and provides clinically useful information on sepsis survivors’ risk of unplanned rehospitalization or death in the first year after hospital discharge.

    Introduction

    Sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to infection.1 Studies, mainly from critical care settings in high-income countries, consistently report an increasing incidence of sepsis, as well as improving acute mortality from sepsis.2-4 This epidemiologic pattern is associated with increasing numbers of patients surviving to hospital discharge after a critical care unit admission for sepsis, who are hereafter referred to as sepsis survivors. Sepsis survivors have ongoing health care needs and are at increased risk of adverse outcomes.5

    Two common and patient-centered adverse outcomes for sepsis survivors are the greater risk of rehospitalization and of long-term mortality compared with survivors of nonsepsis hospitalizations. Systematic reviews have reported a mean 1-year rehospitalization rate of 39.0% (95% CI, 22.0%-59.4%)6 and a mean 1-year mortality rate of 16.1% (95% CI, 14.1%-18.1%) among adult sepsis survivors.7 A systematic review of rehospitalization risk factors (predictors) for sepsis survivors that included 56 studies highlighted generic characteristics (such as older age, male sex, comorbidities, nonelective admissions, and hospitalizations prior to index sepsis admission) and index sepsis admission characteristics (such as infection and illness severity) as consistent predictors across studies.6 The long-term mortality risk for sepsis survivors is also associated with these acute illness characteristics.8 More important, because health care professionals will know these acute illness characteristics before the sepsis survivors are discharged, it is feasible to use these characteristics to predict the risk of these patient-centered adverse outcomes.

    With the use of these acute illness characteristics, there are no prognostic scores to assess the 1-year risk of unplanned rehospitalizations or death after hospital discharge for adult sepsis survivors. Such a score might provide clinicians with objective prognostic information to supplement their clinical judgment when advising patients and families, to help identify a higher-risk sepsis survivor population for targeted follow-up care, and to design trials aiming at improving sepsis survivor outcomes.9-11 Therefore, we derived and validated a parsimonious prognostic score for the outcome of unplanned rehospitalization or death in the first year after hospital discharge of adult sepsis survivors, using multivariable logistic regression.12 We then assessed the clinical usefulness of the prognostic score using decision curve analysis (DCA),13 which informs clinicians when to intervene selectively13 and, more importantly, where data on treatment effectiveness and patient preferences do not exist,14 which is the current scenario for sepsis survivors.

    Methods
    Study Database and Approvals

    Data were extracted from the Intensive Care National Audit & Research Centre Case Mix Programme database,15 which is the national clinical audit covering all adult critical care units in England, United Kingdom, collecting data on consecutive critical care admissions. This anonymized study data set was linked with the Hospital Episode Statistics database and Office for National Statistics death registrations,16 using patient-level unique identifiers.15 Approval for the collection and use of these data has been obtained under Section 251 of the National Health Service Act 2006. Waiver of informed consent for use of this data was obtained from Wales Research Ethics Committee 5 and Confidentiality Advisory Group, which approved this study as part of the Long-term Outcomes After Critical Illness Project. Our study conforms to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline12 for deriving and validating prognostic scores (eFigure 1 in the Supplement).

    Patient Population and Outcome Assessment

    From consecutive critical care admissions to 192 of 200 adult general critical care units (96%) in England, we identified a cohort of sepsis survivors, defined as adult patients who survived to hospital discharge after an index critical care admission for sepsis meeting Sepsis-3 criteria.1 We have previously reported 5-year mortality outcomes in this sepsis survivor cohort.8 Unplanned rehospitalization episodes and rehospitalization diagnoses were ascertained from the linked data set and grouped as per the Healthcare Cost and Utilization Project Clinical Classifications Software multilevel categories.17 This mapping was similar to previous reports in the literature.18,19

    Derivation and Validation Cohorts

    The derivation cohort included all consecutive admissions between April 1, 2009, and March 31, 2014, with the date of rehospitalization or death ascertained as of March 31, 2015.8 The validation cohort included all consecutive admissions between April 1, 2014, and March 31, 2015, with the date of rehospitalization or death ascertained as of March 31, 2016. This validation cohort meets the temporal (narrow external) validation definitions in the TRIPOD statement12 because it uses the same predictors, the same outcome definitions, and the same measurements from the same setting, but sampled from a later period.

    Statistical Analysis
    Identification and Handling of Predictors in All Models

    Statistical analysis was performed from July 5 to October 31, 2019. Potential generic predictors and sepsis-specific predictors8 associated with either all-cause rehospitalization or 1-year mortality among sepsis survivors were identified from systematic reviews6,7 (eTable 1 in the Supplement). The generic predictors were as follows: the Acute Physiology and Chronic Health Evaluation (APACHE) II physiology score20 in 5-point increments, the hospital length of stay in 7-day increments, the number of hospitalizations in the year preceding the index sepsis admission (base category = none), age in 10-year increments (base category = 18-30 years), race/ethnicity (base category = White), quintiles of 2015 Index of Multiple Deprivation (IMD2015) in England as a proxy for socioeconomic status (base category = least deprived), prehospitalization dependence status (base category = no dependence), the number of comorbidities (base category = none), surgical status (base category = medical), admission blood hemoglobin level (≤7, 7.1-11, and 9.1-11 g/dL, with a base category of >11 g/dL [to convert to grams per liter, multiply by 10.0]), and admission blood lactate concentrations (18-36 mg/dL and >36 mg/dL, with a base category of <18 mg/dL [to convert to millimoles per liter, multiply by 0.111]), with sex (base category = female) and hospital type (base category = nonuniversity) as binary predictors. The sepsis-specific predictors were as follows: site of infection (respiratory = base category) and numbers of organ dysfunction1,3 (base category = 1 organ dysfunction), with types of organ support (renal, respiratory, gastrointestinal, neurologic, and cardiovascular) at any point during the critical care unit stay21 as binary predictors.

    The blood hemoglobin level cutoffs were chosen based on the transfusion guidelines for critically ill patients (transfusion threshold, <7 g/dL; target level, 7-9 g/dL)22 and blood lactate level cutoffs were based on the literature.23 The most severe comorbidity was identified for each organ system; individual comorbidities were given equal weights. Further details on predictors are provided in the eMethods in the Supplement.

    Prognostic Model Development

    The prognostic model was derived using multivariable logistic regression with unplanned rehospitalization or death in the first year after hospital discharge as the outcome. The prognostic model derivation occurred in the following steps, after all the described predictors underwent univariable testing for association with the outcome. First, the preliminary model with all the predictors was generated. Second, the post–stepwise reduction model was generated with the stepwise reduction method used to remove the least-significant predictors, until all remaining predictors had a significance of P < .001. Third, because our goal was to derive a parsimonious prognostic score, we sequentially eliminated complex predictors and predictors with the lowest coefficients to generate the final model with the final set of predictors. At each step, the discrimination of the model was assessed by the C statistic (area under the receiver operating characteristic curve [AUROC]) and the overall accuracy was assessed by the Brier score. Multicollinearity was assessed with variance inflation factors24 for both the preliminary model with all predictors and the final model.

    Prognostic Score Development

    We assigned the largest coefficient in the final model 5 points and rescaled all other coefficients relative to this point value, rounding to the nearest integer. We then calculated a score for each patient by totaling the points based on individual predictors and assessed the distribution. The derivation cohort was then stratified on outcome risk based on the individual scores. The predictive accuracy of the score was then assessed by comparing predicted vs observed outcome (calibration) and discriminant validity with AUROC.25

    Assessment of Clinical Utility of Prognostic Score

    Sepsis survivors and clinicians decide together on follow-up care, based on patients’ risk and benefit preferences. Clinicians can either provide follow-up care to all sepsis survivors (treat all) or provide no follow-up care (treat none) or use the prognostic information assessed with DCA to intervene selectively. Decision curve analysis is underpinned by 2 concepts: threshold probability and net benefit. Threshold probability of outcome refers to the empirical value above which a clinician would choose to treat or a sepsis survivor would choose to receive treatment. In other words, the sepsis survivors make a conscious decision that the benefit of their empirical treatment outweighs the harms associated with the treatment. The net benefit is the difference between the expected benefit and the expected harm. The expected benefit here is that sepsis survivors who will experience 1 or more rehospitalizations or death within 1 year after hospital discharge will choose to receive treatment (true positive). The expected harm is that sepsis survivors who will not experience this outcome will choose to receive treatment (false positive) multiplied by those sepsis survivors’ threshold probabilities. Decision curve analysis uses this theoretical association of net benefit = true positive rate − [false positive rate × [threshold probability/(1 − threshold probability)] to generate decision curves.14 The highest decision curve over the full range of threshold probabilities is the best cutoff for a prognostic score, regardless of sepsis survivors’ preferences. However, if the decision curves cross, then the optimal cutoff will be associated with sepsis survivors’ preferences of threshold probability.

    Prognostic Score Validation

    We performed internal validation with bootstrapping12 and temporal cohort external validation.12 Bootstrap samples (n = 200) were taken from the original data, stepwise selection repeated within each bootstrap sample, with Efron optimism estimated using the difference between the bootstrap data and the original data.26 Bootstrapping provides the nonparametric maximum likelihood of error and stable estimates with low risk of bias,27 assuming that the derivation cohort represents a random sample of sepsis survivors.

    Reported P values are 2-sided, and P < .05 was considered to represent a statistically significant result. Continuous data were summarized as mean (SD) values, when normally distributed, and as median values and interquartile ranges, when not normally distributed. Categorical data were presented as frequency and percentage. We achieved 96% outcome data linkage.8 Because missing data were less than 0.5% for all the predictors and because the Case Mix Programme had between 90% and 100% coverage of adult general critical care units in England during the study period, we performed full case analyses, assuming that data are missing completely at random.28 We did not perform an a priori sample size calculation. All analyses were performed using Stata/SE, version 14.0 (StataCorp LLC).

    Results
    Derivation and Validation Cohorts

    There were 94 748 sepsis survivors in the derivation cohort and 24 669 sepsis survivors in the validation cohort. The study outcome of unplanned rehospitalization or death in the first year after hospital discharge occurred for 48 594 patients (51.3%) in the derivation cohort and 13 129 patients (53.2%) in the validation cohort (Figure 1). The generic and sepsis-specific characteristics were similar between the derivation and validation cohorts (Table 1; eTable 2 in the Supplement).

    Prognostic Model Development

    On univariable testing, age, sex, race/ethnicity, IMD2015 in England quintiles, preadmission dependence, all comorbidities, any hospitalization in the preceding year, admission type, illness severity, hospital length of stay, lower blood hemoglobin level at index admission, site of infection, numbers of organ dysfunction, and organ support were significantly associated with the study outcome of unplanned rehospitalization or death in the first year after hospital discharge (eTable 4 in the Supplement).

    The preliminary model with all predictors is shown in eTable 5 in the Supplement. The stepwise reduction procedure to generate the poststepwise reduction model removed the following predictors: sex, blood lactate level, numbers of organ dysfunction, and organ support (respiratory, renal, gastrointestinal, and neurologic). We then sequentially eliminated complex predictors (APACHE II acute physiology score) and predictors with the lowest coefficients (race/ethnicity, advanced cardiovascular support, hospital length of stay, and hospital type). This did not alter the discrimination and calibration of the model (eTable 5 in the Supplement). All variance inflation factors were considerably less than 10 (with a mean value of 2.2 for the preliminary model and of 2.4 for the final model). The predictors in the final model were previous hospitalization, age, IMD2015 in England quintiles, preadmission dependence, number of comorbidities, admission type, blood hemoglobin level at admission, and site of infection (Table 2).

    Prognostic Score Development

    The prognostic score ranged from 0 to 22 points, with lower scores indicating a lower risk of the outcome. (Table 2). We generated 4 risk strata by grouping patients based on their individual prognostic score (Figure 2A; Table 2). The outcome of unplanned rehospitalization or death in the first year after hospital discharge occurred for 5088 of 16 684 patients (30.5%) with scores between 0 and 4 points and 15 732 of 21 641 patients (72.7%) with 11 points or more (Figure 2B; Table 2). The derivation and validation cohorts had similar distributions of prognostic scores of 0 to 4 points (5088 of 16 684 patients [30.5%] and 471 of 1725 patients [27.3%]) and prognostic scores of 11 points or more (15 732 of 21 641 patients [72.7%] and 5753 of 7952 patients [72.3%]) (Table 2). The discrimination was moderate (AUROC, 0.675; 95% CI, 0.672-0.679) (Figure 2C-D).

    Assessment of Clinical Utility of the Prognostic Score

    Net benefit was maximized with threshold probabilities of 0% to 30% by the treat-all approach. The DCA highlights that the net benefit of this score is positive and higher than the net benefit for the treat-all scenario at threshold probabilities of 30% or more (Figure 3A). In the intervening selective scenario of our analysis, a score of 5 or more has the highest curve for a threshold probability of less than 45%. At a threshold probability of 45% or more and a score of 7 or more, the true positives were 34.7% (32 868 of 94 748 patients), the false positives were 16.6% (15 725 of 94 748 patients), and the estimated net benefit was 0.21; at this cutoff, 55.3% (52 433 of 94 748 patients) of the study cohort may choose follow-up care (eTable 7 in the Supplement and Figure 3A). The individual outcome events mirrored changes in the composite outcome with our prognostic score (Figure 3B and eTable 6 in the Supplement).

    Prognostic Score Validation

    The description and performance of our prognostic score in our external validation cohort are shown in eTable 8 and eFigure 2 in the Supplement. The use of predictors during the 200-sample bootstrap validation models is shown in eFigure 3 in the Supplement. With bootstrap internal validation, the mean optimism estimate was 0.0007 (95% CI, 0.0005-0.0010) for the C statistic, −0.0002 (95% CI, −0.0003 to −0.0001) for the Brier score, and 0.0005 (95% CI, 0.0003-0.0006) for the Shapiro R test score.

    Discussion

    We derived and validated a parsimonious prognostic score for the outcome of unplanned rehospitalization or death in the first year after hospital discharge, using acute illness characteristics from the index sepsis admission as predictors. The score is calculated using 8 predictors: hospitalizations in the preceding year, age, socioeconomic status using IMD2015 in England quintiles, preadmission dependence, number of comorbidities, admission type, site of infection, and admission blood hemoglobin level, a globally feasible blood test.

    What is the clinical value and use of this prognostic score? We and others have highlighted that 1 in 2 sepsis survivors have unplanned rehospitalizations or die in the first year after hospital discharge.19,29-31 Currently, the follow-up care of sepsis survivors is neither risk based nor uniform within and between health care systems. Seldom does every sepsis survivor receive follow-up care. In this context, we report, to our knowledge, the first prognostic score that helps to identify sepsis survivors’ risk of these 2 patient-centered adverse outcomes. Although predictive accuracy at the individual level is at best modest, our score highlights a clinically useful gradient of risk to target follow-up care. We envision that, in the first instance, the score would be used to inform prognosis discussions with sepsis survivors and follow-up care designed based on patient preferences. More important, because DCA does not mandate information on the costs or effectiveness of treatment or how patients value different health states,14 our analysis highlights that it is possible to intervene selectively for sepsis survivors, using a systematic risk-based approach. An example of the follow-up care that we envision would be ensuring better postdischarge adherence to recommended management for common comorbidities, as better adherence could improve sepsis survivor outcomes32,33 because effective outpatient management exists for these common rehospitalization diagnoses.18 We expect our score to be refined over time based on preliminary use because clinicians and researchers may identify what additional predictors could add value. We expect an increased emphasis on sepsis survivors’ health with unrestricted availability of our score. A similar approach with prognostic scoring for addressing care for older adults after hospitalizations has been reported.34,35

    The predictors in the final list are consistently reported in the literature as risk factors for rehospitalization6 and/or 1-year mortality7—hospitalization in the preceding year,36,37 increasing age,19,38,39 different proxies for socioeconomic status (such as insurance status,11,38-40 urban residence, and lower income11), preadmission dependence, higher comorbidity burden,10,11,19,38-40 admission type, blood hemoglobin level at admission, and site of infection.10 Increased risk of outcome with a lower blood hemoglobin level at index sepsis admission in the score is analogous to the association observed with a lower blood hemoglobin level at critical care discharge.36-38 We contend that the admission blood hemoglobin level may also represent chronic health status.

    Strengths and Limitations

    Our study has some strengths. We present a novel score to stratify sepsis survivors’ risk of rehospitalization or death in the first year after hospital discharge, whereas other tools focus on 30-day rehospitalization risk.41 Our primary outcome of all-cause rehospitalizations or death has the lowest risk of ascertainment bias. The score, albeit feasible to measure even in resource-limited settings, requires recalibration and potentially revision before use as standard of care in other health care settings. As part of development work for this analysis, we published the operational definition for sepsis from this database,3 long-term mortality in the derivation cohort,8 and systematic reviews to identify key predictors.6,7 Our method of handling comorbidities functions even when patients have 2 or more comorbidities, is transferable between health care systems that use different comorbidity definitions, and the incremental risk with increasing numbers of comorbidities has clinical validity. We followed the recommendations by Steyerberg and colleagues27 for bootstrap validations.

    Our study also has some limitations. Our study population was limited to patients who survived to hospital discharge after a critical care unit admission for sepsis. Our model predicts the 1-year risk, not necessarily at specific time points during follow-up. We did not have information on patient preferences after discharge, so 1-year mortality in some cases may reflect a lack of seeking care or limitations of care, such as end-of-life care in community settings.42,43 However, this is unlikely to be a major factor because death without rehospitalization was observed for only 4.3% of the derivation cohort (4035 of 94 748) and 4.5% of the validation cohort (1118 of 24 669). Although we used a national deprivation index (IMD2015 in England) as a proxy for socioeconomic strata, there are other national or regional indces. Because additional risk is only for individuals in the lowest socioeconomic status, this is an easy limitation to overcome when using other indices. A similar consideration applies to preadmission dependence assessments. Perhaps the biggest limitation of our score is that currently there are no proven interventions to prevent posthospitalization death or readmissions of sepsis survivors.

    Our study highlights several future research questions. Although we report internal and temporal external validations, generalizability of the score should be assessed in other settings, including resource-limited settings. There is insufficient knowledge regarding sepsis survivors’ preferences on the relevant range of threshold probabilities within DCA, which needs to be determined. Because there are no proven interventions to weigh the risk vs benefit within DCA, the utility of the score as a prognostic enrichment44 tool could be evaluated by ongoing trials (eg, NCT03565159) and data from completed trials.45 The score could be assessed as a benchmarking tool between hospitals to reduce variation in sepsis survivor outcomes.

    Conclusions

    In this study, unplanned rehospitalization or death in the first year after hospital discharge were common adverse outcomes among sepsis survivors. This study validated a simple and internationally feasible prognostic score for this composite outcome, which could inform prognosis discussions, trial design, and follow-up care of sepsis survivors.

    Back to top
    Article Information

    Accepted for Publication: June 2, 2020.

    Published: September 14, 2020. doi:10.1001/jamanetworkopen.2020.13580

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Shankar-Hari M et al. JAMA Network Open.

    Corresponding Author: Manu Shankar-Hari, MSc, MD, PhD, First Floor, East Wing, St Thomas’ Hospital, Guy’s and St Thomas’ NHS Foundation Trust London, London SE17EH, UK (manu.shankar-hari@kcl.ac.uk).

    Author Contributions: Drs Shankar-Hari and Harrison had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Harrison and Rowan contributed equally to this work.

    Concept and design: Shankar-Hari, Rubenfeld, Harrison, Rowan.

    Acquisition, analysis, or interpretation of data: Shankar-Hari, Ferrando-Vivas, Harrison, Rowan.

    Drafting of the manuscript: Shankar-Hari, Rowan.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Shankar-Hari, Rubenfeld, Harrison.

    Obtained funding: Shankar-Hari.

    Administrative, technical, or material support: Rowan.

    Supervision: Harrison, Rowan.

    Conflict of Interest Disclosures: Dr Rubenfeld reported serving as a consultant for Endpoint Health. Dr Harrison reported receiving grants from the National Institute for Health Research during the conduct of the study. No other disclosures were reported.

    Funding/Support: Dr Shankar-Hari is funded by a National Institute for Health Research Clinician Scientist Award (CS-2016-16-011) for this research project. This publication presents independent research funded by the National Institute for Health Research.

    Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The views expressed in this publication are those of the authors and not necessarily those of the National Health Service, the National Institute for Health Research, or the Department of Health and Social Care. Dr Rubenfeld is an associate editor of JAMA Network Open, but he was not involved in any of the decisions regarding review of the manuscript or its acceptance.

    Additional Contributions: As part of the preparation for this research, in the write up of theNational Institute for Health Research Clinician Scientist Award application, Dr Manu Shankar-Hari (CS-2016-16-011) consulted survivors of sepsis to explore the longer-term health care requirements, which informed the analysis plan for this study.

    Additional Information: The data used in this study are accessible from the UK National Intensive Care Admissions dataset managed by the Intensive Care National Audit & Research Centre data set. A free-to-use online tool is available at https://sepsis-ssip-score.org.uk/ssip_score.

    References
    1.
    Singer  M, Deutschman  CS, Seymour  CW,  et al.  The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).   JAMA. 2016;315(8):801-810. doi:10.1001/jama.2016.0287 PubMedGoogle ScholarCrossref
    2.
    Shankar-Hari  M, Harrison  DA, Rowan  KM.  Differences in impact of definitional elements on mortality precludes international comparisons of sepsis epidemiology—a cohort study illustrating the need for standardized reporting.   Crit Care Med. 2016;44(12):2223-2230. doi:10.1097/CCM.0000000000001876 PubMedGoogle ScholarCrossref
    3.
    Shankar-Hari  M, Harrison  DA, Rubenfeld  GD, Rowan  K.  Epidemiology of sepsis and septic shock in critical care units: comparison between Sepsis-2 and Sepsis-3 populations using a national critical care database.   Br J Anaesth. 2017;119(4):626-636. doi:10.1093/bja/aex234 PubMedGoogle ScholarCrossref
    4.
    Kaukonen  KM, Bailey  M, Suzuki  S, Pilcher  D, Bellomo  R.  Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012.   JAMA. 2014;311(13):1308-1316. doi:10.1001/jama.2014.2637 PubMedGoogle ScholarCrossref
    5.
    Prescott  HC, Angus  DC.  Enhancing recovery from sepsis: a review.   JAMA. 2018;319(1):62-75. doi:10.1001/jama.2017.17687 PubMedGoogle ScholarCrossref
    6.
    Shankar-Hari  M, Saha  R, Wilson  J,  et al.  Rate and risk factors for rehospitalisation in sepsis survivors: systematic review and meta-analysis.   Intensive Care Med. 2020;46(4):619-636. doi:10.1007/s00134-019-05908-3 PubMedGoogle ScholarCrossref
    7.
    Shankar-Hari  M, Ambler  M, Mahalingasivam  V, Jones  A, Rowan  K, Rubenfeld  GD.  Evidence for a causal link between sepsis and long-term mortality: a systematic review of epidemiologic studies.   Crit Care. 2016;20(1):101. doi:10.1186/s13054-016-1276-7 PubMedGoogle ScholarCrossref
    8.
    Shankar-Hari  M, Harrison  DA, Ferrando-Vivas  P, Rubenfeld  GD, Rowan  K.  Risk factors at index hospitalization associated with longer-term mortality in adult sepsis survivors.   JAMA Netw Open. 2019;2(5):e194900. doi:10.1001/jamanetworkopen.2019.4900 PubMedGoogle Scholar
    9.
    Norman  BC, Cooke  CR, Ely  EW, Graves  JA.  Sepsis-associated 30-day risk-standardized readmissions: analysis of a nationwide Medicare sample.   Crit Care Med. 2017;45(7):1130-1137. doi:10.1097/CCM.0000000000002476 PubMedGoogle ScholarCrossref
    10.
    Donnelly  JP, Hohmann  SF, Wang  HE.  Unplanned readmissions after hospitalization for severe sepsis at academic medical center-affiliated hospitals.   Crit Care Med. 2015;43(9):1916-1927. doi:10.1097/CCM.0000000000001147 PubMedGoogle ScholarCrossref
    11.
    Chang  DW, Tseng  CH, Shapiro  MF.  Rehospitalizations following sepsis: common and costly.   Crit Care Med. 2015;43(10):2085-2093. doi:10.1097/CCM.0000000000001159 PubMedGoogle ScholarCrossref
    12.
    Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement.   Ann Intern Med. 2015;162(1):55-63. doi:10.7326/M14-0697 PubMedGoogle ScholarCrossref
    13.
    Fitzgerald  M, Saville  BR, Lewis  RJ.  Decision curve analysis.   JAMA. 2015;313(4):409-410. doi:10.1001/jama.2015.37 PubMedGoogle ScholarCrossref
    14.
    Vickers  AJ, Elkin  EB.  Decision curve analysis: a novel method for evaluating prediction models.   Med Decis Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361 PubMedGoogle ScholarCrossref
    15.
    Young  JD, Goldfrad  C, Rowan  K.  Development and testing of a hierarchical method to code the reason for admission to intensive care units: the ICNARC Coding Method: Intensive Care National Audit & Research Centre.   Br J Anaesth. 2001;87(4):543-548. doi:10.1093/bja/87.4.543 PubMedGoogle ScholarCrossref
    17.
    Healthcare Cost and Utilization Project. Clinical Classifications Software (CCS) for ICD-9-CM. Accessed October 31, 2019. https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp
    18.
    Prescott  HC, Langa  KM, Iwashyna  TJ.  Readmission diagnoses after hospitalization for severe sepsis and other acute medical conditions.   JAMA. 2015;313(10):1055-1057. doi:10.1001/jama.2015.1410 PubMedGoogle ScholarCrossref
    19.
    Liu  V, Lei  X, Prescott  HC, Kipnis  P, Iwashyna  TJ, Escobar  GJ.  Hospital readmission and healthcare utilization following sepsis in community settings.   J Hosp Med. 2014;9(8):502-507. doi:10.1002/jhm.2197 PubMedGoogle ScholarCrossref
    20.
    Quach  S, Hennessy  DA, Faris  P, Fong  A, Quan  H, Doig  C.  A comparison between the APACHE II and Charlson Index Score for predicting hospital mortality in critically ill patients.   BMC Health Serv Res. 2009;9:129. doi:10.1186/1472-6963-9-129 PubMedGoogle ScholarCrossref
    21.
    Information Standards Board for Health and Social Care. Critical care minimum dataset full specification critical care minimum dataset version 8.0. Published March 2010. Accessed July 5, 2019. https://webarchive.nationalarchives.gov.uk/+/http://www.isb.nhs.uk/documents/isb-0153/amd-81-2010/0153812010spec.pdf
    22.
    Retter  A, Wyncoll  D, Pearse  R,  et al; British Committee for Standards in Haematology.  Guidelines on the management of anaemia and red cell transfusion in adult critically ill patients.   Br J Haematol. 2013;160(4):445-464. doi:10.1111/bjh.12143 PubMedGoogle ScholarCrossref
    23.
    Shankar-Hari  M, Phillips  GS, Levy  ML,  et al; Sepsis Definitions Task Force.  Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).   JAMA. 2016;315(8):775-787. doi:10.1001/jama.2016.0289 PubMedGoogle ScholarCrossref
    24.
    Thompson  CG, Kim  RS, Aloe  AM, Becker  BJ.  Extracting the variance inflation factor and other multicollinearity diagnostics from typical regression results.   Basic Appl Soc Psych. 2017;39(2):81-90. doi:10.1080/01973533.2016.1277529 Google ScholarCrossref
    25.
    Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.   Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2 PubMedGoogle ScholarCrossref
    26.
    Efron  B.  Estimating the error rate of a prediction rule—improvement on cross-validation.   J Am Stat Assoc. 1983;78(382):316-331. doi:10.1080/01621459.1983.10477973 Google ScholarCrossref
    27.
    Steyerberg  EW, Harrell  FE  Jr, Borsboom  GJ, Eijkemans  MJ, Vergouwe  Y, Habbema  JD.  Internal validation of predictive models: efficiency of some procedures for logistic regression analysis.   J Clin Epidemiol. 2001;54(8):774-781. doi:10.1016/S0895-4356(01)00341-9 PubMedGoogle ScholarCrossref
    28.
    Seaman  S, Galati  J, Jackson  D, Carlin  J.  What is meant by “missing at random”?   Stat Sci. 2013;28(2):257-268. doi:10.1214/13-STS415 Google ScholarCrossref
    29.
    Braun  L, Riedel  AA, Cooper  LM.  Severe sepsis in managed care: analysis of incidence, one-year mortality, and associated costs of care.   J Manag Care Pharm. 2004;10(6):521-530. doi:10.18553/jmcp.2004.10.6.521 PubMedGoogle Scholar
    30.
    Prescott  HC, Langa  KM, Liu  V, Escobar  GJ, Iwashyna  TJ.  Increased 1-year healthcare use in survivors of severe sepsis.   Am J Respir Crit Care Med. 2014;190(1):62-69. doi:10.1164/rccm.201403-0471OC PubMedGoogle ScholarCrossref
    31.
    Yende  S, Kellum  JA, Talisa  VB,  et al.  Long-term host immune response trajectories among hospitalized patients with sepsis.   JAMA Netw Open. 2019;2(8):e198686. doi:10.1001/jamanetworkopen.2019.8686 PubMedGoogle Scholar
    32.
    Taylor  SP, Chou  SH, Sierra  MF,  et al.  Association between adherence to recommended care and outcomes for adult survivors of sepsis.   Ann Am Thorac Soc. 2020;17(1):89-97. doi:10.1513/AnnalsATS.201907-514OC PubMedGoogle ScholarCrossref
    33.
    Deb  P, Murtaugh  CM, Bowles  KH,  et al.  Does early follow-up improve the outcomes of sepsis survivors discharged to home health care?   Med Care. 2019;57(8):633-640. doi:10.1097/MLR.0000000000001152 PubMedGoogle ScholarCrossref
    34.
    Walter  LC, Brand  RJ, Counsell  SR,  et al.  Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization.   JAMA. 2001;285(23):2987-2994. doi:10.1001/jama.285.23.2987 PubMedGoogle ScholarCrossref
    35.
    Lee  SJ, Lindquist  K, Segal  MR, Covinsky  KE.  Development and validation of a prognostic index for 4-year mortality in older adults.   JAMA. 2006;295(7):801-808. doi:10.1001/jama.295.7.801 PubMedGoogle ScholarCrossref
    36.
    Sun  A, Netzer  G, Small  DS,  et al.  Association between index hospitalization and hospital readmission in sepsis survivors.   Crit Care Med. 2016;44(3):478-487. doi:10.1097/CCM.0000000000001464 PubMedGoogle ScholarCrossref
    37.
    Jones  TK, Fuchs  BD, Small  DS,  et al.  Post-acute care use and hospital readmission after sepsis.   Ann Am Thorac Soc. 2015;12(6):904-913. doi:10.1513/AnnalsATS.201411-504OC PubMedGoogle ScholarCrossref
    38.
    Dietz  BW, Jones  TK, Small  DS, Gaieski  DF, Mikkelsen  ME.  The relationship between index hospitalizations, sepsis, and death or transition to hospice care during 30-day hospital readmissions.   Med Care. 2017;55(4):362-370. doi:10.1097/MLR.0000000000000669PubMedGoogle ScholarCrossref
    39.
    Goodwin  AJ, Rice  DA, Simpson  KN, Ford  DW.  Frequency, cost, and risk factors of readmissions among severe sepsis survivors.   Crit Care Med. 2015;43(4):738-746. doi:10.1097/CCM.0000000000000859 PubMedGoogle ScholarCrossref
    40.
    Hua  M, Gong  MN, Brady  J, Wunsch  H.  Early and late unplanned rehospitalizations for survivors of critical illness.   Crit Care Med. 2015;43(2):430-438. doi:10.1097/CCM.0000000000000717 PubMedGoogle ScholarCrossref
    41.
    Weinreich  M, Nguyen  OK, Wang  D,  et al.  Predicting the risk of readmission in pneumonia. a systematic review of model performance.   Ann Am Thorac Soc. 2016;13(9):1607-1614. doi:10.1513/AnnalsATS.201602-135SR PubMedGoogle ScholarCrossref
    42.
    Pivodic  L, Pardon  K, Morin  L,  et al; EURO IMPACT.  Place of death in the population dying from diseases indicative of palliative care need: a cross-national population-level study in 14 countries.   J Epidemiol Community Health. 2016;70(1):17-24. doi:10.1136/jech-2014-205365 PubMedGoogle ScholarCrossref
    43.
    Bekelman  JE, Halpern  SD, Blankart  CR,  et al; International Consortium for End-of-Life Research (ICELR).  Comparison of site of death, health care utilization, and hospital expenditures for patients dying with cancer in 7 developed countries.   JAMA. 2016;315(3):272-283. doi:10.1001/jama.2015.18603 PubMedGoogle ScholarCrossref
    44.
    Shankar-Hari  M, Rubenfeld  GD.  Population enrichment for critical care trials: phenotypes and differential outcomes.   Curr Opin Crit Care. 2019;25(5):489-497. doi:10.1097/MCC.0000000000000641 PubMedGoogle ScholarCrossref
    45.
    Schmidt  K, Worrack  S, Von Korff  M,  et al; SMOOTH Study Group.  Effect of a primary care management intervention on mental health–related quality of life among survivors of sepsis: a randomized clinical trial.   JAMA. 2016;315(24):2703-2711. doi:10.1001/jama.2016.7207 PubMedGoogle ScholarCrossref
    ×