Development, Validation, and Clinical Utility Assessment of a Prognostic Score for 1-Year Unplanned Rehospitalization or Death of Adult Sepsis Survivors

Key Points

Question  Could a prognostic score for unplanned rehospitalizations or death in the first year after hospital discharge of adult sepsis survivors be developed using index sepsis illness characteristics as predictors?

Findings  In this cohort study of 94 748 patients in adult general critical care units in England, unplanned rehospitalization or death in the first year after hospital discharge occurred for 51% of patients in the derivation cohort and 53% of patients in the validation cohort. The prognostic score is calculated using 8 predictors: previous hospitalizations in the preceding year, age, socioeconomic status, preadmission dependence, number of comorbidities, admission type, site of infection, and admission blood hemoglobin level.

Meaning  This score provides clinically useful information for prognosis discussions and planning follow-up care for sepsis survivors.

Importance  The longer-term risk of rehospitalizations and death of adult sepsis survivors is associated with index sepsis illness characteristics.

Objective  To derive and validate a parsimonious prognostic score for unplanned rehospitalizations or death in the first year after hospital discharge of adult sepsis survivors.

Design, Setting, and Participants  This cohort study used data from the Intensive Care National Audit & Research Centre Case Mix Programme database on adult sepsis survivors identified from consecutive critical care admissions to 192 adult general critical care units in England, United Kingdom, between April 1, 2009, and March 31, 2014 (94 748 patients in the derivation cohort), and between April 1, 2014, and March 31, 2015 (24 669 patients in the validation cohort). Statistical analysis was performed from July 5 to October 31, 2019. Generic characteristics (age, sex, race/ethnicity, 2015 Index of Multiple Deprivation [IMD2015] in England quintiles, preadmission dependence, previous hospitalizations in the year preceding index sepsis admission, comorbidity, admission type, Acute Physiology and Chronic Health Evaluation II physiology score, hospital length of stay, worst blood lactate and blood hemoglobin concentrations, and type of hospital) and sepsis-specific characteristics (site of infection, numbers of organ dysfunctions, and organ support) at the index sepsis admission were used as predictors.

Main Outcomes and Measures  Prognostic score derived and validated using multivariable logistic regression for the outcome of unplanned rehospitalization or death in the first year after hospital discharge of adult sepsis survivors, as well as clinical usefulness assessed using decision curve analysis. Prognostic score validation was performed for internal validation with bootstrapping and temporal cohort external validation.

Results  This cohort study included 94 748 patients (51 164 men [54.0%]; mean [SD] age, 61.3 [17.0] years) in the derivation cohort and 24 669 patients (13 255 men [53.7%]; mean [SD] age, 62.1 [16.8%]) in the validation cohort. Unplanned rehospitalization or death in the first year after hospital discharge occurred for 48 594 patients (51.3%) in the derivation cohort and 13 129 patients (53.2%) in the validation cohort. Eight independent predictors were identified and weighted to generate a prognostic score for every patient: previous hospitalizations, age in 10-year increments, IMD2015 in England quintiles, preadmission dependence, comorbidities, admission type, blood hemoglobin level, and site of infection. The total prognostic score ranged from 0 to 22 points, with lower scores indicating a lower risk of the outcome. The derivation and validation cohorts had similar rates of prognostic scores of 0 to 4 points (5088 of 16 684 patients [30.5%] and 471 of 1725 patients [27.3%]) and prognostic scores of 11 points or more (15 732 of 21 641 patients [72.7%] and 5753 of 7952 patients [72.3%]). The area under the receiver operating characteristic curve for the prognostic score was 0.675 (95% CI, 0.672-0.679). The decision curve analysis highlighted an optimal score cutoff of 7 points or more.

Conclusions and Relevance  The prognostic score reported in this study uses 8 internationally feasible predictors measured during the index sepsis admission and provides clinically useful information on sepsis survivors’ risk of unplanned rehospitalization or death in the first year after hospital discharge.

Sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to infection.¹ Studies, mainly from critical care settings in high-income countries, consistently report an increasing incidence of sepsis, as well as improving acute mortality from sepsis.^2-4 This epidemiologic pattern is associated with increasing numbers of patients surviving to hospital discharge after a critical care unit admission for sepsis, who are hereafter referred to as sepsis survivors. Sepsis survivors have ongoing health care needs and are at increased risk of adverse outcomes.⁵

Two common and patient-centered adverse outcomes for sepsis survivors are the greater risk of rehospitalization and of long-term mortality compared with survivors of nonsepsis hospitalizations. Systematic reviews have reported a mean 1-year rehospitalization rate of 39.0% (95% CI, 22.0%-59.4%)⁶ and a mean 1-year mortality rate of 16.1% (95% CI, 14.1%-18.1%) among adult sepsis survivors.⁷ A systematic review of rehospitalization risk factors (predictors) for sepsis survivors that included 56 studies highlighted generic characteristics (such as older age, male sex, comorbidities, nonelective admissions, and hospitalizations prior to index sepsis admission) and index sepsis admission characteristics (such as infection and illness severity) as consistent predictors across studies.⁶ The long-term mortality risk for sepsis survivors is also associated with these acute illness characteristics.⁸ More important, because health care professionals will know these acute illness characteristics before the sepsis survivors are discharged, it is feasible to use these characteristics to predict the risk of these patient-centered adverse outcomes.

With the use of these acute illness characteristics, there are no prognostic scores to assess the 1-year risk of unplanned rehospitalizations or death after hospital discharge for adult sepsis survivors. Such a score might provide clinicians with objective prognostic information to supplement their clinical judgment when advising patients and families, to help identify a higher-risk sepsis survivor population for targeted follow-up care, and to design trials aiming at improving sepsis survivor outcomes.^9-11 Therefore, we derived and validated a parsimonious prognostic score for the outcome of unplanned rehospitalization or death in the first year after hospital discharge of adult sepsis survivors, using multivariable logistic regression.¹² We then assessed the clinical usefulness of the prognostic score using decision curve analysis (DCA),¹³ which informs clinicians when to intervene selectively¹³ and, more importantly, where data on treatment effectiveness and patient preferences do not exist,¹⁴ which is the current scenario for sepsis survivors.

Data were extracted from the Intensive Care National Audit & Research Centre Case Mix Programme database,¹⁵ which is the national clinical audit covering all adult critical care units in England, United Kingdom, collecting data on consecutive critical care admissions. This anonymized study data set was linked with the Hospital Episode Statistics database and Office for National Statistics death registrations,¹⁶ using patient-level unique identifiers.¹⁵ Approval for the collection and use of these data has been obtained under Section 251 of the National Health Service Act 2006. Waiver of informed consent for use of this data was obtained from Wales Research Ethics Committee 5 and Confidentiality Advisory Group, which approved this study as part of the Long-term Outcomes After Critical Illness Project. Our study conforms to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline¹² for deriving and validating prognostic scores (eFigure 1 in the Supplement).

From consecutive critical care admissions to 192 of 200 adult general critical care units (96%) in England, we identified a cohort of sepsis survivors, defined as adult patients who survived to hospital discharge after an index critical care admission for sepsis meeting Sepsis-3 criteria.¹ We have previously reported 5-year mortality outcomes in this sepsis survivor cohort.⁸ Unplanned rehospitalization episodes and rehospitalization diagnoses were ascertained from the linked data set and grouped as per the Healthcare Cost and Utilization Project Clinical Classifications Software multilevel categories.¹⁷ This mapping was similar to previous reports in the literature.^18,19

The derivation cohort included all consecutive admissions between April 1, 2009, and March 31, 2014, with the date of rehospitalization or death ascertained as of March 31, 2015.⁸ The validation cohort included all consecutive admissions between April 1, 2014, and March 31, 2015, with the date of rehospitalization or death ascertained as of March 31, 2016. This validation cohort meets the temporal (narrow external) validation definitions in the TRIPOD statement¹² because it uses the same predictors, the same outcome definitions, and the same measurements from the same setting, but sampled from a later period.

Statistical analysis was performed from July 5 to October 31, 2019. Potential generic predictors and sepsis-specific predictors⁸ associated with either all-cause rehospitalization or 1-year mortality among sepsis survivors were identified from systematic reviews^6,7 (eTable 1 in the Supplement). The generic predictors were as follows: the Acute Physiology and Chronic Health Evaluation (APACHE) II physiology score²⁰ in 5-point increments, the hospital length of stay in 7-day increments, the number of hospitalizations in the year preceding the index sepsis admission (base category = none), age in 10-year increments (base category = 18-30 years), race/ethnicity (base category = White), quintiles of 2015 Index of Multiple Deprivation (IMD2015) in England as a proxy for socioeconomic status (base category = least deprived), prehospitalization dependence status (base category = no dependence), the number of comorbidities (base category = none), surgical status (base category = medical), admission blood hemoglobin level (≤7, 7.1-11, and 9.1-11 g/dL, with a base category of >11 g/dL [to convert to grams per liter, multiply by 10.0]), and admission blood lactate concentrations (18-36 mg/dL and >36 mg/dL, with a base category of <18 mg/dL [to convert to millimoles per liter, multiply by 0.111]), with sex (base category = female) and hospital type (base category = nonuniversity) as binary predictors. The sepsis-specific predictors were as follows: site of infection (respiratory = base category) and numbers of organ dysfunction^1,3 (base category = 1 organ dysfunction), with types of organ support (renal, respiratory, gastrointestinal, neurologic, and cardiovascular) at any point during the critical care unit stay²¹ as binary predictors.

The blood hemoglobin level cutoffs were chosen based on the transfusion guidelines for critically ill patients (transfusion threshold, <7 g/dL; target level, 7-9 g/dL)²² and blood lactate level cutoffs were based on the literature.²³ The most severe comorbidity was identified for each organ system; individual comorbidities were given equal weights. Further details on predictors are provided in the eMethods in the Supplement.

The prognostic model was derived using multivariable logistic regression with unplanned rehospitalization or death in the first year after hospital discharge as the outcome. The prognostic model derivation occurred in the following steps, after all the described predictors underwent univariable testing for association with the outcome. First, the preliminary model with all the predictors was generated. Second, the post–stepwise reduction model was generated with the stepwise reduction method used to remove the least-significant predictors, until all remaining predictors had a significance of P < .001. Third, because our goal was to derive a parsimonious prognostic score, we sequentially eliminated complex predictors and predictors with the lowest coefficients to generate the final model with the final set of predictors. At each step, the discrimination of the model was assessed by the C statistic (area under the receiver operating characteristic curve [AUROC]) and the overall accuracy was assessed by the Brier score. Multicollinearity was assessed with variance inflation factors²⁴ for both the preliminary model with all predictors and the final model.

We assigned the largest coefficient in the final model 5 points and rescaled all other coefficients relative to this point value, rounding to the nearest integer. We then calculated a score for each patient by totaling the points based on individual predictors and assessed the distribution. The derivation cohort was then stratified on outcome risk based on the individual scores. The predictive accuracy of the score was then assessed by comparing predicted vs observed outcome (calibration) and discriminant validity with AUROC.²⁵

Sepsis survivors and clinicians decide together on follow-up care, based on patients’ risk and benefit preferences. Clinicians can either provide follow-up care to all sepsis survivors (treat all) or provide no follow-up care (treat none) or use the prognostic information assessed with DCA to intervene selectively. Decision curve analysis is underpinned by 2 concepts: threshold probability and net benefit. Threshold probability of outcome refers to the empirical value above which a clinician would choose to treat or a sepsis survivor would choose to receive treatment. In other words, the sepsis survivors make a conscious decision that the benefit of their empirical treatment outweighs the harms associated with the treatment. The net benefit is the difference between the expected benefit and the expected harm. The expected benefit here is that sepsis survivors who will experience 1 or more rehospitalizations or death within 1 year after hospital discharge will choose to receive treatment (true positive). The expected harm is that sepsis survivors who will not experience this outcome will choose to receive treatment (false positive) multiplied by those sepsis survivors’ threshold probabilities. Decision curve analysis uses this theoretical association of net benefit = true positive rate − [false positive rate × [threshold probability/(1 − threshold probability)] to generate decision curves.¹⁴ The highest decision curve over the full range of threshold probabilities is the best cutoff for a prognostic score, regardless of sepsis survivors’ preferences. However, if the decision curves cross, then the optimal cutoff will be associated with sepsis survivors’ preferences of threshold probability.

We performed internal validation with bootstrapping¹² and temporal cohort external validation.¹² Bootstrap samples (n = 200) were taken from the original data, stepwise selection repeated within each bootstrap sample, with Efron optimism estimated using the difference between the bootstrap data and the original data.²⁶ Bootstrapping provides the nonparametric maximum likelihood of error and stable estimates with low risk of bias,²⁷ assuming that the derivation cohort represents a random sample of sepsis survivors.

Reported P values are 2-sided, and P < .05 was considered to represent a statistically significant result. Continuous data were summarized as mean (SD) values, when normally distributed, and as median values and interquartile ranges, when not normally distributed. Categorical data were presented as frequency and percentage. We achieved 96% outcome data linkage.⁸ Because missing data were less than 0.5% for all the predictors and because the Case Mix Programme had between 90% and 100% coverage of adult general critical care units in England during the study period, we performed full case analyses, assuming that data are missing completely at random.²⁸ We did not perform an a priori sample size calculation. All analyses were performed using Stata/SE, version 14.0 (StataCorp LLC).

There were 94 748 sepsis survivors in the derivation cohort and 24 669 sepsis survivors in the validation cohort. The study outcome of unplanned rehospitalization or death in the first year after hospital discharge occurred for 48 594 patients (51.3%) in the derivation cohort and 13 129 patients (53.2%) in the validation cohort (Figure 1). The generic and sepsis-specific characteristics were similar between the derivation and validation cohorts (Table 1; eTable 2 in the Supplement).

On univariable testing, age, sex, race/ethnicity, IMD2015 in England quintiles, preadmission dependence, all comorbidities, any hospitalization in the preceding year, admission type, illness severity, hospital length of stay, lower blood hemoglobin level at index admission, site of infection, numbers of organ dysfunction, and organ support were significantly associated with the study outcome of unplanned rehospitalization or death in the first year after hospital discharge (eTable 4 in the Supplement).

The preliminary model with all predictors is shown in eTable 5 in the Supplement. The stepwise reduction procedure to generate the poststepwise reduction model removed the following predictors: sex, blood lactate level, numbers of organ dysfunction, and organ support (respiratory, renal, gastrointestinal, and neurologic). We then sequentially eliminated complex predictors (APACHE II acute physiology score) and predictors with the lowest coefficients (race/ethnicity, advanced cardiovascular support, hospital length of stay, and hospital type). This did not alter the discrimination and calibration of the model (eTable 5 in the Supplement). All variance inflation factors were considerably less than 10 (with a mean value of 2.2 for the preliminary model and of 2.4 for the final model). The predictors in the final model were previous hospitalization, age, IMD2015 in England quintiles, preadmission dependence, number of comorbidities, admission type, blood hemoglobin level at admission, and site of infection (Table 2).

The prognostic score ranged from 0 to 22 points, with lower scores indicating a lower risk of the outcome. (Table 2). We generated 4 risk strata by grouping patients based on their individual prognostic score (Figure 2A; Table 2). The outcome of unplanned rehospitalization or death in the first year after hospital discharge occurred for 5088 of 16 684 patients (30.5%) with scores between 0 and 4 points and 15 732 of 21 641 patients (72.7%) with 11 points or more (Figure 2B; Table 2). The derivation and validation cohorts had similar distributions of prognostic scores of 0 to 4 points (5088 of 16 684 patients [30.5%] and 471 of 1725 patients [27.3%]) and prognostic scores of 11 points or more (15 732 of 21 641 patients [72.7%] and 5753 of 7952 patients [72.3%]) (Table 2). The discrimination was moderate (AUROC, 0.675; 95% CI, 0.672-0.679) (Figure 2C-D).

Net benefit was maximized with threshold probabilities of 0% to 30% by the treat-all approach. The DCA highlights that the net benefit of this score is positive and higher than the net benefit for the treat-all scenario at threshold probabilities of 30% or more (Figure 3A). In the intervening selective scenario of our analysis, a score of 5 or more has the highest curve for a threshold probability of less than 45%. At a threshold probability of 45% or more and a score of 7 or more, the true positives were 34.7% (32 868 of 94 748 patients), the false positives were 16.6% (15 725 of 94 748 patients), and the estimated net benefit was 0.21; at this cutoff, 55.3% (52 433 of 94 748 patients) of the study cohort may choose follow-up care (eTable 7 in the Supplement and Figure 3A). The individual outcome events mirrored changes in the composite outcome with our prognostic score (Figure 3B and eTable 6 in the Supplement).

The description and performance of our prognostic score in our external validation cohort are shown in eTable 8 and eFigure 2 in the Supplement. The use of predictors during the 200-sample bootstrap validation models is shown in eFigure 3 in the Supplement. With bootstrap internal validation, the mean optimism estimate was 0.0007 (95% CI, 0.0005-0.0010) for the C statistic, −0.0002 (95% CI, −0.0003 to −0.0001) for the Brier score, and 0.0005 (95% CI, 0.0003-0.0006) for the Shapiro R test score.

We derived and validated a parsimonious prognostic score for the outcome of unplanned rehospitalization or death in the first year after hospital discharge, using acute illness characteristics from the index sepsis admission as predictors. The score is calculated using 8 predictors: hospitalizations in the preceding year, age, socioeconomic status using IMD2015 in England quintiles, preadmission dependence, number of comorbidities, admission type, site of infection, and admission blood hemoglobin level, a globally feasible blood test.

What is the clinical value and use of this prognostic score? We and others have highlighted that 1 in 2 sepsis survivors have unplanned rehospitalizations or die in the first year after hospital discharge.^19,29-31 Currently, the follow-up care of sepsis survivors is neither risk based nor uniform within and between health care systems. Seldom does every sepsis survivor receive follow-up care. In this context, we report, to our knowledge, the first prognostic score that helps to identify sepsis survivors’ risk of these 2 patient-centered adverse outcomes. Although predictive accuracy at the individual level is at best modest, our score highlights a clinically useful gradient of risk to target follow-up care. We envision that, in the first instance, the score would be used to inform prognosis discussions with sepsis survivors and follow-up care designed based on patient preferences. More important, because DCA does not mandate information on the costs or effectiveness of treatment or how patients value different health states,¹⁴ our analysis highlights that it is possible to intervene selectively for sepsis survivors, using a systematic risk-based approach. An example of the follow-up care that we envision would be ensuring better postdischarge adherence to recommended management for common comorbidities, as better adherence could improve sepsis survivor outcomes^32,33 because effective outpatient management exists for these common rehospitalization diagnoses.¹⁸ We expect our score to be refined over time based on preliminary use because clinicians and researchers may identify what additional predictors could add value. We expect an increased emphasis on sepsis survivors’ health with unrestricted availability of our score. A similar approach with prognostic scoring for addressing care for older adults after hospitalizations has been reported.^34,35

The predictors in the final list are consistently reported in the literature as risk factors for rehospitalization⁶ and/or 1-year mortality⁷—hospitalization in the preceding year,^36,37 increasing age,^19,38,39 different proxies for socioeconomic status (such as insurance status,^11,38-40 urban residence, and lower income¹¹), preadmission dependence, higher comorbidity burden,^{10,11,19,38-40} admission type, blood hemoglobin level at admission, and site of infection.¹⁰ Increased risk of outcome with a lower blood hemoglobin level at index sepsis admission in the score is analogous to the association observed with a lower blood hemoglobin level at critical care discharge.^36-38 We contend that the admission blood hemoglobin level may also represent chronic health status.

Our study has some strengths. We present a novel score to stratify sepsis survivors’ risk of rehospitalization or death in the first year after hospital discharge, whereas other tools focus on 30-day rehospitalization risk.⁴¹ Our primary outcome of all-cause rehospitalizations or death has the lowest risk of ascertainment bias. The score, albeit feasible to measure even in resource-limited settings, requires recalibration and potentially revision before use as standard of care in other health care settings. As part of development work for this analysis, we published the operational definition for sepsis from this database,³ long-term mortality in the derivation cohort,⁸ and systematic reviews to identify key predictors.^6,7 Our method of handling comorbidities functions even when patients have 2 or more comorbidities, is transferable between health care systems that use different comorbidity definitions, and the incremental risk with increasing numbers of comorbidities has clinical validity. We followed the recommendations by Steyerberg and colleagues²⁷ for bootstrap validations.

Our study also has some limitations. Our study population was limited to patients who survived to hospital discharge after a critical care unit admission for sepsis. Our model predicts the 1-year risk, not necessarily at specific time points during follow-up. We did not have information on patient preferences after discharge, so 1-year mortality in some cases may reflect a lack of seeking care or limitations of care, such as end-of-life care in community settings.^42,43 However, this is unlikely to be a major factor because death without rehospitalization was observed for only 4.3% of the derivation cohort (4035 of 94 748) and 4.5% of the validation cohort (1118 of 24 669). Although we used a national deprivation index (IMD2015 in England) as a proxy for socioeconomic strata, there are other national or regional indces. Because additional risk is only for individuals in the lowest socioeconomic status, this is an easy limitation to overcome when using other indices. A similar consideration applies to preadmission dependence assessments. Perhaps the biggest limitation of our score is that currently there are no proven interventions to prevent posthospitalization death or readmissions of sepsis survivors.

Our study highlights several future research questions. Although we report internal and temporal external validations, generalizability of the score should be assessed in other settings, including resource-limited settings. There is insufficient knowledge regarding sepsis survivors’ preferences on the relevant range of threshold probabilities within DCA, which needs to be determined. Because there are no proven interventions to weigh the risk vs benefit within DCA, the utility of the score as a prognostic enrichment⁴⁴ tool could be evaluated by ongoing trials (eg, NCT03565159) and data from completed trials.⁴⁵ The score could be assessed as a benchmarking tool between hospitals to reduce variation in sepsis survivor outcomes.

In this study, unplanned rehospitalization or death in the first year after hospital discharge were common adverse outcomes among sepsis survivors. This study validated a simple and internationally feasible prognostic score for this composite outcome, which could inform prognosis discussions, trial design, and follow-up care of sepsis survivors.

Accepted for Publication: June 2, 2020.

Published: September 14, 2020. doi:10.1001/jamanetworkopen.2020.13580

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Shankar-Hari M et al. JAMA Network Open.

Corresponding Author: Manu Shankar-Hari, MSc, MD, PhD, First Floor, East Wing, St Thomas’ Hospital, Guy’s and St Thomas’ NHS Foundation Trust London, London SE17EH, UK (manu.shankar-hari@kcl.ac.uk).

Author Contributions: Drs Shankar-Hari and Harrison had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Harrison and Rowan contributed equally to this work.

Concept and design: Shankar-Hari, Rubenfeld, Harrison, Rowan.

Acquisition, analysis, or interpretation of data: Shankar-Hari, Ferrando-Vivas, Harrison, Rowan.

Drafting of the manuscript: Shankar-Hari, Rowan.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Shankar-Hari, Rubenfeld, Harrison.

Obtained funding: Shankar-Hari.

Administrative, technical, or material support: Rowan.

Supervision: Harrison, Rowan.

Conflict of Interest Disclosures: Dr Rubenfeld reported serving as a consultant for Endpoint Health. Dr Harrison reported receiving grants from the National Institute for Health Research during the conduct of the study. No other disclosures were reported.

Funding/Support: Dr Shankar-Hari is funded by a National Institute for Health Research Clinician Scientist Award (CS-2016-16-011) for this research project. This publication presents independent research funded by the National Institute for Health Research.

Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The views expressed in this publication are those of the authors and not necessarily those of the National Health Service, the National Institute for Health Research, or the Department of Health and Social Care. Dr Rubenfeld is an associate editor of JAMA Network Open, but he was not involved in any of the decisions regarding review of the manuscript or its acceptance.

Additional Contributions: As part of the preparation for this research, in the write up of theNational Institute for Health Research Clinician Scientist Award application, Dr Manu Shankar-Hari (CS-2016-16-011) consulted survivors of sepsis to explore the longer-term health care requirements, which informed the analysis plan for this study.

Additional Information: The data used in this study are accessible from the UK National Intensive Care Admissions dataset managed by the Intensive Care National Audit & Research Centre data set. A free-to-use online tool is available at https://sepsis-ssip-score.org.uk/ssip_score.