Association Between State-Mandated Protocolized Sepsis Care and In-hospital Mortality Among Adults With Sepsis | Critical Care Medicine | JAMA | JAMA Network
[Skip to Content]
[Skip to Content Landing]
Figure 1.  Quarter-Specific Estimates of the Primary Outcome
Quarter-Specific Estimates of the Primary Outcome

The preregulation period includes hospitalizations from January 1, 2011, through March 31, 2013. The postregulation period includes hospitalizations from April 1, 2013, through September 30, 2015.

In A, the circles and squares represent the per-quarter estimates along with fitted ordinary least-squares lines within the preperiod and postperiod. In B, comparative interrupted time series model results are shown. During the preregulation period, the dotted lines represent the risk-adjusted temporal trends in outcome from the fully adjusted model. These lines are extended into the postregulation period, where they represent the counterfactual adjusted outcomes had the preregulation trends continued. The circles and squares represent the adjusted quarter-specific postintervention estimates from the fully adjusted model. Values above the dotted line indicate that adjusted values for that quarter are higher than the counterfactual, and values below the dotted line indicate that adjusted values are lower than the counterfactual. Estimates were adjusted for age, sex, race and ethnicity, admission through the emergency department, transfer from an acute care hospital, number of organ failures present at hospital admission, sepsis infection categories, chronic comorbid conditions, hospital characteristics, season, and preregulation temporal trends.

Figure 2.  Quarter-Specific Estimates of the 4 Secondary Outcomes
Quarter-Specific Estimates of the 4 Secondary Outcomes

In A, C, E, and G, the circles and squares represent the per-quarter estimates along with fitted ordinary least-squares lines within the preperiod and postperiod. In B, D, F, and H, the circles and squares represent quarter-specific postintervention estimates from the fully adjusted model. During the preregulation period, the dotted lines represent the risk-adjusted temporal trends in outcome from the fully adjusted model. These lines are extended into the postregulation period, where they represent the counterfactual adjusted outcomes had the preregulation trends continued.

Table 1.  Characteristics of Study Hospitals
Characteristics of Study Hospitals
Table 2.  Admission Characteristics, Discharge Disposition, and Unadjusted Study Outcomes for Patients With Sepsis in the Primary Analysis
Admission Characteristics, Discharge Disposition, and Unadjusted Study Outcomes for Patients With Sepsis in the Primary Analysis
Table 3.  Adjusted Quarter-Specific Estimates of the Association Between the New York State Sepsis Regulations and Patient Outcomes From the Analysis for the Primary Outcome and the 4 Secondary Outcomes
Adjusted Quarter-Specific Estimates of the Association Between the New York State Sepsis Regulations and Patient Outcomes From the Analysis for the Primary Outcome and the 4 Secondary Outcomes
1.
Singer  M, Deutschman  CS, Seymour  CW,  et al.  The third internatonal consensus definitions for sepsis and septic shock (Sepsis-3).  JAMA. 2016;315(8):801-810.PubMedGoogle ScholarCrossref
2.
Dellinger  RP, Levy  MM, Rhodes  A,  et al; Surviving Sepsis Campaign Guidelines Committee including the Pediatric Subgroup.  Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012.  Intensive Care Med. 2013;39(2):165-228. doi:10.1007/s00134-012-2769-8PubMedGoogle ScholarCrossref
3.
Levy  MM, Dellinger  RP, Townsend  SR,  et al.  The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis.  Intensive Care Med. 2010;36(2):222-231. doi:10.1007/s00134-009-1738-3PubMedGoogle ScholarCrossref
4.
Levy  MM, Rhodes  A, Phillips  GS,  et al.  Surviving Sepsis Campaign: association between performance metrics and outcomes in a 7.5-year study.  Crit Care Med. 2015;43(1):3-12. doi:10.1097/CCM.0000000000000723PubMedGoogle ScholarCrossref
5.
Rhodes  A, Phillips  G, Beale  R,  et al.  The Surviving Sepsis Campaign bundles and outcome: results from the International Multicentre Prevalence Study on Sepsis (the IMPreSS study).  Intensive Care Med. 2015;41(9):1620-1628. doi:10.1007/s00134-015-3906-yPubMedGoogle ScholarCrossref
6.
Cooke  CR, Iwashyna  TJ.  Sepsis mandates: improving inpatient care while advancing quality improvement.  JAMA. 2014;312(14):1397-1398. doi:10.1001/jama.2014.11350PubMedGoogle ScholarCrossref
7.
Hershey  TB, Kahn  JM.  State sepsis mandates—a new era for regulation of hospital quality.  N Engl J Med. 2017;376(24):2311-2313. doi:10.1056/NEJMp1611928PubMedGoogle ScholarCrossref
8.
Seymour  CW, Gesten  F, Prescott  HC,  et al.  Time to treatment and mortality during mandated emergency care for sepsis.  N Engl J Med. 2017;376(23):2235-2244. doi:10.1056/NEJMoa1703058PubMedGoogle ScholarCrossref
9.
Liu  VX, Fielding-Singh  V, Greene  JD,  et al.  The timing of early antibiotics and hospital mortality in sepsis.  Am J Respir Crit Care Med. 2017;196(7):856-863. doi:10.1164/rccm.201609-1848OCPubMedGoogle ScholarCrossref
10.
Yealy  DM, Kellum  JA, Huang  DT,  et al; ProCESS Investigators.  A randomized trial of protocol-based care for early septic shock.  N Engl J Med. 2014;370(18):1683-1693. doi:10.1056/NEJMoa1401602PubMedGoogle ScholarCrossref
11.
Rhee  C, Gohil  S, Klompas  M.  Regulatory mandates for sepsis care—reasons for caution.  N Engl J Med. 2014;370(18):1673-1676. doi:10.1056/NEJMp1400276PubMedGoogle ScholarCrossref
12.
Dimick  JB, Ryan  AM.  Methods for evaluating changes in health care policy: the difference-in-differences approach.  JAMA. 2014;312(22):2401-2402. doi:10.1001/jama.2014.16153PubMedGoogle ScholarCrossref
13.
Kahn  JM. Quantitative evaluation of the effects the 2013 New York State sepsis regulations. https://osf.io/jcwdv/. Accessed May 18, 2018.
14.
Dombrovskiy  VY, Martin  AA, Sunderram  J, Paz  HL.  Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003.  Crit Care Med. 2007;35(5):1244-1250. doi:10.1097/01.CCM.0000261890.41311.E9PubMedGoogle ScholarCrossref
15.
Iwashyna  TJ, Odden  A, Rohde  J,  et al.  Identifying patients with severe sepsis using administrative claims: patient-level validation of the angus implementation of the international consensus conference definition of severe sepsis.  Med Care. 2014;52(6):e39-e43. doi:10.1097/MLR.0b013e318268ac86PubMedGoogle ScholarCrossref
16.
Prescott  HC, Cope  TM, Gesten  FC,  et al.  Reporting of sepsis cases for performance measurement versus for reimbursement in New York State.  Crit Care Med. 2018;46(5):666-673. doi:10.1097/CCM.0000000000003005PubMedGoogle ScholarCrossref
17.
Angus  DC, Barnato  AE, Bell  D,  et al.  A systematic review and meta-analysis of early goal-directed therapy for septic shock: the ARISE, ProCESS and ProMISe Investigators.  Intensive Care Med. 2015;41(9):1549-1560. doi:10.1007/s00134-015-3822-1PubMedGoogle ScholarCrossref
18.
Rowan  KM, Angus  DC, Bailey  M,  et al; PRISM Investigators.  Early, goal-directed therapy for septic shock—a patient-level meta-analysis.  N Engl J Med. 2017;376(23):2223-2234. doi:10.1056/NEJMoa1701380PubMedGoogle ScholarCrossref
19.
Darby  JL, Davis  BS, Barbash  IJ, Kahn  JM.  An administrative model for benchmarking hospitals on their 30-day sepsis mortality.  BMC Health Serv Res. 2019;19(1):221. doi:10.1186/s12913-019-4037-xPubMedGoogle ScholarCrossref
20.
Elias  KM, Moromizato  T, Gibbons  FK, Christopher  KB.  Derivation and validation of the acute organ failure score to predict outcome in critically ill patients: a cohort study.  Crit Care Med. 2015;43(4):856-864. doi:10.1097/CCM.0000000000000858PubMedGoogle ScholarCrossref
21.
Ames  SG, Davis  BS, Angus  DC, Carcillo  JA, Kahn  JM.  Hospital variation in risk-adjusted pediatric sepsis mortality.  Pediatr Crit Care Med. 2018;19(5):390-396. doi:10.1097/PCC.0000000000001502PubMedGoogle ScholarCrossref
22.
Quan  H, Sundararajan  V, Halfon  P,  et al.  Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.  Med Care. 2005;43(11):1130-1139. doi:10.1097/01.mlr.0000182534.19832.83PubMedGoogle ScholarCrossref
23.
Mayr  FB, Yende  S, Linde-Zwirble  WT,  et al.  Infection rate and acute organ dysfunction risk as explanations for racial differences in severe sepsis.  JAMA. 2010;303(24):2495-2503. doi:10.1001/jama.2010.851PubMedGoogle ScholarCrossref
24.
Abadie  A.  Semiparametric difference-in-differences estimators.  Rev Econ Stud. 2005;72(1):1-19. doi:10.1111/0034-6527.00321Google ScholarCrossref
25.
Imbens  GW, Wooldridge  JM.  Recent developments in the econometrics of program evaluation.  J Econ Lit. 2009;47(1):5-86. doi:10.1257/jel.47.1.5Google ScholarCrossref
26.
Jacob  R, Somers  M-A, Zhu  P, Bloom  H.  The validity of the comparative interrupted time series design for evaluating the effect of school-level interventions.  Eval Rev. 2016;40(3):167-198. doi:10.1177/0193841X16663414PubMedGoogle ScholarCrossref
27.
St Clair  T, Hallberg  K, Cook  TD.  The validity and precision of the comparative interrupted time-series design: three within-study comparisons.  J Educ Behav Stat. 2016;41(3):269-299. doi:10.3102/1076998616636854Google ScholarCrossref
28.
Danai  PA, Sinha  S, Moss  M, Haber  MJ, Martin  GS.  Seasonal variation in the epidemiology of sepsis.  Crit Care Med. 2007;35(2):410-415. doi:10.1097/01.CCM.0000253405.17038.43PubMedGoogle ScholarCrossref
29.
Kaukonen  K-M, Bailey  M, Suzuki  S, Pilcher  D, Bellomo  R.  Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012.  JAMA. 2014;311(13):1308-1316. doi:10.1001/jama.2014.2637PubMedGoogle ScholarCrossref
30.
Greater New York Hospital Association. STOP sepsis. https://www.gnyha.org/topic/sepsis/. Accessed May 18, 2018.
31.
Levy  MM, Gesten  FC, Phillips  GS,  et al.  Mortality changes associated with mandated public reporting for sepsis: results of the New York State initiative.  Am J Respir Crit Care Med. 2018;198(11):1406-1412. doi:10.1164/rccm.201712-2545OCPubMedGoogle ScholarCrossref
32.
Rhee  C, Dantes  R, Epstein  L,  et al; CDC Prevention Epicenter Program.  Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.  JAMA. 2017;318(13):1241-1249. doi:10.1001/jama.2017.13836PubMedGoogle ScholarCrossref
33.
Thomas  L, Peterson  ED.  The value of statistical analysis plans in observational research: defining high-quality research from the start.  JAMA. 2012;308(8):773-774. doi:10.1001/jama.2012.9502PubMedGoogle ScholarCrossref
Original Investigation
July 16, 2019

Association Between State-Mandated Protocolized Sepsis Care and In-hospital Mortality Among Adults With Sepsis

Author Affiliations
  • 1CRISMA Center, Department of Critical Care Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania
  • 2Department of Health Policy and Management, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania
  • 3Division of General Internal Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania
  • 4Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania
  • 5Division of Pulmonary, Allergy and Critical Care Medicine, Columbia University Medical Center, New York, New York
  • 6Department of Acute and Tertiary Care, School of Nursing, University of Pittsburgh, Pittsburgh, Pennsylvania
  • 7RAND Corporation, Pittsburgh, Pennsylvania
JAMA. 2019;322(3):240-250. doi:10.1001/jama.2019.9021
Key Points

Question  Were the 2013 New York State regulations mandating the use of protocols for sepsis recognition and treatment associated with in-hospital mortality differences compared with states that did not implement sepsis regulations?

Findings  In this retrospective cohort study of 1 012 410 hospitalized adults with sepsis, mandated protocolized sepsis care in New York State was associated with a significantly greater decline in risk-adjusted mortality in New York compared with a group of control states that did not implement mandated protocolized sepsis care. By the 10th quarter after implementation of the regulations, the adjusted absolute mortality was 3.2% lower than expected in New York State relative to the control states.

Meaning  The New York State sepsis regulations were associated with significantly reduced sepsis mortality, but whether broader adoption of state-level sepsis mandates in other states would lead to further reductions in sepsis-related mortality is unknown.

Abstract

Importance  Beginning in 2013, New York State implemented regulations mandating that hospitals implement evidence-based protocols for sepsis management, as well as report data on protocol adherence and clinical outcomes to the state government. The association between these mandates and sepsis outcomes is unknown.

Objective  To evaluate the association between New York State sepsis regulations and the outcomes of patients hospitalized with sepsis.

Design, Setting, and Participants  Retrospective cohort study of adult patients hospitalized with sepsis in New York State and in 4 control states (Florida, Maryland, Massachusetts, and New Jersey) using all-payer hospital discharge data (January 1, 2011-September 30, 2015) and a comparative interrupted time series analytic approach.

Exposures  Hospitalization for sepsis before (January 1, 2011-March 31, 2013) vs after (April 1, 2013-September 30, 2015) implementation of the 2013 New York State sepsis regulations.

Main Outcomes and Measures  The primary outcome was 30-day in-hospital mortality. Secondary outcomes were intensive care unit admission rates, central venous catheter use, Clostridium difficile infection rates, and hospital length of stay.

Results  The final analysis included 1 012 410 sepsis admissions to 509 hospitals. The mean age was 69.5 years (SD, 16.4 years) and 47.9% were female. In New York State and in the control states, 139 019 and 289 225 patients, respectively, were admitted before implementation of the sepsis regulations and 186 767 and 397 399 patients, respectively, were admitted after implementation of the sepsis regulations. Unadjusted 30-day in-hospital mortality was 26.3% in New York State and 22.0% in the control states before the regulations, and was 22.0% in New York State and 19.1% in the control states after the regulations. Adjusting for patient and hospital characteristics as well as preregulation temporal trends and season, mortality after implementation of the regulations decreased significantly in New York State relative to the control states (P = .02 for the joint test of the comparative interrupted time series estimates). For example, by the 10th quarter after implementation of the regulations, adjusted absolute mortality was 3.2% (95% CI, 1.0% to 5.4%) lower than expected in New York State relative to the control states (P = .004). The regulations were associated with no significant differences in intensive care unit admission rates (P = .09) (10th quarter adjusted difference, 2.8% [95% CI, −1.7% to 7.2%], P = .22), a significant relative decrease in hospital length of stay (P = .04) (10th quarter adjusted difference, 0.50 days [95% CI, −0.47 to 1.47 days], P = .31), a significant relative decrease in the C difficile infection rate (P < .001) (10th quarter adjusted difference, −1.8% [95% CI, −2.6% to −1.0%], P < .001), and a significant relative increase in central venous catheter use (P = .02) (10th quarter adjusted difference, 4.8% [95% CI, 2.3% to 7.4%], P < .001).

Conclusions and Relevance  In New York State, mandated protocolized sepsis care was associated with a greater decrease in sepsis mortality compared with sepsis mortality in control states that did not implement sepsis regulations. Because baseline mortality rates differed between New York and comparison states, it is uncertain whether these findings are generalizable to other states.

Introduction

Sepsis is a leading cause of morbidity and mortality in the United States.1 Several treatments are of proven effectiveness in this population, including timely administration of antibiotics and early resuscitation with intravenous fluids.2 However, many patients with sepsis do not receive these evidence-based practices, leading to excess morbidity and mortality.3-5 To address this problem, policy makers are increasingly turning to regulatory mechanisms designed to mandate sepsis performance improvement in the form of care protocols for early recognition and treatment.6

A pioneering example of these mandates is the regulations issued by the New York State Department of Health during May 2013, known as Rory’s Regulations after a 12-year-old boy who died of sepsis.7 These regulations require all acute care hospitals in the state to develop and implement protocols for timely recognition and treatment of sepsis, including administration of antibiotics by 3 hours and resuscitation with intravenous fluid by 6 hours for patients with signs of hypoperfusion. The regulations also require hospitals to routinely train their staff in protocol implementation and report both protocol adherence and clinical outcomes to the state’s department of health.

Although timely sepsis treatment is supported by robust observational and clinical trial data,8-10 the role of governmental mandates as a strategy to enforce the use of sepsis protocols remains controversial.11 Sepsis mandates could encourage uptake of evidence-based care practices, leading to reduced mortality, but could also encourage overuse of intravenous fluids and antibiotics, leading to adverse consequences.7 The goal of this study was to examine sepsis outcomes before and after implementation of the sepsis regulations in New York State, comparing these changes with the outcomes in other states that did not implement sepsis regulations during this time.

Methods
Study Design

We performed a retrospective cohort study of hospitalized patients with sepsis. The study was approved by the University of Pittsburgh Human Research Protection Office, which deemed the study exempt from human subjects review because it was a secondary analysis of existing data and waived the need for informed consent (PRO17110272).

We used a comparative interrupted time series study design, comparing New York State with the 4 control states of Florida, Maryland, Massachusetts, and New Jersey. These control states were chosen because they have similar demographic characteristics to New York and, except for Florida, they are geographically proximal to New York. A comparative interrupted time series study compares the longitudinal outcome changes between an intervention group and a control group, thereby subtracting underlying secular trends and any other changes that may have occurred in both groups. By using this approach, we were able make inferences about the regulations that would not be possible using New York State data alone.12

To support the rigor and reproducibility of our results, all analyses were prespecified prior to receipt of the final data set and a detailed statistical analysis plan (Supplement 1) was published online13 (additional details appear in the eMethods in Supplement 2). Deviations from this plan due to unforeseen circumstances are noted as post hoc, and a rationale for all deviations appears in the eMethods in Supplement 2.

Data Sources

Our primary data source was the Agency for Healthcare Research and Quality’s Healthcare Cost and Utilization Project State Inpatient Database. The State Inpatient Database contains patient-level administrative data for all hospitalizations in participating states. We used the State Inpatient Database to identify hospitalizations that occurred from January 1, 2011, through September 30, 2015. We linked the data from the State Inpatient Database to hospital-level data from the 2015 Centers for Medicare & Medicaid Healthcare Cost Reporting Information System to obtain hospital characteristics such as hospital type, number of beds, and academic status; and used the 2010 US Census to obtain data on each hospital’s metropolitan statistical area population.

Patients and Hospitals

We identified hospital admissions with sepsis using validated International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes for infection and organ failure.14 This strategy, known as the Dombrovskiy strategy, is less specific but more sensitive than approaches that rely solely on the explicit ICD-9-CM codes for sepsis,15 and captures a slightly larger patient population than is identified by retrospective chart review.16 In choosing a broad sepsis identification strategy, we sought to account for the fact that many patients with sepsis may be missed by chart review yet are still eligible for evidence-based practices. We excluded admissions for patients younger than 18 years, admissions to hospitals that could not be identified in the Healthcare Cost Reporting Information System, and admissions with missing data for key covariates. We further excluded hospitals that were not classified as short-stay acute care hospitals by the Healthcare Cost Reporting Information System, hospitals with no sepsis admissions, and, to create a more homogenous sample, hospital types that were not shared across New York State and the control states both before and after the introduction of the regulations (more detail on this process appears in the eMethods in Supplement 2).

Outcomes

The primary outcome variable was 30-day in-hospital mortality. We also examined 4 secondary outcome variables that reflected potential adverse unintended consequences of the regulations: intensive care unit (ICU) admission rate, hospital length of stay, central venous catheter use, and Clostridium difficile infection rate. The ICU admission rates were examined as a marker for health care intensity because data suggest that protocolized sepsis treatment may increase ICU admissions.17 Hospital length of stay was examined as a proxy for resource use because data suggest that protocolized sepsis treatment may increase hospital costs.18 Rates for central venous catheter use were examined based on the hypothesis that the sepsis mandate could lead to an increase in invasive central catheter insertion for monitoring and resuscitation.11C difficile infection rates were examined based on the hypothesis that the sepsis mandate may encourage antibiotic overuse, leading to an increase in cases of C difficile infection.11

Additional Variables

Variables for case-mix adjustment were based on a previously published risk-adjustment model for sepsis,19 and included age; sex; race and ethnicity; admission through the emergency department; transfer from an acute care hospital; cases of organ failure present at admission, which were defined similar to the study by Elias et al20; sepsis infection categories, which were defined similar to the study by Ames et al21; and Elixhauser chronic comorbid conditions.22 We included race as a potential confounder based on prior studies demonstrating an independent relationship between race and sepsis outcomes.23 Race and ethnicity were obtained directly from the hospital discharge record, which is based on patient self-report either directly or via the patient’s primary insurer. A full list of variables and their definitions appears in eTable 1 in Supplement 2.

Statistical Analysis
Primary Analysis

The hospital characteristics were compared between New York State and the control states using the χ2 test. We examined patient characteristics between New York State and the control states before and after implementation of the regulations, but did not formally test for differences because the large sample size made it likely that all tests would be significant. For cases in which unexpected differences were found between New York State and the control states, additional post hoc comparisons were performed to better understand and provide additional context for the results.

A comparative interrupted time series analysis was performed to test the relationship between the New York State sepsis regulations and outcomes.24,25 We performed the analysis separately for each outcome variable described above. In specifying the models, we accounted for the possibility that the association between the regulations and the outcomes might change over time due to their staged implementation (eTable 2 in Supplement 2 contains a complete policy timeline). Accordingly, rather than specifying a single postimplementation temporal trend, we fit a model with indicators for each postimplementation quarter.26,27 The preimplementation period was defined as hospital discharge from January 1, 2011, through March 31, 2013, before the filing of the regulations. The postimplementation period was defined as hospital discharge from April 1, 2013, through September 30, 2015.

All models were fit using linear regression with robust standard errors clustered at the hospital level. For binary outcomes, this approach corresponds to a linear probability model and the coefficients represent the between-group absolute risk differences. All models controlled for the patient and hospital characteristics listed above, as well as seasonality based on calendar quarter.28 We also controlled for preregulation temporal trends using a continuous time variable, implemented as quarters, as well as a treatment indicator × continuous time variable interaction term. This approach accounts for the fact that sepsis outcomes were generally improving over time,29 as well as the possibility that preimplementation temporal trends might differ between New York State and the control states.26,27

To test the association between the regulations and the patient outcomes, we included a postimplementation quarter × intervention group interaction term (ie, New York State vs the control states). The estimates for these interaction terms are interpreted as the difference in the deviations from the counterfactual preregulation trend between New York State and the control states during that quarter; or, more simply, the estimated association between the regulations and the patient outcomes during the given quarter. The primary test of the association between the regulations and the patient outcomes was a joint test that all of the quarter-specific estimates were equal to zero. To understand the direction and magnitude of any observed overall associations, we also calculated point estimates and 95% CIs for each individual interaction term. The results from the 10th quarter after implementation (ie, during 2015, quarter 3) are highlighted herein as a representative example of the quarter-specific associations.

The comparative interrupted time series model can be simplified to a traditional difference-in-differences model if trends for the outcomes were parallel in New York State and in the control states prior to the regulations. We directly examined for this possibility by fitting a model containing a treatment indicator, a continuous time variable, the interaction of these 2 variables, and all patient- and hospital-level covariates, restricted to the preregulation period. We did this separately for each outcome. We considered parallel trends as being present if the interaction term from this model was not significant. In cases in which there were parallel trends, we simplified the comparative interrupted time series model to a difference-in-differences model by excluding the term for the interaction of the treatment indicator with the continuous time variable.26,27 The P values from the tests of parallel trends and the respective models used are presented alongside the model results.

Secondary and Sensitivity Analyses

A concern regarding the use of administrative codes to identify sepsis is that the regulations could have changed sepsis coding patterns, potentially biasing the results. To understand this issue, a secondary analysis was performed in which we fit a similar model as described above, except with all adult hospital admissions as the population and an indicator for sepsis as the primary dependent variable. A negative test of the interaction terms would indicate that the regulations were not associated with changes in administrative coding for sepsis.

A number of prespecified sensitivity analyses were also performed to examine the robustness of the results to our design decisions. Specifically, we repeated the primary analysis, limiting the sample to patients with severe sepsis and septic shock as defined using the ICD-9-CM codes that explicitly identify sepsis and septic shock15; expanding the sample to patients with sepsis according to a broader definition that includes additional organ failures15; excluding New York City hospitals that had participated in an earlier Greater New York Hospital Association sepsis quality improvement initiative30; and shifting the preimplementation period back in time by 2 quarters to account for the possibility that hospitals began implementing the regulations when they were first announced. One post hoc sensitivity analysis was performed, restricting the control states to those with preregulation temporal trends that were most similar to New York to account for the possibility that the results were driven by unmeasured differences between New York State and the control states.

In addition, we performed subgroup analyses on our primary model based on age, number of comorbidities, number of organ failures, emergency department use, hospital size, hospital academic status, and hospital sepsis volume. For each subgroup, we tested for heterogeneity of the association between the regulations and patient outcomes using 3-way interaction terms and applying the Bonferroni method to correct for multiple comparisons.

Because of the potential for type I error due to multiple comparisons, findings for the analyses of the secondary end points should be interpreted as exploratory. Additional details about the modeling strategy appear in Supplement 1 and in the eMethods in Supplement 2. Statistical analyses were performed using Stata version 15.0 (StataCorp). All tests were 2-sided and a P value of .05 or less was considered significant.

Results
Patients and Hospitals

A patient flow diagram appears in the eFigure in Supplement 2. The final analysis included 1 012 410 sepsis admissions to 509 hospitals. After patient and hospital exclusions, there were 325 786 sepsis admissions to 163 hospitals in New York State, and 686 624 sepsis admissions to 346 hospitals in the control states. The mean age was 69.5 years (SD, 16.4 years) and 47.9% were female. In New York State and in the control states, 139 019 and 289 225 patients, respectively, were admitted before implementation of the sepsis regulations and 186 767 and 397 399 patients, respectively, were admitted after implementation of the sepsis regulations. Compared with hospitals in the control states, the hospitals in New York State were more likely to be teaching hospitals and tended to have smaller ICUs (Table 1). The characteristics of patients with sepsis were generally similar in New York State and in the control states both before and after implementation of the sepsis regulations (Table 2).

Unadjusted mortality was modestly higher in New York State compared with the control states both before and after implementation of the sepsis regulations (Table 2). Unadjusted mortality was 26.3% in New York State and 22.0% in the control states before implementation of the sepsis regulations, and was 22.0% in New York State and 19.1% in the control states after implementation of the sepsis regulations. The ICU admission rates were substantially lower in New York State compared with the control states (preregulation period: 59.2% vs 76.4%, respectively); a post hoc analysis indicated that these differences were related to differences in ICU bed supply (eTable 3 in Supplement 2), with similar ICU admission rates at hospitals with similar numbers of beds (eg, preregulation ICU admission rates at hospitals with <10 beds: 52.3% in New York State vs 65.4% in the control states). Central venous catheter use was also substantially lower in New York State compared with the control states (preregulation period: 37.3% vs 48.0%, respectively); a post hoc analysis indicated that these differences were related to differences in ICU admission rates (eTable 4 in Supplement 2), with similar rates of central venous catheter use conditional on admission to the ICU (eg, preregulation central venous catheter use in patients admitted to the ICU: 49.0% in New York State vs 55.0% in the control states).

Primary and Secondary Outcome Analyses

The results of the primary and secondary outcome analyses appear in Table 3. Controlling for patient characteristics, hospital characteristics, seasonality, and preregulation temporal trends, the introduction of the regulations was associated with a significant relative decrease in the adjusted risk of 30-day in-hospital mortality (primary outcome; P = .02 for the joint test of the comparative interrupted time series estimates). The association was consistent across all periods. For the 10 postregulation quarters, all point estimates were negative and 7 were statistically significantly different than 0.

For example, during the last study quarter after implementation of the regulations (July 2015 through September 2015), the adjusted absolute mortality was 3.2% (95% CI, 1.0%-5.4%) lower than expected in New York State relative to the control states (P = .004). This estimate represents the absolute reduction in mortality associated with the regulations during this quarter. These results, along with unadjusted mortality trends before and after implementation of the sepsis regulations are shown graphically in Figure 1. The figure suggests that baseline risk-adjusted mortality was higher in New York State compared with the control states, declined over time in both groups both before and after implementation of the sepsis regulations, and declined more rapidly over time in New York State after implementation, although not to the point that it was level with the control states. Quarterly absolute adjusted mortality rates for New York State and the control states appear in eTable 5 in Supplement 2.

The results for the secondary outcomes appear in Table 3 and in Figure 2 (with the quarterly absolute adjusted outcomes for New York State and the control states in eTables 6-9 in Supplement 2). The sepsis regulations were associated with no significant differences in ICU admission rates (P = .09 for the joint test of significance) (10th quarter [labeled 2015, quarter 3, in Table 3] adjusted difference, 2.8% [95% CI, −1.7% to 7.2%], P = .22), a significant relative decrease in C difficile infection rates (P < .001 for the joint test of significance) (10th quarter adjusted difference, −1.8% [95% CI, −2.6% to −1.0%], P < .001), and a significant relative increase in central venous catheter use (P = .02 for the joint test significance) (10th quarter adjusted difference, 4.8% [95% CI, 2.3% to 7.4%], P < .001). Although the overall estimates for hospital length of stay were significant (P = .04 for the joint test of significance), these results appeared to be related to early differences with no significant differences later in the study period (10th quarter adjusted difference, 0.50 days [95% CI, −0.47 to 1.47 days], P = .31).

Sensitivity Analyses

There was no significant association between the regulations and administrative coding of sepsis, indicating that the results are unlikely to be related to changes in coding practices (eTable 10 in Supplement 2). Repeating the analysis of 30-day in-hospital mortality using 2 different administrative definitions for sepsis, excluding hospitals that participated in an earlier Greater New York Hospital Association sepsis quality improvement initiative,30 and shifting the preimplementation period back by 2 quarters yielded similar results to the primary analysis (eTables 11-14 in Supplement 2).

Repeating the primary analysis in states with preregulation temporal trends that were not significantly different than New York State also yielded similar estimates; however, the results were not statistically significant in light of the smaller sample size and risk estimates that were slightly closer to the null (eTable 15 in Supplement 2). The observed associations were not significantly different among any prespecified subgroups (eTable 16 in Supplement 2).

Discussion

A health policy mandate requiring protocolized sepsis care in New York State was associated with a statistically significant reduction in risk-adjusted mortality among adults hospitalized with sepsis compared with 4 control states that had not adopted such regulations.

The mechanism of this finding is unknown. It is likely related to a combination of factors, including increased use of early antibiotics and intravenous fluids and staff education as mandated by the regulations. It may also be reflective of a postregulation organizational culture shift that emphasizes continuous quality improvement for acutely ill patients.

There was no evidence of a change in ICU use associated with the regulations, indicating no large-scale changes in treatment intensity. However, there was a significant relative increase in central venous catheter use, which might occur if the regulations were associated with more insertions of central venous catheters to facilitate intensive physiological monitoring. It is notable that this difference appeared to be related to a steeper decline in use of central venous catheters in the 4 control states compared with New York State. This finding may be related to the publication of a large clinical trial suggesting the lack of a role for early-goal directed sepsis treatment, which occurred about the same time as the New York State sepsis regulations.10 It is likely that this trial’s publication was associated with a general decrease in central venous catheter use that was offset by an increase in New York State under the regulations.

The regulations were also associated with a significant relative increase in hospital length of stay; however, this finding was not consistent across the postregulation periods. In addition, implementation of the sepsis regulations was associated with a significant relative decrease in C difficile infection rates. Although the hypothesis was that implementation of the sepsis regulations would be associated with more antibiotic use and thus more C difficile infection, it is possible that better sepsis treatment was associated with decreased cases of organ failure, which lowered the risk for C difficile. Because the analysis was limited to patients with sepsis, it is still possible that the regulations were associated with antibiotic use and C difficile infection in the general population.

These results extend the findings from recent studies8,31 using clinical data reported to the New York State Department of Health under the regulations. These studies showed that early use of antibiotics was associated with lower mortality,8 and that sepsis outcomes in New York State were improving over time,31 observations that are consistent with others in the sepsis literature.9,29 Although these studies demonstrate the value of early sepsis treatment, they could not directly examine the regulations because sepsis outcomes are known to be improving over time.29,32 The present study overcomes that limitation by using data from before implementation of the sepsis regulations and data from 4 control states that did not implement the sepsis regulations.

Taken together, these results provide support for the New York State sepsis policy and others like it. To our knowledge, this policy is the first example of a government-issued sepsis policy designed to incentivize quality improvement by mandating evidence-based care.7 Similar policies were later implemented in Illinois and New Jersey, and a number of other states have sepsis policies under development. At the same time, the observed patterns of sepsis mortality raise concerns about the generalizability of the findings outside of New York State. Specifically, baseline sepsis mortality was higher in New York State compared with the 4 control states, and mortality was declining less rapidly in New York State compared with the control states prior to implementation of the sepsis regulations. Thus, the regulations may have served primarily to correct relatively poor-quality sepsis care in New York State. States with high-quality sepsis care may not see similar results if they adopt sepsis regulations.

Limitations

This study has several limitations. First, by identifying sepsis using administrative codes it is possible that some patients in the study did not have sepsis or that some patients with sepsis were missed. However, there was no evidence that the regulations were associated with sepsis coding patterns in a way that would cause bias, and the primary coding schema yielded a patient population similar to that identified through retrospective chart review, mitigating this concern.19 It is also possible that other changes in coding patterns (such as differential upcoding of suspected sepsis diagnoses, or better documentation of comorbidities in New York State vs the control states) could have influenced the results, but the similarities between the adjusted and unadjusted findings make this unlikely.

Second, the study did not examine postdischarge outcomes such as postdischarge mortality, long-term mortality, or functional status. However, there was no evidence of an increase in postacute care use in New York State after implementation of the regulations, suggesting that the observed mortality findings are not simply due to earlier discharge to postacute care (Table 2).

Third, the study was unable to directly test the relationship between the sepsis protocol and outcome. Rather, the study was designed to examine the policy as a whole, rather than sepsis protocols, which are just 1 part of the larger policy.

Fourth, the results could be sensitive to the modeling approach. However, the results were robust to several sensitivity analyses addressing that possibility, and the act of prepublishing a statistical analysis plan reduced the likelihood that the findings are an artifact of the statistical methods.33

Conclusions

In New York State, mandated protocolized sepsis care was associated with a greater decrease in sepsis mortality compared with sepsis mortality in control states that did not implement sepsis regulations. Because baseline mortality rates differed between New York and comparison states, it is uncertain whether these findings are generalizable to other states.

Back to top
Article Information

Corresponding Author: Jeremy M. Kahn, MD, MS, University of Pittsburgh, 3550 Terrace St, Scaife Hall, Room 602-B, Pittsburgh, PA 15261 (jeremykahn@pitt.edu).

Accepted for Publication: June 6, 2019.

Author Contributions: Dr Kahn had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Kahn, Yabes, Chang, Chong, Hershey, Angus.

Acquisition, analysis, or interpretation of data: Kahn, Davis, Yabes, Chang, Chong, Martsolf.

Drafting of the manuscript: Kahn, Yabes, Angus.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Davis, Yabes, Chang, Martsolf, Angus.

Obtained funding: Kahn.

Administrative, technical, or material support: Chong, Angus.

Supervision: Yabes, Chong, Angus.

Conflict of Interest Disclosures: Dr Yabes reported receiving grants from the National Institutes of Health. Dr Chong reported receiving grants from the University of Pittsburg. Dr Angus reported receiving grants from the National Institutes of Health; receiving personal fees from Ferring Pharmaceutical Inc, Bristol-Myers Squibb, Bayer AG, GenMark Diagnostics, Sobi Inc, Beckman Coulter Inc, and ALung Technologies Inc; and having patents pending for Selepressin (compounds, compositions, and methods for treating sepsis) and Proteomic biomarkers of sepsis in elderly patients. No other disclosures were reported.

Funding/Support: This work was funded by grant R01HS025146 from the Agency for Healthcare Research and Quality and was conducted with the support of the New York State Department of Health.

Role of the Funder/Sponsor: Neither the funder nor the policy sponsor had any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.

Disclaimer: Dr Angus is an Associate Editor of JAMA, but he was not involved in any of the decisions regarding review of the manuscript or its acceptance.

Additional Contributions: We gratefully acknowledge the support of Foster Gesten, MD, and Marcus Friedrich, MD, MBA (both with the New York State Department of Health Office of Quality and Patient Safety), for providing contextual information and policy details prior to the design of the analysis. No compensation was provided for this assistance.

References
1.
Singer  M, Deutschman  CS, Seymour  CW,  et al.  The third internatonal consensus definitions for sepsis and septic shock (Sepsis-3).  JAMA. 2016;315(8):801-810.PubMedGoogle ScholarCrossref
2.
Dellinger  RP, Levy  MM, Rhodes  A,  et al; Surviving Sepsis Campaign Guidelines Committee including the Pediatric Subgroup.  Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012.  Intensive Care Med. 2013;39(2):165-228. doi:10.1007/s00134-012-2769-8PubMedGoogle ScholarCrossref
3.
Levy  MM, Dellinger  RP, Townsend  SR,  et al.  The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis.  Intensive Care Med. 2010;36(2):222-231. doi:10.1007/s00134-009-1738-3PubMedGoogle ScholarCrossref
4.
Levy  MM, Rhodes  A, Phillips  GS,  et al.  Surviving Sepsis Campaign: association between performance metrics and outcomes in a 7.5-year study.  Crit Care Med. 2015;43(1):3-12. doi:10.1097/CCM.0000000000000723PubMedGoogle ScholarCrossref
5.
Rhodes  A, Phillips  G, Beale  R,  et al.  The Surviving Sepsis Campaign bundles and outcome: results from the International Multicentre Prevalence Study on Sepsis (the IMPreSS study).  Intensive Care Med. 2015;41(9):1620-1628. doi:10.1007/s00134-015-3906-yPubMedGoogle ScholarCrossref
6.
Cooke  CR, Iwashyna  TJ.  Sepsis mandates: improving inpatient care while advancing quality improvement.  JAMA. 2014;312(14):1397-1398. doi:10.1001/jama.2014.11350PubMedGoogle ScholarCrossref
7.
Hershey  TB, Kahn  JM.  State sepsis mandates—a new era for regulation of hospital quality.  N Engl J Med. 2017;376(24):2311-2313. doi:10.1056/NEJMp1611928PubMedGoogle ScholarCrossref
8.
Seymour  CW, Gesten  F, Prescott  HC,  et al.  Time to treatment and mortality during mandated emergency care for sepsis.  N Engl J Med. 2017;376(23):2235-2244. doi:10.1056/NEJMoa1703058PubMedGoogle ScholarCrossref
9.
Liu  VX, Fielding-Singh  V, Greene  JD,  et al.  The timing of early antibiotics and hospital mortality in sepsis.  Am J Respir Crit Care Med. 2017;196(7):856-863. doi:10.1164/rccm.201609-1848OCPubMedGoogle ScholarCrossref
10.
Yealy  DM, Kellum  JA, Huang  DT,  et al; ProCESS Investigators.  A randomized trial of protocol-based care for early septic shock.  N Engl J Med. 2014;370(18):1683-1693. doi:10.1056/NEJMoa1401602PubMedGoogle ScholarCrossref
11.
Rhee  C, Gohil  S, Klompas  M.  Regulatory mandates for sepsis care—reasons for caution.  N Engl J Med. 2014;370(18):1673-1676. doi:10.1056/NEJMp1400276PubMedGoogle ScholarCrossref
12.
Dimick  JB, Ryan  AM.  Methods for evaluating changes in health care policy: the difference-in-differences approach.  JAMA. 2014;312(22):2401-2402. doi:10.1001/jama.2014.16153PubMedGoogle ScholarCrossref
13.
Kahn  JM. Quantitative evaluation of the effects the 2013 New York State sepsis regulations. https://osf.io/jcwdv/. Accessed May 18, 2018.
14.
Dombrovskiy  VY, Martin  AA, Sunderram  J, Paz  HL.  Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003.  Crit Care Med. 2007;35(5):1244-1250. doi:10.1097/01.CCM.0000261890.41311.E9PubMedGoogle ScholarCrossref
15.
Iwashyna  TJ, Odden  A, Rohde  J,  et al.  Identifying patients with severe sepsis using administrative claims: patient-level validation of the angus implementation of the international consensus conference definition of severe sepsis.  Med Care. 2014;52(6):e39-e43. doi:10.1097/MLR.0b013e318268ac86PubMedGoogle ScholarCrossref
16.
Prescott  HC, Cope  TM, Gesten  FC,  et al.  Reporting of sepsis cases for performance measurement versus for reimbursement in New York State.  Crit Care Med. 2018;46(5):666-673. doi:10.1097/CCM.0000000000003005PubMedGoogle ScholarCrossref
17.
Angus  DC, Barnato  AE, Bell  D,  et al.  A systematic review and meta-analysis of early goal-directed therapy for septic shock: the ARISE, ProCESS and ProMISe Investigators.  Intensive Care Med. 2015;41(9):1549-1560. doi:10.1007/s00134-015-3822-1PubMedGoogle ScholarCrossref
18.
Rowan  KM, Angus  DC, Bailey  M,  et al; PRISM Investigators.  Early, goal-directed therapy for septic shock—a patient-level meta-analysis.  N Engl J Med. 2017;376(23):2223-2234. doi:10.1056/NEJMoa1701380PubMedGoogle ScholarCrossref
19.
Darby  JL, Davis  BS, Barbash  IJ, Kahn  JM.  An administrative model for benchmarking hospitals on their 30-day sepsis mortality.  BMC Health Serv Res. 2019;19(1):221. doi:10.1186/s12913-019-4037-xPubMedGoogle ScholarCrossref
20.
Elias  KM, Moromizato  T, Gibbons  FK, Christopher  KB.  Derivation and validation of the acute organ failure score to predict outcome in critically ill patients: a cohort study.  Crit Care Med. 2015;43(4):856-864. doi:10.1097/CCM.0000000000000858PubMedGoogle ScholarCrossref
21.
Ames  SG, Davis  BS, Angus  DC, Carcillo  JA, Kahn  JM.  Hospital variation in risk-adjusted pediatric sepsis mortality.  Pediatr Crit Care Med. 2018;19(5):390-396. doi:10.1097/PCC.0000000000001502PubMedGoogle ScholarCrossref
22.
Quan  H, Sundararajan  V, Halfon  P,  et al.  Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.  Med Care. 2005;43(11):1130-1139. doi:10.1097/01.mlr.0000182534.19832.83PubMedGoogle ScholarCrossref
23.
Mayr  FB, Yende  S, Linde-Zwirble  WT,  et al.  Infection rate and acute organ dysfunction risk as explanations for racial differences in severe sepsis.  JAMA. 2010;303(24):2495-2503. doi:10.1001/jama.2010.851PubMedGoogle ScholarCrossref
24.
Abadie  A.  Semiparametric difference-in-differences estimators.  Rev Econ Stud. 2005;72(1):1-19. doi:10.1111/0034-6527.00321Google ScholarCrossref
25.
Imbens  GW, Wooldridge  JM.  Recent developments in the econometrics of program evaluation.  J Econ Lit. 2009;47(1):5-86. doi:10.1257/jel.47.1.5Google ScholarCrossref
26.
Jacob  R, Somers  M-A, Zhu  P, Bloom  H.  The validity of the comparative interrupted time series design for evaluating the effect of school-level interventions.  Eval Rev. 2016;40(3):167-198. doi:10.1177/0193841X16663414PubMedGoogle ScholarCrossref
27.
St Clair  T, Hallberg  K, Cook  TD.  The validity and precision of the comparative interrupted time-series design: three within-study comparisons.  J Educ Behav Stat. 2016;41(3):269-299. doi:10.3102/1076998616636854Google ScholarCrossref
28.
Danai  PA, Sinha  S, Moss  M, Haber  MJ, Martin  GS.  Seasonal variation in the epidemiology of sepsis.  Crit Care Med. 2007;35(2):410-415. doi:10.1097/01.CCM.0000253405.17038.43PubMedGoogle ScholarCrossref
29.
Kaukonen  K-M, Bailey  M, Suzuki  S, Pilcher  D, Bellomo  R.  Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012.  JAMA. 2014;311(13):1308-1316. doi:10.1001/jama.2014.2637PubMedGoogle ScholarCrossref
30.
Greater New York Hospital Association. STOP sepsis. https://www.gnyha.org/topic/sepsis/. Accessed May 18, 2018.
31.
Levy  MM, Gesten  FC, Phillips  GS,  et al.  Mortality changes associated with mandated public reporting for sepsis: results of the New York State initiative.  Am J Respir Crit Care Med. 2018;198(11):1406-1412. doi:10.1164/rccm.201712-2545OCPubMedGoogle ScholarCrossref
32.
Rhee  C, Dantes  R, Epstein  L,  et al; CDC Prevention Epicenter Program.  Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.  JAMA. 2017;318(13):1241-1249. doi:10.1001/jama.2017.13836PubMedGoogle ScholarCrossref
33.
Thomas  L, Peterson  ED.  The value of statistical analysis plans in observational research: defining high-quality research from the start.  JAMA. 2012;308(8):773-774. doi:10.1001/jama.2012.9502PubMedGoogle ScholarCrossref
×