A, Unplanned transfers to the intensive care unit (ICU). B, Critical deterioration. C, Mechanical ventilation within 12 hours after transfer to the ICU. D, Vasopressor infusion within 12 hours after transfer to the ICU. The vertical lines at month 0 (February 2010) indicate the month of rapid response system (RRS) implementation.
Kaplan-Meier failure plots of time to mechanical ventilation or vasopressor infusion among all unplanned transfers to the intensive care unit (ICU) (hazard ratio = 0.79; 95% CI, 0.63-0.995; P = .046) (A) and restricted to transfers with mechanical ventilation or vasopressors (hazard ratio [over first 4 ICU hours] = 0.72; 95% CI, 0.55-0.93; P = .01) (B).
Customize your JAMA Network experience by selecting one or more topics from the list below.
Bonafide CP, Localio AR, Roberts KE, Nadkarni VM, Weirich CM, Keren R. Impact of Rapid Response System Implementation on Critical Deterioration Events in Children. JAMA Pediatr. 2014;168(1):25–33. doi:10.1001/jamapediatrics.2013.3266
Rapid response systems aim to identify and rescue deteriorating hospitalized patients. Previous pediatric rapid response system implementation studies have shown variable effectiveness in preventing rare, catastrophic outcomes such as cardiac arrest and death.
To evaluate the impact of pediatric rapid response system implementation inclusive of a medical emergency team and an early warning score on critical deterioration, a proximate outcome defined as unplanned transfer to the intensive care unit with noninvasive or invasive mechanical ventilation or vasopressor infusion in the 12 hours after transfer.
Design, Setting, and Participants
Quasi-experimental study with interrupted time series analysis using piecewise regression. At an urban, tertiary care children’s hospital in the United States, we evaluated 1810 unplanned transfers from the general medical and surgical wards to the pediatric and neonatal intensive care units that occurred during 370 504 non–intensive care patient-days between July 1, 2007, and May 31, 2012.
Implementation of a hospital-wide rapid response system inclusive of a medical emergency team and an early warning score in February 2010.
Main Outcomes and Measures
Rate of critical deterioration events, adjusted for season, ward, and case mix.
Rapid response system implementation was associated with a significant downward change in the preintervention trajectory of critical deterioration and a 62% net decrease relative to the preintervention trend (adjusted incidence rate ratio = 0.38; 95% CI, 0.20-0.75). We observed absolute reductions in ward cardiac arrests (from 0.03 to 0.01 per 1000 non–intensive care patient-days) and deaths during ward emergencies (from 0.01 to 0.00 per 1000 non–intensive care patient-days), but these were not statistically significant (P = .21 and P = .99, respectively). Among all unplanned transfers, critical deterioration was associated with a 4.97-fold increased risk of death (95% CI, 3.33-7.40; P < .001).
Conclusions and Relevance
Rapid response system implementation reversed an increasing trend of critical deterioration. Cardiac arrest and death were extremely rare at baseline, and their reductions were not statistically significant despite using nearly 5 years of data. Hospitals seeking to measure rapid response system performance may consider using valid proximate outcomes like critical deterioration in addition to rare, catastrophic outcomes.
Rapid response systems (RRSs) aim to identify hospitalized patients exhibiting signs of deterioration and intervene before respiratory or cardiac arrest occurs. These include afferent and efferent limbs. The role of the afferent limb is to identify deteriorating patients using tools such as early warning scores (EWSs).1 The role of the efferent limb is to deploy medical emergency teams (METs) to hospital wards to rescue deteriorating patients.1
The effectiveness of RRSs in improving patient safety is controversial.2 A key limitation of prior pediatric RRS research is that the outcome measures traditionally evaluated—respiratory arrests, cardiac arrests, and deaths—are exceedingly rare.
The critical deterioration (CD) metric, defined as transfer to the intensive care unit (ICU) followed by noninvasive or invasive mechanical ventilation or vasopressor infusion within 12 hours, has been shown to be a valid proximate outcome for evaluating RRS performance.3 We previously demonstrated that CD was more than 8 times more common than respiratory and cardiac arrests and was associated with a more than 13-fold increased risk of in-hospital death.3
In this study, we aim to evaluate the impact of RRS implementation inclusive of an EWS and an MET on CD as well as rare, catastrophic outcomes used more commonly in RRS studies.
We performed this quasi-experimental study at The Children’s Hospital of Philadelphia between July 1, 2007, and May 31, 2012. The Children’s Hospital of Philadelphia is an urban, tertiary care children’s hospital with 516 beds, including a 55-bed pediatric ICU, an 83-bed neonatal ICU, and a 24-bed cardiac ICU. On February 8, 2010, the hospital implemented an RRS that served all acute care wards except the cardiology, tracheostomy-ventilator, and obstetric units and the ICUs. The RRS consisted of (1) an EWS based on Parshuram and colleagues’ Bedside Pediatric Early Warning System4-6 with corresponding escalation guidelines, and (2) a 30-minute response MET available 24 hours per day, 7 days per week. The MET could be activated by any clinician for any clinical concern, regardless of the EWS.
The MET was composed of 3 responding pediatric ICU clinicians: (1) a fellow, attending, or nurse practitioner, (2) a nurse, and (3) a respiratory therapist. Following an MET activation, the patient could be transferred to the pediatric or neonatal ICU or remain on the ward. For urgent concerns that could not wait 30 minutes, the immediate-response code blue team remained available. Prior to RRS implementation, physicians could page the pediatric ICU fellow to discuss patients who did not require the code blue team but were clinically deteriorating.
The Children’s Hospital of Philadelphia Institutional Review Board approved the study. A waiver of consent, assent, and parental permission was approved per 45 CFR §46.116(d) and a waiver of Health Insurance Portability and Accountability Act of 1996 authorization was approved per 45 CFR §164.512(i)(2)(ii) by The Children’s Hospital of Philadelphia Institutional Review Board.
We identified transfers from acute care wards to the pediatric or neonatal ICU using the electronic health record (Epic Systems). We did not include transfers from areas not served by the RRS. We reviewed each medical record and excluded transfers that did not meet unplanned criteria: those that were elective or planned in advance, those coming directly from the operating room or sleep laboratory, and those in which the need for ICU care was attributed to the need to recover from sedation. Determination of unplanned transfers was performed by 2 research assistants with experience in the nursing field. Before collecting data, both research assistants reviewed the same 50 medical records independently and achieved 94% agreement (κ = 0.88).
For the transfers categorized as unplanned, we reviewed ICU flow sheets for the 12 hours after transfer and recorded the time from ICU arrival to life-sustaining interventions, including initiation of noninvasive ventilation (continuous or bilevel positive airway pressure), invasive mechanical ventilation via endotracheal tube or tracheostomy, and vasopressor infusion (ie, dopamine, dobutamine, epinephrine, norepinephrine, phenylephrine, isoproterenol, or milrinone). We classified events requiring any of these interventions in the first 12 hours after ICU transfer as CD.3 In addition, any patients who died during an emergency response on the ward before they could be transferred to the ICU experienced CD. We calculated interrater reliability for CD with 50 medical records using the methods described earlier and achieved 100% agreement (κ = 1.00).
We obtained frequencies of code blue calls, MET calls, respiratory arrests, cardiac arrests, and deaths occurring on general medical and surgical wards from the hospital’s Resuscitation Committee database. We defined respiratory arrests as emergent intubations with subsequent transfer to ICU or death on the ward, and we defined cardiac arrests as events with documented pulselessness or a pulse with inadequate perfusion receiving chest compressions and/or defibrillation, with subsequent transfer to ICU or death on the ward. We obtained the Pediatric Risk of Mortality III score at 12 hours (PRISM III-12)7 from the hospital’s Critical Care Center for Evidence and Outcomes database.
We considered July 1, 2007, through February 28, 2010, to be the preimplementation period (numbered as months −31 through 0) and March 1, 2010, through May 31, 2012, to be the postimplementation period (numbered as months +1 through +27). For unadjusted rates, we calculated Poisson confidence intervals and compared preimplementation and postimplementation using Poisson regression.
We then performed an interrupted time series analysis using segmented (piecewise) regression to evaluate dynamic changes in outcomes following the RRS implementation while controlling for secular trends.8,9 We used multivariable generalized estimating equations models with the negative binomial distribution10,11 to evaluate the effect of RRS implementation. We evaluated additional models with quadratic terms and found that these terms did not approach statistical significance; therefore, we did not include them in our final model. We adjusted for season, ward, and case-mix index. As in similar studies,12 case mix was based on the US Centers for Medicare and Medicaid Services cost weights and assessed monthly for the hospital.
We used the regression model to ask the following questions: (1) What was the preimplementation rate trajectory? (2) Was there an immediate change in the rate immediately following implementation? (3) What was the postimplementation rate trajectory? (4) Were the preimplementation and postimplementation rate trajectories significantly different? and (5) What was the net effect of the intervention, estimated as the adjusted difference between the fitted rate at the end of the postintervention period and the expected rate if the preintervention trajectory had continued, uninterrupted by the intervention (expressed as an adjusted incidence rate ratio [IRR])?13
To evaluate the stability of our results, we performed a sensitivity analysis evaluating the impact of censoring 3 months (months +1 through +3) and 6 months (months +1 through +6) immediately following implementation on the net effect for the primary outcome, CD.
As a secondary analysis, we also evaluated the time to life-sustaining interventions in the ICU among unplanned transfers using a Kaplan-Meier failure plot and calculated the hazard ratio between groups. We tested the equality of the time to vasopressors or mechanical ventilation between the preimplementation and postimplementation periods using Cox regression.14
To revalidate the CD metric using a different comparison group than in the original study,3 among unplanned transfers to the ICU we calculated the relative risk of death prior to hospital discharge comparing admissions with CD with those without CD.
We managed the data using Research Electronic Data Capture (REDCap).15 We analyzed the data using Stata version 12.1 statistical software (StataCorp LP).
During the 59-month study period, there were 1810 unplanned transfers from the wards to the pediatric and neonatal ICUs and 370 504 non-ICU patient-days. The patients were similar in age, sex, transferring ward, and destination ICU between the preimplementation and postimplementation periods (Table 1). The PRISM III-12 scores were available in 1558 (86%) of the transfers; the median preimplementation score was 2, and the median postimplementation score was 0 (Table 1).
In the preimplementation period, there were 102 code blue team activations (0.53 per 1000 non-ICU patient-days). In the postimplementation period, there were 115 code blue team activations (0.65 per 1000 non-ICU patient-days) (P = .15 for the preimplementation vs postimplementation difference) and 1534 MET activations (8.61 per 1000 non-ICU patient-days), for a combined utilization rate of 9.26 per 1000 non-ICU patient-days or 24.41 per 1000 all-hospital admissions.
We observed reductions in unadjusted rates of ward cardiac arrests and deaths during ward emergencies after RRS implementation, but these reductions did not reach statistical significance (Table 2). There were no significant differences in the rates of respiratory arrests on the wards or mortality prior to discharge among those patients transferred to the ICU between the preimplementation and postimplementation periods (Table 2). Owing to their low rates of occurrence, performing an adjusted analysis of traditional clinical outcomes was not feasible without risk of overfitting.16
In unadjusted analysis, the rate of unplanned transfers to the ICU was significantly higher in the postimplementation period than in the preimplementation period (Table 2). In the adjusted interrupted time series model, RRS implementation was not associated with a change in the transfer rate trajectory and had no net difference in the transfer rate relative to the preintervention trend (Figure 1 and Table 3).
In unadjusted analysis, RRS implementation was not associated with a significant difference in the rate of CD (Table 2). In the adjusted interrupted time series model, RRS implementation was associated with a significant downward change in the CD rate trajectory and a net reduction in events by 62% relative to the preintervention trend (IRR = 0.38; 95% CI, 0.20-0.75) (Figure 1 and Table 3).
In unadjusted analysis, RRS implementation was not associated with significant differences in mechanical ventilation use following transfer to the ICU (Table 2). In the adjusted interrupted time series model, RRS implementation was associated with a significant downward change in the trajectory of mechanical ventilation use in the 12 hours following transfer to the ICU and a net reduction in events by 83% relative to the preintervention trend (IRR = 0.17; 95% CI, 0.07-0.44) (Figure 1 and Table 3).
In unadjusted analysis, RRS implementation was associated with a reduction in the proportion of unplanned transfers requiring vasopressors in the first 1 hour following transfer but not in the first 12 hours (Table 2). In addition, of the patients with unplanned transfers who required vasopressors in the first 12 hours following ICU transfer, significantly fewer required them in the first 1 hour after transfer (Table 2). In the adjusted interrupted time series model, RRS implementation was associated with a significant downward change in the trajectory of vasopressor use in the 12 hours following transfer to the ICU and a net reduction in events by 80% relative to the preintervention trend (IRR = 0.20; 95% CI, 0.06-0.62) (Figure 1 and Table 3).
Censoring 3 or 6 months following implementation had minimal impact on the results. A 3-month censored period changed the IRR for the net intervention effect on CD from 0.38 to 0.41 (95% CI, 0.22-0.76; P = .005). A 6-month censored period changed the IRR to 0.40 (95% CI, 0.20-0.79; P = .008).
Implementation of the RRS was associated with a longer interval from ICU arrival to vasopressor administration or mechanical ventilation. Figure 2A illustrates that this finding existed among all patients transferred to the ICU (hazard ratio = 0.79; 95% CI, 0.63-0.995; P = .046), primarily because fewer transferred patients required these life-sustaining interventions in the first 12 ICU hours. This finding persisted when the sample was restricted to patients who did require vasopressor administration or mechanical ventilation in the first 12 ICU hours. In this case, after RRS implementation, we found that fewer patients required these interventions emergently as evidenced by the more gradual initial slope of the postimplementation Kaplan-Meier failure plot over the first 4 ICU hours (hazard ratio = 0.72; 95% CI, 0.55-0.93; P = .01) (Figure 2B).
Among all 1681 admissions during the preimplementation and postimplementation periods with unplanned transfer to the ICU, we found that CD during the admission was associated with a 4.97-fold increased risk of death (95% CI, 3.33-7.40; P < .001).
This study found that implementation of an RRS inclusive of an EWS and an MET was associated with a significant downward change in the CD rate trajectory as well as a reduction in its rate relative to its preintervention trend. We did not see significant reductions in traditional metrics used to evaluate RRSs—respiratory arrests, cardiac arrests, and deaths—probably because these outcomes were already rare prior to RRS implementation. While the change in the trajectory of CD was apparent graphically (Figure 1), detecting this improvement mathematically required modeling the trajectory over time and adjusting for covariates. If we had simply compared the unadjusted preintervention and postintervention rates shown in Table 2, we might have erroneously concluded that there was no significant impact or that CD rates might have worsened as a result of implementation. Given the significant change in trajectory, we suspect that unadjusted postimplementation CD and traditional metric rates will become significantly lower over time.
When we evaluated the time to life-sustaining interventions, we found that after RRS implementation, fewer patients needed emergent vasopressors and mechanical ventilation in the first 4 hours in the ICU, even among those who eventually needed them by ICU hour 12. Similarly, we found that the postimplementation transfers may have been at a lower mortality risk than preimplementation transfers as measured by the median PRISM III-12 score, although this difference was not statistically significant. These findings have face validity; it is likely that implementation of the EWS improved recognition of shock and respiratory insufficiency, appropriately triggering the MET to rescue these patients earlier and allowing for a smoother transition to the ICU.
We also revalidated the association between CD and death. As expected, the strength of this 5-fold association was smaller than the 13-fold association observed in our original study3 because in this study we compared admissions with CD vs admissions with unplanned ICU transfer that did not meet CD criteria, whereas in the previous study we compared admissions with CD vs other admissions with any MET event, including those that did not even result in ICU transfer. Therefore, one would expect the relative risk to be smaller in this study’s comparison.
We evaluated the impact of simultaneous EWS and MET implementation. Seven prior studies have evaluated the impact of MET implementation without EWSs in children’s hospitals.12,17-22 The studies are all single center without adjustment for other key variables, with the exceptions of the study by Sharek et al12 that, like this study, used interrupted time series modeling and adjusted for season and case mix and the multicenter study by Kotsakis et al.22 Two of the 7 studies showed no impact on arrest or mortality rates.18,21 One showed an impact on mortality only in a group of patients readmitted to the pediatric ICU.22 Two demonstrated reductions in hospital-wide mortality,12,20 2 demonstrated reductions in combined respiratory and cardiac arrests outside the ICU,12,17 and 1 demonstrated only a reduction in respiratory arrests outside the ICU.19 In a recent meta-analysis, implementation of a pediatric MET was associated with a 38% reduction in cardiac arrests outside the ICU and a 21% reduction in hospital mortality. However, this finding was not robust to sensitivity analysis and was greatly influenced by studies in which mortality reduction could not be explained by MET interventions.23
We chose to report the mortality rate only among unplanned transfers to the ICU rather than hospital-wide mortality because the majority of in-hospital pediatric deaths are not plausibly preventable by RRSs.23,24 For example, RRSs cannot prevent deaths among patients who never receive care outside an ICU, such as 500-g infants, children immediately following cardiac surgery, or major trauma victims. They should be excluded from analyses of the impact of RRSs.
Addressing the challenges of demonstrating improvements in exceedingly rare events attributable to RRS implementation, 2 other proximate outcomes have been developed. Significant clinical deterioration events defined using the Children’s Resuscitation Intensity Scale25 are interhospital transfers to a pediatric ICU following any of these in the 12 hours prior to transfer: invasive ventilation, more than 60 mL/kg of intravenous fluid, vasopressors, cardiopulmonary resuscitation, or death. In one study, the rate of significant clinical deterioration events decreased following implementation of an EWS, although this included only 2 events in the preintervention period and 1 event in the postintervention period.25 Unrecognized Situation Awareness Failure Events (UNSAFE transfers) are defined as intrahospital ward to ICU transfers resulting in intubation, vasopressors, or at least 3 fluid boluses in the first 1 hour after transfer.26 In one study, reductions in the UNSAFE transfer rate were observed following improvements to the RRS afferent arm.26 Head-to-head comparison of these 2 alternative metrics with CD and validation in a multicenter cohort are needed.
This study has several limitations. First, this is a single-center quasi-experimental study with historical controls. These results occurred in a unique milieu of concurrent hospital safety initiatives that cannot be fully adjusted for, including work to reduce central line infections and implementation of a new electronic health record. While a large multicenter cluster-randomized trial would be the ideal design to determine the effect of pediatric RRS implementation, given their nearly universal presence, identifying an adequate number of eligible sites is not likely.27 Therefore, we must perform studies using longitudinal observational designs28,29 to determine the relationship between RRSs and clinical outcomes. Second, our estimation of the net effect is based on the difference between the fitted rate at the end of the postintervention period and the expected rate if the preintervention trajectory had continued. It is possible that the statistical model might not have captured an attenuation or ceiling effect of the increasing preintervention trajectory; therefore, the modeled net effect might be higher than would be observed clinically. Third, our key finding was that there was a clear change in the rate trajectory following RRS implementation. We do not know why the trajectory of CD was trending upward during the preintervention period. Fourth, we did not include cardiac patients because ward-specific mechanisms that are separate from the RRS exist to recognize deterioration and rescue these children, most of whom are recovering from cardiac surgery. Future studies should evaluate this population.
Implementation of the RRS resulted in improved patient safety outcomes. Detecting this improvement required using interrupted time series methods to evaluate proximate outcome measures. This study adds to the evidence supporting RRS implementation in children’s hospitals. Further, our results suggest that longitudinal evaluation and optimization of RRS performance might focus on incrementally reducing more common proximate outcomes like CD while continuing to measure rare, catastrophic outcomes.
Corresponding Author: Christopher P. Bonafide, MD, MSCE, The Children’s Hospital of Philadelphia, 34th Street and Civic Center Boulevard, Ste 12NW80, Philadelphia, PA 19104 (firstname.lastname@example.org).
Accepted for Publication: June 20, 2013.
Published Online: November 11, 2013. doi:10.1001/jamapediatrics.2013.3266.
Author Contributions: Dr Bonafide had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Bonafide, Localio, Roberts, Nadkarni, Keren.
Acquisition of data: Bonafide, Roberts, Weirich.
Analysis and interpretation of data: All authors.
Drafting of the manuscript: Bonafide, Localio, Roberts, Nadkarni.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Bonafide, Localio, Roberts, Keren.
Obtained funding: Bonafide, Keren.
Administrative, technical, or material support: Nadkarni, Weirich, Keren.
Study supervision: Localio, Nadkarni, Keren.
Conflict of Interest Disclosures: None reported.
Funding/Support: This work was supported by the Health Research Formula Fund Award from the Pennsylvania Department of Health (Dr Keren).
Role of the Sponsor: The funding organization had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Additional Contributions: Donna Czaplicki, RN, BSN, assisted with medical record review and Sherri Kubis, RN, Judy Verger, RN, PhD, Rich Lin, MD, Akira Nishisaki, MD, MSCE, and the staff of the Critical Care Center for Evidence and Outcomes, The Children’s Hospital of Philadelphia assisted with data queries.
Create a personal account or sign in to: