Reduction in Hospital Mortality Over Time in a Hospital Without a Pediatric Medical Emergency Team: Limitations of Before-and-After Study Designs

Objective  To determine whether hospital mortality has decreased over time in a hospital that has not introduced a pediatric medical emergency team (PMET).

Design  Retrospective observational study.

Setting  Quaternary children's hospital.

Participants  All pediatric inpatient separations (defined as any discharge, including death) during 10 fiscal years.

Main Outcome Measures  We searched our hospital administrative database to determine the number of pediatric inpatient separations and deaths, and we searched the hospital switchboard and pediatric intensive care databases to determine ward code and cardiopulmonary arrest rates. Relative risks (RRs) with 95% confidence intervals (CIs) and logistic regression compared results over time.

Results  During the periods of the 2 PMET studies showing a reduction in hospital mortality, we found a decrease in hospital mortality: for 1999-2002 vs 2002-2006, 212 deaths among 14 161 patients (1.50%) vs 219 of 26 767 (0.82%), RR, 0.55 (95% CI, 0.44-0.69); for 2000-2005 vs 2005-2007, 300 deaths among 29 497 patients (1.02%) vs 98 of 14 005 (0.70%), RR, 0.69 (95% CI, 0.55-0.86). During the periods of the 3 PMET studies showing no change in or not examining hospital mortality, we found no significant change in hospital mortality. The annual odds ratio for survival was 1.13 (95% CI, 1.09-1.16). There were no changes in ward code and cardiopulmonary arrest rates over time.

Conclusions  We found a reduction in hospital mortality over time in a children's hospital without a PMET. This demonstrates the limitation of before-and-after study designs, and we hypothesize that multiple cointerventions account for the decrease in mortality. Whether a PMET could have reduced mortality further is unknown.

Medical emergency teams (METs) have been advocated “as a means to decrease in-hospital cardiopulmonary arrest (CPA) and mortality.”^1(p2311) Published studies advocate use of a pediatric MET (PMET) to “prevent unexpected in-hospital ward cardiac arrest with subsequent death,”^{2(p309),3(p1148)} “decrease inpatient mortality rates,”^4(p2268) “be called before a patient has a CPA in hopes of improving outcomes,”^5(p117) “reduce the rate of codes outside the intensive care unit . . . [and] rescue patients and reduce hospital mortality rates,”^6(p237) or “intervene before an arrest occurs.”^7(p418)Previous studies⁶ report that 8.5% to 14% of all in-hospital CPAs occur outside the pediatric intensive care unit (PICU); once a CPA occurs, the mortality is 50% to 67%.

We suggest that there are at least 4 “links” in the “chain” of patient rescue from hospital CPA and mortality, as follows: (1) symptoms and signs of deterioration (“calling criteria”) occur and are recognized by ward personnel; (2) the ward personnel are empowered to call for assistance without delay, circumventing established hierarchies if necessary; (3) a readily available urgent response occurs; and (4) this response, along with associated interventions, improves outcomes. To improve the collective chain, there have been several before-and-after studies of PMET implementation with inconsistent effects on hospital mortality and CPA rates (eTable 1).^2-7 Nevertheless, the data from these cohort studies using historical controls have been considered sufficient to recommend implementation of a PMET at pediatric hospitals.¹

We hypothesized that favorable results in PMET studies may be due to limitations of study design, including the use of historical controls, inadequate risk and temporal trend adjustment, and inadequate accounting for cointerventions that confound the comparisons. To test this hypothesis, we obtained administrative data to determine whether the temporal differences in hospital mortality and ward CPA rates found in some PMET studies also occurred at our quaternary children's hospital, which has not implemented a PMET.

The published PMET studies were reviewed to determine the comparability of our single-center hospital and our study design with (1) the single-center hospitals that have published their experience with PMET; (2) the type of analysis performed in the PMET studies, including years of data collection and adjustment for case mix, severity of illness, and temporal trends; and (3) the primary outcomes that demonstrate the efficacy of PMET in the published studies.

Administrative data were obtained from the following:

• A hospital patient admission database to determine the number of pediatric inpatient separations (defined as any discharge, including death) and deaths, including and excluding patients in the neonatal intensive care unit (ICU), for each fiscal year (April 1 through March 31) from 1999-2000 through 2008-2009.

• A hospital switchboard database of code calls to determine the number of pediatric codes called each fiscal year with data available from 2001-2002 through 2008-2009.

• A PICU database (PICU evaluation) to determine the number of pediatric inpatient ward CPA events (defined as an event requiring chest compression) that resulted in admission to the PICU during each fiscal year with data available from 2002-2003 through 2007-2008. At our center, all ward patients who experience CPA with return of spontaneous circulation are admitted to the PICU, and most patients without return of spontaneous circulation during the first few minutes of chest compressions are also taken to the PICU for ongoing resuscitation and possible extracorporeal cardiopulmonary resuscitation (CPR). We cross-checked data with the minutes from our PICU code committee meetings from April 1, 2004, through March 31, 2009, and did not find any missed ward CPA events. We excluded CPA events occurring before arrival at our hospital, in the emergency department, in the operating room, or in the postanesthetic recovery room.

The primary outcomes were to compare hospital mortality, ward CPA, and code call rates during periods as similar as possible to the before-and-after periods used in the published PMET studies. Secondary outcomes were to compare our hospital mortality, ward CPA, and code call rates during 2 equal periods and using annual trends.

Our hospital is a quaternary referral children's hospital serving a population of 1.7 million, with 134 inpatient beds, including 18 mixed medical-surgical-cardiac PICU, 13 neonatal ICU, and 15 high-dependency unit beds. In 2008-2009, there were 6984 patient separations. We maintain active heart and solid-organ transplant services, neurosurgery and cardiac surgery (>400 cases annually) programs, and an extracorporeal life support program that includes extracorporeal CPR, and we have a full complement of pediatric medical and surgical subspecialties. The PICU has 24-hour coverage by 8 pediatric intensivists. Although intensivists are not required to be in-house overnight, a PICU fellow often is. The code team responds to emergencies on the wards and is composed of a pediatric intensivist (if in-house at the time), a PICU fellow and/or resident, a PICU respiratory therapist, and 2 PICU nurses. In addition, PICU consultations (requested by ward residents or medical staff) are seen by the PICU fellow and/or resident and reviewed with the intensivist. The chair of the health research ethics board of our university approved this project as a quality improvement study for publication.

We compared the data between study periods by presenting relative risks and 95% confidence intervals. We compared annual mortality rates using logistic regression and present odds ratios and 95% confidence intervals.

The results of the review of published PMET studies are shown in eTable 1.^2-7 The results suggest that our hospital is similar to the hospitals in the published PMET studies, that the administrative data on hospital mortality and CPAs are from years similar to those used in the PMET studies, and that lack of adjustment for case mix, severity of illness, and temporal trends in our study design is similar to the published PMET studies.

The results for hospital mortality are shown in Table 1. During the periods of the 2 PMET studies^2,4 that found a significant reduction in hospital mortality between the before and after periods, we also found a highly significant reduction in mortality at our hospital. During the periods of the 3 PMET studies^5-7 that did not find a significant reduction in hospital mortality between the before and after periods, we also did not find a significant change in mortality at our hospital. When we divided our data into equal 4- and 5-year periods, there was a highly significant reduction in hospital mortality in the after periods compared with the before periods at our hospital. By logistic regression, the annual odds ratio for survival was 1.13 (95% confidence interval, 1.09-1.16).

There was no statistically significant change in CPA rates during the periods similar to those used in the published PMET studies or when we divided our data into equal 3-year periods (Table 2). This finding is similar to those of the published PMET studies,^2,3,5,6 with none reporting a statistically significant change in ward CPA rates (eTable 1). We did not review the CPA events to determine mortality from these events or to investigate whether they might have been determined retrospectively to be preventable. The published PMET studies^3,5,6 have not shown a significant reduction in hospital mortality from ward CPA events. One center^2,3 found a statistically significant reduction in preventable ward CPAs with a statistically significant increase in nonpreventable ward CPAs, while another center⁶ did not find a statistically significant reduction in preventable ward codes (eTable 1).

The rate of calls to our code team each year from 2001-2009 showed no change during the periods similar to those used in the published PMET studies or when we divided our data into 2 equal 4-year periods (Table 3). Two PMET centers^2,3,5 reported this outcome, and both found an increase in calls to their single-tier PMET compared with the before period, when they had only a code team (eTable 1). Our code team call rate was 0.43% of hospital admissions during 2005-2009; this rate is similar to the PMET call rate of Brilli et al⁶ (0.32%) and of Tibballs and Kinney² and Tibballs et al³ (0.51%-0.69%) but lower than the rate found by the other 3 centers^4,5,7 (1.17%-1.97%) (eTable 1).

The published PMET studies,^1-7 despite some inconsistent results (eTable 1), have been used to suggest that hospitals should implement PMETs. In our children's hospital, we demonstrated statistically significant decreases in hospital mortality rate during the periods of the 2 PMET studies^2,4 reporting this positive outcome, when our hospital data were divided into equal 4- and 5-year periods, and when our hospital data were analyzed annually by logistic regression. We found no significant change over time in the ward CPA rate, similar to the 4 PMET studies^2,3,5,6 reporting this outcome. We also found no change in the rate of code team calls during these periods. These results call into question the strength of the PMET data from the cohort studies with historical controls.

There are several potential reasons implementation of a PMET may not improve hospital outcomes. If, as we suggest, there are at least 4 complex links in the chain of patient rescue, there could be 1 or more links for which a PMET is ineffective. Each PMET study used different calling criteria based on expert opinion with unknown sensitivity and specificity. Whether ward personnel used these criteria or instead used a more subjective assessment (as suggested by 2 of the studies^6,7) is unclear. If a more subjective assessment was used, then the problem remains whether ward personnel can recognize the clinically deteriorating patient in time. The highly variable PMET calling rates in the published studies (eTable 1)—all less than the number of calls expected from current pediatric early warning scores^8,9—suggest that the PMET is not always activated when triggers are present. This finding suggests that there could be weak links at recognition of clinical deterioration, at empowerment to call a PMET without delay, or both. In addition, the concept of PMETs assumes not only that recognizable clinical deterioration occurs before a CPA but also that intervention by a PMET during this clinical deterioration improves outcomes compared with a PICU consultation and/or a graded response by ward personnel. It is possible that many pediatric CPAs develop acutely and are not easily preventable.^5,6 Several PMET studies report no changes in the number of unexpected CPAs occurring in the hospital^2,3,5,6 or in hospital mortality from CPAs or codes^3,6; one center with a PMET^2,3 found an unexplained increase in non-PMET–preventable hospital CPAs. It is also possible that clinical deterioration may be due to a poor response to appropriate therapy rather than to a delay in or lack of available expertise on the wards. Finally, it is possible that there are better ways to improve weak links other than by implementation of a PMET, such as focusing on education, empowerment, respect, and availability of staff.

A systematic review¹⁰ of pediatric alert criteria for identifying hospitalized children at risk of critical deterioration found that the tools are unvalidated, are inconsistent, do not address the issue of recognition in time to intervene, and have poor positive predictive value. The authors called for “prospective evaluation of validity, reliability, and utility before widespread adoption into clinical practice.”^10(p600) Another systematic review¹¹ of METs found that the mortality benefit in PMET studies is 3 to 100 times higher than the number of CPAs averted. This finding suggests that the benefit reported in PMET studies is likely confounded by interventions unknown to the authors.

Several factors may explain the statistically significant reduction in our hospital mortality rate during the periods studied. Any study with historical controls carries a risk of confounding by known or unknown cointerventions. For example, as a result of largely unclear factors, mortality has improved recently for some conditions, such as acute respiratory distress syndrome¹² and hypoplastic left heart syndrome.¹³ The mortality of critically ill patients depends on many variables, including time of day at admission,¹⁴ having an in-house fellow at night,¹⁴ source of admission,¹⁵ intensivist staffing patterns,¹⁶ and surgical volume.¹⁷ Many cointerventions occurred at our hospital and at the hospitals that published their PMET experience (eTable 2). These or other unclear cointerventions may account for improved hospital mortality over time.

There are some strengths to our study. First, we used administrative data from existing databases that were collected prospectively and based on objective outcomes of mortality, ward CPAs requiring chest compressions, and code team calls. Second, our children's hospital is similar to most high-acuity referral hospitals with a PICU.^18,19 According to data from a 2005 North American survey¹⁸ of children's hospitals, our center is in the higher quartiles for the number of acute care, high-dependency unit, and PICU beds and is among those with PICUs that have care always directed by an intensivist (47%), offer extracorporeal life support (56%) or extracorporeal CPR (38%), and have urgent-response personnel (75%). In that survey, the urgent-response personnel were staff from the PICU in 51% of the hospitals, a code team in 8%, and a PMET in 16%. We acknowledge that our ward consultations and code calls may be considered a “functional PMET”; however, the survey results indicate that most hospitals have similar urgent-response personnel systems, supporting the claim that PMETs may not be necessary. According to data from a survey conducted in the United States in 2004,¹⁹ our center is in the higher quartiles for the number of PICU beds, pediatric intensivists, and pediatric intensivist in-house coverage hours and is similar to those having pediatric intensivist coverage 24 hours a day (71%). Therefore, our findings are likely to be similar to those of most children's hospitals in North America.

We believe there are 2 main explanations for our results and those of the PMET studies with positive outcomes: improving care over time due to unmeasured cointerventions and having mechanisms that make a PMET redundant. For example, code calls at most pediatric hospitals far outnumber the true ward CPA rate,¹⁸ allowing the code team to respond to clinical deteriorations other than CPAs. Our hospital's rate of code calls is similar to the rates in some published PMET studies^2,3,6 but much lower than others^4,5,7 for unclear reasons (eTable 1). The PICU ward consultations, which we estimate to number at least 1 a week, are similar to the rate of PMET calls in some studies.^4,5,7 We also estimate that the interventions performed when our code team is called for many codes or when consultations are done are similar to those described in the PMET studies (eTable 1).

There are some limitations to this study. First, a single-center study may not be generalizable. Second, we cannot identify what factors were responsible for our reduction in hospital mortality; however, we can say with certainty that it was not because of introduction of a PMET. If we had implemented a PMET, we would likely have misattributed the reduction in hospital mortality over time to this intervention. Third, we did not adjust for case-mix, severity of illness, or temporal trends between the periods examined. Nevertheless, this lack of adjustment is similar to the published PMET studies (eTable 1) and may explain the differences between periods. Fourth, we were unable to determine the number of respiratory arrests that occurred at our hospital. Finally, we have not demonstrated that (or even examined whether) a PMET is ineffective in reducing hospital mortality, ward CPA, or respiratory arrest rates. We believe we have shown that before-and-after study designs have limitations that make the published PMET studies unable to prove that PMET implementation improves outcomes. We recognize that some hospital deaths and CPAs are preventable; how to prevent these adverse outcomes requires further study.

We found a reduction in hospital mortality over time even though we have not introduced a PMET. We do not assign our reduction in mortality to absence of a PMET; rather, we claim that this finding demonstrates the limitation of before-and-after study designs (cohort studies with historical controls) in determining the effect of PMET implementation. A large cluster-randomized controlled trial of PMET implementation with validated calling criteria is required to determine whether PMETs are effective.

Correspondence: Ari R. Joffe, MD, Division of Pediatric Intensive Care, Department of Pediatrics, Room 3A3.07, Stollery Children's Hospital, 8440 112th St, Edmonton, AB T6G 2B7, Canada (ari.joffe@albertahealthservices.ca).

Accepted for Publication: September 16, 2010.

Author Contributions: Dr Joffe had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Joffe. Acquisition of data: Joffe. Analysis and interpretation of data: Joffe, Anton, and Burkholder. Drafting of the manuscript: Joffe. Critical revision of the manuscript for important intellectual content: Joffe, Anton, and Burkholder. Statistical analysis: Joffe. Administrative, technical, and material support: Joffe, Anton, and Burkholder.

Financial Disclosure: None reported.

Online-Only Material: This article is featured in the Archives Journal Club. Go here to download teaching PowerPoint slides.