[Skip to Content]
[Skip to Content Landing]
Figure 1.
Risk-Adjusted Rates of Outcomes in the Neonatal Intensive Care Unit at the 10th, 25th, 50th, 75th, and 90th Percentiles, 2005-2014
Risk-Adjusted Rates of Outcomes in the Neonatal Intensive Care Unit at the 10th, 25th, 50th, 75th, and 90th Percentiles, 2005-2014

These charts illustrate percentiles of risk-adjusted rates for mortality and neonatal morbidities by year. A, Mortality. B, Chronic lung disease. C, Late-onset infection. D, Necrotizing enterocolitis. E, Severe intraventricular hemorrhage. F, Severe retinopathy of prematurity.

Figure 2.
Percentage of Neonatal Intensive Care Units Reaching the 10th and 25th Percentile Rates for 6 Outcomes From 2005
Percentage of Neonatal Intensive Care Units Reaching the 10th and 25th Percentile Rates for 6 Outcomes From 2005

These charts illustrate the percentage of neonatal intensive care units reaching the 10th and 25th percentiles for mortality and neonatal morbidities from 2005 by year. A, Mortality. B, Chronic lung disease. C, Late-onset infection. D, Necrotizing enterocolitis. E, Severe intraventricular hemorrhage. F, Severe retinopathy of prematurity.

Table 1.  
Risk-Adjusted NICU Rates for 6 Neonatal Outcomes
Risk-Adjusted NICU Rates for 6 Neonatal Outcomes
Table 2.  
Percentage of NICUs in 2014 Reaching the 10th and 25th Percentile Rates From 2005 for 6 Neonatal Outcomes
Percentage of NICUs in 2014 Reaching the 10th and 25th Percentile Rates From 2005 for 6 Neonatal Outcomes
1.
Horbar  JD, Badger  GJ, Carpenter  JH,  et al; Members of the Vermont Oxford Network.  Trends in mortality and morbidity for very low birth weight infants, 1991-1999.  Pediatrics. 2002;110(1, pt 1):143-151.PubMedGoogle ScholarCrossref
2.
Horbar  JD, Carpenter  JH, Badger  GJ,  et al.  Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009.  Pediatrics. 2012;129(6):1019-1026.PubMedGoogle ScholarCrossref
3.
Stoll  BJ, Hansen  NI, Bell  EF,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Trends in care practices, morbidity, and mortality of extremely preterm neonates, 1993-2012.  JAMA. 2015;314(10):1039-1051.PubMedGoogle ScholarCrossref
4.
Vermont Oxford Network. https://nightingale.vtoxford.org. Accessed July 25, 2016.
5.
Schmidt  B, Asztalos  EV, Roberts  RS, Robertson  CM, Sauve  RS, Whitfield  MF; Trial of Indomethacin Prophylaxis in Preterms (TIPP) Investigators.  Impact of bronchopulmonary dysplasia, brain injury, and severe retinopathy on the outcome of extremely low-birth-weight infants at 18 months: results from the Trial of Indomethacin Prophylaxis in Preterms.  JAMA. 2003;289(9):1124-1129.PubMedGoogle ScholarCrossref
6.
Stoll  BJ, Hansen  NI, Adams-Chapman  I,  et al; National Institute of Child Health and Human Development Neonatal Research Network.  Neurodevelopmental and growth impairment among extremely low-birth-weight infants with neonatal infection.  JAMA. 2004;292(19):2357-2365.PubMedGoogle ScholarCrossref
7.
Hintz  SR, Kendrick  DE, Stoll  BJ,  et al; NICHD Neonatal Research Network.  Neurodevelopmental and growth outcomes of extremely low birth weight infants after necrotizing enterocolitis.  Pediatrics. 2005;115(3):696-703.PubMedGoogle ScholarCrossref
8.
Zupancic  JAF. A systematic review of costs associated with preterm birth. In: Behrman  RE, Butler  AS, eds.  Preterm Birth: Causes, Consequences, and Prevention. Washington, DC: National Academies Press; 2007:688-724.
9.
Singer  LT, Salvator  A, Guo  S, Collin  M, Lilien  L, Baley  J.  Maternal psychological distress and parenting stress after the birth of a very low-birth-weight infant.  JAMA. 1999;281(9):799-805.PubMedGoogle ScholarCrossref
10.
Soll  RF, Edwards  EM, Badger  GJ,  et al.  Obstetric and neonatal care practices for infants 501 to 1500 g from 2000 to 2009.  Pediatrics. 2013;132(2):222-228.PubMedGoogle ScholarCrossref
11.
Horbar  JD, Soll  RF, Edwards  WH.  The Vermont Oxford Network: a community of practice.  Clin Perinatol. 2010;37(1):29-47.PubMedGoogle ScholarCrossref
12.
Horbar  JD, Soll  RF.  2015 Manual of Operations, Part 2, Release 19.0. Burlington, VT: Vermont Oxford Network; 2016.
13.
Papile  LA, Burstein  J, Burstein  R, Koffler  H.  Incidence and evolution of subependymal and intraventricular hemorrhage: a study of infants with birth weights less than 1,500 gm.  J Pediatr. 1978;92(4):529-534.PubMedGoogle ScholarCrossref
14.
International Committee for the Classification of Retinopathy of Prematurity.  The International Classification of Retinopathy of Prematurity revisited.  Arch Ophthalmol. 2005;123(7):991-999.PubMedGoogle ScholarCrossref
15.
Zupancic  JA, Richardson  DK, Horbar  JD, Carpenter  JH, Lee  SK, Escobar  GJ; Vermont Oxford Network SNAP Pilot Project Participants.  Revalidation of the Score for Neonatal Acute Physiology in the Vermont Oxford Network.  Pediatrics. 2007;119(1):e156-e163.PubMedGoogle ScholarCrossref
16.
National Center for Health Statistics, Centers for Disease Control and Prevention. Vital statistics data. http://www.cdc.gov/nchs/data_access/vitalstatsonline.htm. Updated September 19, 2016. Accessed November 21, 2016.
17.
Coory  M, Gibberd  R.  New measures for reporting the magnitude of small-area variation in rates.  Stat Med. 1998;17(22):2625-2634.PubMedGoogle ScholarCrossref
18.
Simpson  JM, Evans  N, Gibberd  RW, Heuchan  AM, Henderson-Smart  DJ; Australian and New Zealand Neonatal Network.  Analysing differences in clinical outcomes between hospitals.  Qual Saf Health Care. 2003;12(4):257-262.PubMedGoogle ScholarCrossref
19.
Wears  RL, Normand  SL.  When less is more: using shrinkage to increase accuracy.  Ann Emerg Med. 2010;55(6):553-555.PubMedGoogle ScholarCrossref
20.
Shahian  DM, Normand  SL.  What is a performance outlier?  BMJ Qual Saf. 2015;24(2):95-99.PubMedGoogle ScholarCrossref
21.
Cohen  ME, Ko  CY, Bilimoria  KY,  et al.  Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus.  J Am Coll Surg. 2013;217(2):336-346.e1.PubMedGoogle ScholarCrossref
22.
MacKenzie  TA, Grunkemeier  GL, Grunwald  GK,  et al.  A primer on using shrinkage to compare in-hospital mortality between centers.  Ann Thorac Surg. 2015;99(3):757-761.PubMedGoogle ScholarCrossref
23.
Silber  JH, Rosenbaum  PR, Brachet  TJ,  et al.  The Hospital Compare mortality model and the volume-outcome relationship.  Health Serv Res. 2010;45(5, pt 1):1148-1167.PubMedGoogle ScholarCrossref
24.
Carlin  BP, Louis  TA.  Bayesian Methods for Data Analysis. 3rd ed. Boca Raton, FL: Chapman & Hall/CRC; 2008.
25.
Zeitlin  J, Manktelow  BN, Piedvache  A,  et al; EPICE Research Group.  Use of evidence based practices to improve survival without severe morbidity for very preterm infants: results from the EPICE population based cohort.  BMJ. 2016;354:i2976.PubMedGoogle ScholarCrossref
26.
Roberts  D, Dalziel  S.  Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth.  Cochrane Database Syst Rev. 2006;(3):CD004454.PubMedGoogle Scholar
27.
ACOG Committee on Obstetric Practice.  ACOG Committee Opinion No. 475: antenatal corticosteroid therapy for fetal maturation.  Obstet Gynecol. 2011;117(2, pt 1):422-424.PubMedGoogle ScholarCrossref
28.
Lee  HC, Kurtin  PS, Wight  NE,  et al.  A quality improvement project to increase breast milk use in very low birth weight infants.  Pediatrics. 2012;130(6):e1679-e1687.PubMedGoogle ScholarCrossref
29.
Centers for Disease Control and Prevention. State perinatal quality collboratives. http://www.cdc.gov/reproductivehealth/MaternalInfantHealth/PQC-States.html. Updated March 2, 2016. Accessed July 25, 2016.
30.
Gould  JB.  The role of regional collaboratives: the California Perinatal Quality Care Collaborative model.  Clin Perinatol. 2010;37(1):71-86.PubMedGoogle ScholarCrossref
31.
Ellsbury  DL, Clark  RH, Ursprung  R, Handler  DL, Dodd  ED, Spitzer  AR.  A multifaceted approach to improving outcomes in the NICU: the Pediatrix 100 000 Babies Campaign.  Pediatrics. 2016;137(4):e20150389.PubMedGoogle ScholarCrossref
32.
Payne  NR, Finkelstein  MJ, Liu  M, Kaempf  JW, Sharek  PJ, Olsen  S.  NICU practices and outcomes associated with 9 years of quality improvement collaboratives.  Pediatrics. 2010;125(3):437-446.PubMedGoogle ScholarCrossref
33.
Kaplan  HC, Lannon  C, Walsh  MC, Donovan  EF; Ohio Perinatal Quality Collaborative.  Ohio statewide quality-improvement collaborative to reduce late-onset sepsis in preterm infants.  Pediatrics. 2011;127(3):427-435.PubMedGoogle ScholarCrossref
34.
Schulman  J, Stricof  R, Stevens  TP,  et al; New York State Regional Perinatal Care Centers.  Statewide NICU central-line-associated bloodstream infection rates decline after bundles and checklists.  Pediatrics. 2011;127(3):436-444.PubMedGoogle ScholarCrossref
35.
Wirtschafter  DD, Powers  RJ, Pettit  JS,  et al.  Nosocomial infection reduction in VLBW infants with a statewide quality-improvement model.  Pediatrics. 2011;127(3):419-426.PubMedGoogle ScholarCrossref
36.
Fisher  D, Cochran  KM, Provost  LP,  et al.  Reducing central line–associated bloodstream infections in North Carolina NICUs.  Pediatrics. 2013;132(6):e1664-e1671.PubMedGoogle ScholarCrossref
37.
Miller  MR, Niedner  MF, Huskins  WC,  et al; National Association of Children’s Hospitals and Related Institutions Pediatric Intensive Care Unit Central Line–Associated Bloodstream Infection Quality Transformation Teams.  Reducing PICU central line–associated bloodstream infections: 3-year results.  Pediatrics. 2011;128(5):e1077-e1083.PubMedGoogle ScholarCrossref
38.
Pronovost  P, Needham  D, Berenholtz  S,  et al.  An intervention to decrease catheter-related bloodstream infections in the ICU.  N Engl J Med. 2006;355(26):2725-2732.PubMedGoogle ScholarCrossref
39.
Plsek  PE.  Quality improvement methods in clinical medicine.  Pediatrics. 1999;103(1)(suppl E):203-214.PubMedGoogle Scholar
40.
Eppstein  MJ, Horbar  JD, Buzas  JS, Kauffman  SA.  Searching the clinical fitness landscape.  PLoS One. 2012;7(11):e49901.PubMedGoogle ScholarCrossref
41.
Auerswald  P, Kauffman  S, Lobo  J, Shell  K.  The production recipes approach to modeling technological innovation: an application to learning by doing.  J Econ Dyn Control. 2000;24(3):389-450. doi:10.1016/S0165-1889(98)00091-8Google ScholarCrossref
42.
Schouten  LM, Hulscher  ME, van Everdingen  JJ, Huijsman  R, Grol  RP.  Evidence for the impact of quality improvement collaboratives: systematic review.  BMJ. 2008;336(7659):1491-1494.PubMedGoogle ScholarCrossref
43.
Horbar  JD, Carpenter  JH, Buzas  J,  et al.  Collaborative quality improvement to promote evidence based surfactant for preterm infants: a cluster randomised trial.  BMJ. 2004;329(7473):1004.PubMedGoogle ScholarCrossref
44.
Walsh  M, Laptook  A, Kazzi  SN,  et al; National Institute of Child Health and Human Development Neonatal Research Network.  A cluster-randomized trial of benchmarking and multimodal quality improvement to improve rates of survival free of bronchopulmonary dysplasia for infants with birth weights of less than 1250 grams.  Pediatrics. 2007;119(5):876-890.PubMedGoogle ScholarCrossref
45.
Lee  SK, Aziz  K, Singhal  N,  et al.  Improving the quality of care for infants: a cluster randomized controlled trial.  CMAJ. 2009;181(8):469-476.PubMedGoogle ScholarCrossref
46.
McEvoy  CT, Jain  L, Schmidt  B, Abman  S, Bancalari  E, Aschner  JL.  Bronchopulmonary dysplasia: NHLBI workshop on the primary prevention of chronic lung diseases.  Ann Am Thorac Soc. 2014;11(suppl 3):S146-S153.PubMedGoogle ScholarCrossref
47.
Ioannidis  JP, Horbar  JD, Ovelman  CM,  et al.  Completeness of main outcomes across randomized trials in entire discipline: survey of chronic lung disease outcomes in preterm infants.  BMJ. 2015;350:h72.PubMedGoogle ScholarCrossref
Original Investigation
Journal Club
March 6, 2017

Variation in Performance of Neonatal Intensive Care Units in the United States

Journal Club PowerPoint Slide Download
Author Affiliations
  • 1Vermont Oxford Network, Burlington
  • 2Department of Pediatrics, College of Medicine, University of Vermont, Burlington
  • 3Department of Mathematics and Statistics, College of Engineering and Mathematical Sciences, University of Vermont, Burlington
  • 4Department of Pediatrics, Geisel School of Medicine, Dartmouth College, Hanover, New Hampshire
JAMA Pediatr. 2017;171(3):e164396. doi:10.1001/jamapediatrics.2016.4396
Key Points

Question  What proportion of neonatal intensive care units in 2014 achieved the risk-adjusted rates from the best quartile and decile from 2005 for death and serious morbidities?

Findings  In this observational study, we calculated the 10th, 25th, 50th, 75th, and 90th percentiles of risk-adjusted neonatal intensive care unit rates for death and serious morbidities from 2005 to 2014 at US Vermont Oxford Network member neonatal intensive care units. Adjusted rates for all outcomes decreased during the study: within 8 years, 75% of neonatal intensive care units achieved the rates from the best quartile in 2005 for all outcomes except chronic lung disease.

Meaning  These findings provide a novel way to quantify the magnitude and pace of improvement in neonatology.

Abstract

Importance  Hospitals use rates from the best quartile or decile as benchmarks for quality improvement aims, but to what extent these aims are achievable is uncertain.

Objective  To determine the proportion of neonatal intensive care units (NICUs) in 2014 that achieved rates for death and major morbidities as low as the shrunken adjusted rates from the best quartile and decile in 2005 and the time it took to achieve those rates.

Design, Setting, and Participants  A total of 408 164 infants with a birth weight of 501 to 1500 g born from January 1, 2005, to December 31, 2014, and cared for at 756 Vermont Oxford Network member NICUs in the United States were evaluated. Logistic regression models with empirical Bayes factors were used to estimate standardized morbidity ratios for each NICU. Each ratio was multiplied by the overall network rate to calculate the 10th, 25th, 50th, 75th, and 90th percentiles of the shrunken adjusted rates for each year. The proportion in 2014 that achieved the 10th and 25th percentile rates from 2005 and the number of years it took for 75% of NICUs to achieve the 2005 rates from the best quartile were estimated.

Main Outcomes and Measures  Death prior to hospital discharge, infection more than 3 days after birth, severe retinopathy of prematurity, severe intraventricular hemorrhage, necrotizing enterocolitis, and chronic lung disease among infants less than 33 weeks’ gestational age at birth.

Results  Of the 756 hospitals, 695 provided data for 2014. The mean unadjusted infant-level rate of death before hospital discharge decreased from 14.0% in 2005 to 10.9% in 2014. In 2014, 689 of 695 NICUs (99.1%; 95% CI, 97.4%-100.0%) achieved the 2005 shrunken adjusted rates from the best quartile for death prior to discharge, 678 of 695 (97.6%; 95% CI, 95.8%-99.6%) for late-onset infection, 558 of 681 (81.9%; 95% CI, 77.2%-86.6%) for severe retinopathy of prematurity, 611 of 693 (88.2%; 95% CI, 81.7%-97.0%) for severe intraventricular hemorrhage, 529 of 696 (76.0%; 95% CI, 71.8%-81.2%) for necrotizing enterocolitis, and 286 of 693 (41.3%; 95% CI, 36.1%-45.6%) for chronic lung disease. It took 3 years before 445 NICUs (75.0%) achieved the 2005 shrunken adjusted rate from the best quartile for death prior to discharge, 5 years to achieve the rate from the best quartile for late-onset infection, 6 years to achieve the rate from the best quartile for severe retinopathy of prematurity and severe intraventricular hemorrhage, and 8 years to achieve the rate from the best quartile for necrotizing enterocolitis.

Conclusions and Relevance  From 2005 to 2014, rates of death prior to discharge and serious morbidities decreased among the NICUs in this study. Within 8 years, 75% of NICUs achieved rates of performance from the best quartile of the 2005 benchmark for all outcomes except chronic lung disease. These findings provide a novel way to quantify the magnitude and pace of improvement in neonatology.

Introduction

Outcomes for very low-birth-weight infants treated in neonatal intensive care units (NICUs) have improved during the past 25 years.1-3 Despite these improvements, more than 40% of these infants still die during their initial hospitalization or experience 1 or more serious morbidities4 that can result in adverse neurodevelopmental outcomes,5-7 prolonged lengths of stay and increased costs,8 and psychological distress for families.9

The development of evidence-based perinatal and obstetrical care practices10 and application of these practices to clinical care contribute to improved outcomes. Quality improvement teams working in NICUs frequently use the hospital rates at the best quartile or best decile when setting measurable improvement aims. In this study, we address the following question: Can all NICUs perform as well as those at the best quartile or decile? By definition, only 10% or 25% of NICUs can perform in the best decile or best quartile for a given outcome in a given year, but we can ask how many NICUs achieve the rates from the best quartile or best decile from a baseline year in subsequent years and how long it took them to do so.

We used data from Vermont Oxford Network, a voluntary collaboration of health care professionals around the world dedicated to improving the quality, safety, and value of care for newborn infants and their families.11 More than 750 NICUs in the United States participate in the Vermont Oxford Network database, which enrolls nearly 90% of all very low-birth-weight infants born in the United States each year. These data provide a unique opportunity to characterize the variation in outcomes among NICUs. The new measures we describe provide insights into the magnitude and pace of improvements that have occurred in neonatal intensive care and will help teams assess their own performance and set measurable aims for quality improvement.

Methods

Quiz Ref IDVermont Oxford Network members submitted standardized data for infants with a birth weight of 401 to 1500 g or a gestational age of 22 weeks 0 days to 29 weeks 6 days who were born in the member hospital or admitted within 28 days of birth without first having been discharged home. This analysis includes infants weighing 501 to 1500 g who were born from January 1, 2005, to December 31, 2014, submitted by 756 hospitals in the United States (eTable 1 in the Supplement). All hospitals that contributed finalized data at any point during the study period were included. Local staff collected data using uniform definitions that did not change during the study period.12 All data underwent automated checks for quality and completeness at the time of submission. The University of Vermont Committee on Human Research approved the use of the Vermont Oxford Network Research Repository for this analysis. Consent for the Research Repository was waived by the University of Vermont Committee on Human Research. All data were deidentified.

Outcomes

Quiz Ref IDMortality was defined as death before hospital discharge. Infants transferred from the reporting hospital to another hospital were tracked for survival status until discharge from that hospital. Infants were classified as having chronic lung disease if they were younger than 33 weeks’ gestational age at birth and received supplemental oxygen at 36 weeks’ postmenstrual age or, if transferred or discharged at 34 to 35 weeks, were receiving supplemental oxygen at discharge. Late-onset infection included recovery of a bacterial pathogen or coagulase-negative Staphylococcus from blood or cerebrospinal fluid obtained more than 3 days after birth or a fungus from a blood specimen obtained more than 3 days after birth. Diagnosis of coagulase-negative staphylococcal infection also required systemic signs of infection and treatment for 5 days or more with intravenous antibiotics. Diagnosis of necrotizing enterocolitis occurred at surgery or postmortem or required at least 1 clinical sign (eg, bilious gastric aspirate or emesis, abdominal distention, or gross or occult blood in the stool) and at least 1 radiographic finding (eg, pneumatosis intestinalis, hepatobiliary gas, or pneumoperitoneum). Intraventricular hemorrhage was diagnosed with cranial imaging within 28 days of birth by using cranial ultrasonography (in 2005) or cranial ultrasonography, magnetic resonance imaging, or computed tomography (from 2006-2014), with severe intraventricular hemorrhage defined as grades 3 and 4 using the classification from Papile et al.13 Diagnosis and staging of retinopathy of prematurity were based on results of retinal examination before discharge, with severe retinopathy of prematurity defined as stages 3 to 5.14

Hospital Characteristics

Members completed surveys of hospital characteristics (response rate, 755 [99.9%]). Type of NICU was derived based on responses to whether the hospital was required by state regulation to transfer infants to another hospital for assisted ventilation based on the infant’s characteristics or duration of ventilation required or whether none of 10 surgical procedures was performed at the hospital (omphalocoele repair, ventriculoperitoneal shunt, tracheoesophageal fistula or esophageal atresia repair, bowel resection or renanastomosis, meningomyelocoele repair, patent ductus arteriosus ligation, cardiac catheterization, or cardiac surgery requiring bypass). Teaching hospitals were defined as those having neonatal fellows, pediatric residents, or other residents participating in direct patient care in the NICU.

Statistical Analysis

Members reported observed counts for each outcome and year. We calculated expected counts using logistic regression models adjusted for case mix (sex, gestational age, location of birth, multiple birth, small size for gestational age, birth defects, Apgar score at 1 minute, and mode of delivery).15 Eligible infants from multiple births were included as separate observations. Small size for gestational age was defined within categories of sex, race/ethnicity, and multiple birth as birth weight below the 10th percentile based on smoothed curves from the US Natality Data set, which has counts and rates of births occurring within the United States.16 For chronic lung disease, we adjusted for altitude above 4000 feet.

We used empirical Bayes shrinkage estimators to compute standardized morbidity ratios for each hospital using a hierarchical logistic regression model that assumed a posterior gamma Poisson distribution.17,18 We multiplied each hospital’s standardized morbidity ratio for each year and outcome by the Vermont Oxford Network rate of the outcome for that year to calculate adjusted rates. Quantiles by year represented the hospital-level rates at the 10th, 25th, 50th, 75th, and 90th percentiles. The 95% CIs for the percentage of NICUs that achieved the 10th and 25th percentile rates from 2005 to 2014 were calculated by bootstrap resampling of observed and expected counts.

To evaluate the potential effect of changes in participating hospitals over time, primary analyses were replicated for the 421 hospitals that participated for all 10 years. Since the results were essentially unchanged, we report only the data for the entire population. Analyses were performed using SAS statistical software, version 9.4 (SAS Institute), and R, version 3.3.0 (R Core Team [2016], R Foundation for Statistical Computing).

Results

Of the 756 hospitals that contributed data to the study, 93 (12.3%) had restrictions on assisted ventilation, 225 (29.8%) had no ventilation restrictions and did not perform neonatal surgery, 312 (41.3%) had no ventilation restrictions and performed neonatal surgery except cardiac surgery requiring bypass, and 114 (15.1%) had no ventilation restrictions and performed neonatal surgery, including cardiac surgery requiring bypass. A total of 362 hospitals (47.9%) were teaching hospitals. Overall, 756 hospitals contributed at least 1 year of data, 631 contributed data for at least 5 years, and 421 contributed data for all 10 years.

Mean unadjusted rates for death before hospital discharge and all neonatal morbidities declined from 2005 to 2014 (eTable 2 in the Supplement). Quiz Ref IDThe mean unadjusted infant-level rate of death before hospital discharge decreased from 14.0% in 2005 to 10.9% in 2014, Quiz Ref IDlate-onset infection decreased from 21.9% to 10.1%, severe intraventricular hemorrhage decreased from 9.4% to 7.9%, necrotizing enterocolitis decreased from 7.1% to 5.2%, severe retinopathy of prematurity decreased from 9.8% to 6.2%, and chronic lung disease decreased from 31.6% to 28.6%. During this period, the percentage of infants who were born at the reporting hospital and discharged home increased, while the percentage of infants transferred decreased.

From 2005 to 2014, the shrunken adjusted quantile rates decreased for every outcome (Figure 1 and Table 1). At the 25th percentile, absolute differences in outcome rates from 2005 to 2014 were 9% for late infection, 3% for mortality, 2.2% for chronic lung disease, 2.4% for severe retinopathy of prematurity, 1.4% for necrotizing enterocolitis, and 1.1% for severe intraventricular hemorrhage. Absolute differences from 2005 to 2014 for the 90th percentile were 16% for infection and 6.1% for severe retinopathy of prematurity, indicating that the highest NICU rates for these outcomes dropped substantially during the 10 years of the study. Infection had the largest absolute decreases at every quantile from 2005 to 2014.

By 2014, more than 75% of the 695 NICUs contributing data in 2014 achieved the shrunken adjusted rate from the 25th percentile from 2005 for all measures except chronic lung disease (Figure 2 and Table 2). Neonatal intensive care units achieved the biggest gains in death before hospital discharge and late-onset infection: by 2014, 689 of 695 NICUs (99.1%; 95% CI, 97.4%-100.0%) achieved shrunken adjusted rates of mortality and 678 of 695 (97.6%; 95% CI, 95.8%-99.6%) achieved shrunken adjusted rates of late-onset infection as low as or lower than the rate of the best quartile in 2005, while 684 of 695 NICUs (98.4%; 95% CI, 94.1%-100.0%) achieved shrunken adjusted mortality rates and 632 of 695 (90.9%; 95% CI, 87.1%-95.0%) achieved shrunken adjusted rates of late-onset infection as low as or lower than the rate of the best decile in 2005. For other measures, 611 of 693 NICUs (88.2%; 95% CI, 81.7%-97.0%) achieved the shrunken adjusted rate from the best quartile for severe intraventricular hemorrhage, 558 of 681 (81.9%; 95% CI, 77.2%-86.6%) for severe retinopathy of prematurity, 529 of 696 (76.0%; 95% CI, 71.8%-81.2%) for necrotizing enterocolitis, and 286 of 693 (41.3%; 95% CI, 36.1%-45.6%) for chronic lung disease. A total of 446 of 693 NICUs (64.4%; 95% CI, 59.7%-72.6%) achieved the shrunken adjusted rate for the best decile in 2005 of severe intraventricular hemorrhage, 310 of 681 (45.5%; 95% CI, 40.8%-49.8%) for severe retinopathy of prematurity, 366 of 696 (52.6%; 95% CI, 46.3%-56.9%) for necrotizing enterocolitis, and 118 of 693 (17.0%; 95% CI, 12.0%-21.9%) for chronic lung disease.

It took 3 years before 445 NICUs (75.0%) had achieved the shrunken adjusted rate from the best quartile in 2005 for death prior to discharge, 5 years to achieve the rate from the best quartile for late-onset infection, 6 years to achieve the rate from the best quartile for severe retinopathy of prematurity and severe intraventricular hemorrhage, and 8 years to achieve the rate from the best quartile for necrotizing enterocolitis.

Discussion

Short-term outcomes for very low-birth-weight infants have improved in recent years.1-3 From 2005 to 2014, unadjusted rates of mortality and 5 neonatal morbidities declined with varying magnitude, from a relative decrease of more than 50% for late-onset infection (from 22% in 2005 to 10% in 2014) to a relative decrease of less than 10% for chronic lung disease (from 32% in 2005 to 29% in 2014).

Variation among NICUs in the magnitude and pace of improvement has not been described. We reported quantiles for shrunken adjusted rates of death before hospital discharge and selected neonatal morbidities from 2005 to 2014 for 756 Vermont Oxford Network member NICUs in the United States. The proportions of NICUs that had achieved rates previously achieved by only the best 10% or 25% of NICUs were high for all outcomes. By 2014, 689 NICUs (99.1%) had achieved the shrunken adjusted rate of the best quartile from 2005 and 684 NICUs (98.4%) had achieved the shrunken adjusted rate of the best decile from 2005 for mortality, while 678 NICUs (97.6%) had achieved the shrunken adjusted rate of the best quartile from 2005 and 632 (90.9%) had achieved the shrunken adjusted rate of the best decile from 2005 for late-onset infection. Neonatal intensive care units were able to make gains in necrotizing enterocolitis, severe intraventricular hemorrhage, and severe retinopathy of prematurity, with 76.0%, 88.2%, and 81.9% of NICUs, respectively, reaching the shrunken adjusted rates of the best quartile from 2005 for these measures in 4 to 8 years. These results illustrate the magnitude and pace of improvement that has occurred over the decade and describe the variation in outcomes that persists among different NICUs. These data provide benchmarks against which NICU teams can assess their own performance, identify focused opportunities for improvement, and set measurable aims.

The metrics we present provide a novel way to quantify the magnitude and pace of improvement for the field of neonatology. The dramatic increases in the proportions of NICUs that had learned to achieve rates that had been achieved by only the best 25% or 10% of units a decade earlier represent the learning curve for the care of very low-birth-weight infants in the United States. The time it took for 75% of NICUs to achieve the shrunken adjusted rate of the best quartile from 2005 highlights the differences in the pace of improvement for the various outcomes that we studied. We are not aware of similar data from other clinical specialties and therefore cannot comment on how the magnitude and pace of improvement in neonatology compares with that of other fields.

We calculated benchmark rates in 2005 and the proportion of NICUs in 2014 below these benchmarks using empirical Bayesian methods applied to individual estimates of NICU standardized morbidity ratios. The estimates for each unit “shrink” toward the overall network mean, with the amount of shrinkage being greater for smaller units, resulting in conservative estimates of the adjusted quantiles. Shrinkage to reduce variation owing to small sample sizes is an accepted method19,20 used by the American College of Surgeons National Surgical Quality Improvement Program21 and other registries.22 Hospitals with fewer infants may appear to perform better through shrinking to the network mean than if they were shrunken to a mean specific to low-volume hospitals.23 There are alternative ways to estimate these benchmarks and proportions, such as squared-error loss ranking procedures or posterior means of an appropriately defined function of the ratios.24 Understanding and comparing the statistical properties of alternative methods, as well as how long a given hospital can sustain a particular quantile rate and the effects of hospital volume, are important areas for future work, particularly if these metrics are to be applied in other clinical fields.

A possible explanation for the improvements we observed is the development and dissemination of evidence-based perinatal and obstetrical care practices and the application of improvement science to translate that evidence into routine practice.25 Use of antenatal corticosteroids,26,27 widespread adoption of less invasive approaches to respiratory support,10 standardized care of very low-birth-weight infants, and improved nutrition management, specifically the recognition of the importance of human milk for feeding preterm infants,28 are examples of evolving practices that can have a measurable effect on these outcomes. The increased adoption of evidence-based practices and their implementation using quality improvement methods may have contributed to the gains in survival without morbidity that we observed.

Since 1995, multidisciplinary teams from more than 550 NICUs have participated in Vermont Oxford Network quality improvement collaboratives. Hundreds more have participated in collaboratives organized by state perinatal organizations29,30 and private neonatology groups.31 Late-onset infection has been a major focus for these efforts, with evidence to suggest that infection in neonatal,32-36 pediatric,37 and adult38 intensive care units can be reduced by participation in quality improvement programs. Neonatology has been at the forefront of applying improvement science to daily care. During the decade under study, many NICU teams have worked together in collaboratives and have tested, implemented, and standardized a variety of potentially better practices designed to reduce the major morbidities we have studied. We refer to them as “potentially better” rather than “better” or “best” practices because a practice is not better or best until it is shown to work in the local context.39 Modeling and simulations based on concepts from complexity science support the suggestion that collaborative learning and learning by doing in the local context are effective strategies for improvement.40,41 The extent to which these efforts have contributed to the improvements we report is uncertain.42

Quiz Ref IDAlthough there were improvements in chronic lung disease, the magnitude of change in rates was smaller than for the other outcomes. Existing interventions for the prevention of chronic lung disease may be less effective than those targeting other outcomes or may be more difficult to reliably implement. There have been 3 cluster randomized clinical trials of quality improvement collaboratives testing interventions that might reduce chronic lung disease.43-45 Although all 3 trials demonstrated changes in practices, only 1 showed a reduction in chronic lung disease.45 Another possibility is that our definition of chronic lung disease is insensitive to detecting changes. Although the definition—receiving supplemental oxygen at 36 weeks’ postnatal age—is widely used in research and clinical reporting,46 it may not capture changes in the more severe cases of chronic lung disease in which infants require either higher levels of supplemental oxygen or continued respiratory assistance. Additional evidence from randomized clinical trials of interventions to prevent chronic lung disease is needed.47

Limitations

Our study has several potential limitations. First, since this was an uncontrolled observational study, we cannot determine the causes for the improvements reported. Second, the number of hospitals that participated in Vermont Oxford Network increased from 2005 to 2014. We saw nearly identical results when we restricted our analysis to the hospitals that participated in all 10 years, so we do not expect that changes in hospital characteristics contribute to the differences. Third, transferred infants were tracked for survival, but not morbidities, after transfer. The percentage of infants transferred during the study period decreased, which may have resulted in an overestimate of morbidity rates in 2014 relative to 2005 and an underestimate of the proportion of NICUs achieving best quintiles in 2014. Finally, this study includes only members of Vermont Oxford Network and thus may not be generalizable to nonmembers. However, since the Vermont Oxford Network database now enrolls approximately 90% of all very low-birth-weight infants born in the United States each year, it is likely that the results we report are representative of outcomes in the United States.

Conclusions

By 2014, more than 75% of NICUs in the United States had learned to perform as well or better than the best 25% of NICUs performed in 2005 for major morbidities other than chronic lung disease. The challenge is to accelerate the pace of learning and achieve in just a few years what previously took a decade.

Back to top
Article Information

Correction: This article was corrected on March 6, 2017, to fix Figure 1.

Accepted for Publication: November 1, 2016.

Corresponding Author: Erika M. Edwards, PhD, Vermont Oxford Network, 33 Kilburn St, Burlington, VT 05401 (eedwards@vtoxford.org).

Published Online: January 9, 2017. doi:10.1001/jamapediatrics.2016.4396

Author Contributions: Dr Edwards and Ms Greenberg had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Horbar, Edwards, Soll, Buus-Frank, Buzas.

Acquisition, analysis, or interpretation of data: Horbar, Edwards, Greenberg, Morrow, Buus-Frank, Buzas.

Drafting of the manuscript: Horbar, Edwards, Soll, Buzas.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Edwards, Greenberg, Morrow, Buzas.

Administrative, technical, or material support: Horbar, Buus-Frank.

Study supervision: Horbar, Soll.

Conflict of Interest Disclosures: Drs Horbar, Soll, and Buus-Frank and Mss Greenberg and Morrow are employees of Vermont Oxford Network. Drs Buzas and Edwards are employees of the University of Vermont and receive salary support from Vermont Oxford Network. No other disclosures were reported.

Additional Contributions: We thank the Vermont Oxford Network member hospitals that contributed data used in this study.

References
1.
Horbar  JD, Badger  GJ, Carpenter  JH,  et al; Members of the Vermont Oxford Network.  Trends in mortality and morbidity for very low birth weight infants, 1991-1999.  Pediatrics. 2002;110(1, pt 1):143-151.PubMedGoogle ScholarCrossref
2.
Horbar  JD, Carpenter  JH, Badger  GJ,  et al.  Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009.  Pediatrics. 2012;129(6):1019-1026.PubMedGoogle ScholarCrossref
3.
Stoll  BJ, Hansen  NI, Bell  EF,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Trends in care practices, morbidity, and mortality of extremely preterm neonates, 1993-2012.  JAMA. 2015;314(10):1039-1051.PubMedGoogle ScholarCrossref
4.
Vermont Oxford Network. https://nightingale.vtoxford.org. Accessed July 25, 2016.
5.
Schmidt  B, Asztalos  EV, Roberts  RS, Robertson  CM, Sauve  RS, Whitfield  MF; Trial of Indomethacin Prophylaxis in Preterms (TIPP) Investigators.  Impact of bronchopulmonary dysplasia, brain injury, and severe retinopathy on the outcome of extremely low-birth-weight infants at 18 months: results from the Trial of Indomethacin Prophylaxis in Preterms.  JAMA. 2003;289(9):1124-1129.PubMedGoogle ScholarCrossref
6.
Stoll  BJ, Hansen  NI, Adams-Chapman  I,  et al; National Institute of Child Health and Human Development Neonatal Research Network.  Neurodevelopmental and growth impairment among extremely low-birth-weight infants with neonatal infection.  JAMA. 2004;292(19):2357-2365.PubMedGoogle ScholarCrossref
7.
Hintz  SR, Kendrick  DE, Stoll  BJ,  et al; NICHD Neonatal Research Network.  Neurodevelopmental and growth outcomes of extremely low birth weight infants after necrotizing enterocolitis.  Pediatrics. 2005;115(3):696-703.PubMedGoogle ScholarCrossref
8.
Zupancic  JAF. A systematic review of costs associated with preterm birth. In: Behrman  RE, Butler  AS, eds.  Preterm Birth: Causes, Consequences, and Prevention. Washington, DC: National Academies Press; 2007:688-724.
9.
Singer  LT, Salvator  A, Guo  S, Collin  M, Lilien  L, Baley  J.  Maternal psychological distress and parenting stress after the birth of a very low-birth-weight infant.  JAMA. 1999;281(9):799-805.PubMedGoogle ScholarCrossref
10.
Soll  RF, Edwards  EM, Badger  GJ,  et al.  Obstetric and neonatal care practices for infants 501 to 1500 g from 2000 to 2009.  Pediatrics. 2013;132(2):222-228.PubMedGoogle ScholarCrossref
11.
Horbar  JD, Soll  RF, Edwards  WH.  The Vermont Oxford Network: a community of practice.  Clin Perinatol. 2010;37(1):29-47.PubMedGoogle ScholarCrossref
12.
Horbar  JD, Soll  RF.  2015 Manual of Operations, Part 2, Release 19.0. Burlington, VT: Vermont Oxford Network; 2016.
13.
Papile  LA, Burstein  J, Burstein  R, Koffler  H.  Incidence and evolution of subependymal and intraventricular hemorrhage: a study of infants with birth weights less than 1,500 gm.  J Pediatr. 1978;92(4):529-534.PubMedGoogle ScholarCrossref
14.
International Committee for the Classification of Retinopathy of Prematurity.  The International Classification of Retinopathy of Prematurity revisited.  Arch Ophthalmol. 2005;123(7):991-999.PubMedGoogle ScholarCrossref
15.
Zupancic  JA, Richardson  DK, Horbar  JD, Carpenter  JH, Lee  SK, Escobar  GJ; Vermont Oxford Network SNAP Pilot Project Participants.  Revalidation of the Score for Neonatal Acute Physiology in the Vermont Oxford Network.  Pediatrics. 2007;119(1):e156-e163.PubMedGoogle ScholarCrossref
16.
National Center for Health Statistics, Centers for Disease Control and Prevention. Vital statistics data. http://www.cdc.gov/nchs/data_access/vitalstatsonline.htm. Updated September 19, 2016. Accessed November 21, 2016.
17.
Coory  M, Gibberd  R.  New measures for reporting the magnitude of small-area variation in rates.  Stat Med. 1998;17(22):2625-2634.PubMedGoogle ScholarCrossref
18.
Simpson  JM, Evans  N, Gibberd  RW, Heuchan  AM, Henderson-Smart  DJ; Australian and New Zealand Neonatal Network.  Analysing differences in clinical outcomes between hospitals.  Qual Saf Health Care. 2003;12(4):257-262.PubMedGoogle ScholarCrossref
19.
Wears  RL, Normand  SL.  When less is more: using shrinkage to increase accuracy.  Ann Emerg Med. 2010;55(6):553-555.PubMedGoogle ScholarCrossref
20.
Shahian  DM, Normand  SL.  What is a performance outlier?  BMJ Qual Saf. 2015;24(2):95-99.PubMedGoogle ScholarCrossref
21.
Cohen  ME, Ko  CY, Bilimoria  KY,  et al.  Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus.  J Am Coll Surg. 2013;217(2):336-346.e1.PubMedGoogle ScholarCrossref
22.
MacKenzie  TA, Grunkemeier  GL, Grunwald  GK,  et al.  A primer on using shrinkage to compare in-hospital mortality between centers.  Ann Thorac Surg. 2015;99(3):757-761.PubMedGoogle ScholarCrossref
23.
Silber  JH, Rosenbaum  PR, Brachet  TJ,  et al.  The Hospital Compare mortality model and the volume-outcome relationship.  Health Serv Res. 2010;45(5, pt 1):1148-1167.PubMedGoogle ScholarCrossref
24.
Carlin  BP, Louis  TA.  Bayesian Methods for Data Analysis. 3rd ed. Boca Raton, FL: Chapman & Hall/CRC; 2008.
25.
Zeitlin  J, Manktelow  BN, Piedvache  A,  et al; EPICE Research Group.  Use of evidence based practices to improve survival without severe morbidity for very preterm infants: results from the EPICE population based cohort.  BMJ. 2016;354:i2976.PubMedGoogle ScholarCrossref
26.
Roberts  D, Dalziel  S.  Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth.  Cochrane Database Syst Rev. 2006;(3):CD004454.PubMedGoogle Scholar
27.
ACOG Committee on Obstetric Practice.  ACOG Committee Opinion No. 475: antenatal corticosteroid therapy for fetal maturation.  Obstet Gynecol. 2011;117(2, pt 1):422-424.PubMedGoogle ScholarCrossref
28.
Lee  HC, Kurtin  PS, Wight  NE,  et al.  A quality improvement project to increase breast milk use in very low birth weight infants.  Pediatrics. 2012;130(6):e1679-e1687.PubMedGoogle ScholarCrossref
29.
Centers for Disease Control and Prevention. State perinatal quality collboratives. http://www.cdc.gov/reproductivehealth/MaternalInfantHealth/PQC-States.html. Updated March 2, 2016. Accessed July 25, 2016.
30.
Gould  JB.  The role of regional collaboratives: the California Perinatal Quality Care Collaborative model.  Clin Perinatol. 2010;37(1):71-86.PubMedGoogle ScholarCrossref
31.
Ellsbury  DL, Clark  RH, Ursprung  R, Handler  DL, Dodd  ED, Spitzer  AR.  A multifaceted approach to improving outcomes in the NICU: the Pediatrix 100 000 Babies Campaign.  Pediatrics. 2016;137(4):e20150389.PubMedGoogle ScholarCrossref
32.
Payne  NR, Finkelstein  MJ, Liu  M, Kaempf  JW, Sharek  PJ, Olsen  S.  NICU practices and outcomes associated with 9 years of quality improvement collaboratives.  Pediatrics. 2010;125(3):437-446.PubMedGoogle ScholarCrossref
33.
Kaplan  HC, Lannon  C, Walsh  MC, Donovan  EF; Ohio Perinatal Quality Collaborative.  Ohio statewide quality-improvement collaborative to reduce late-onset sepsis in preterm infants.  Pediatrics. 2011;127(3):427-435.PubMedGoogle ScholarCrossref
34.
Schulman  J, Stricof  R, Stevens  TP,  et al; New York State Regional Perinatal Care Centers.  Statewide NICU central-line-associated bloodstream infection rates decline after bundles and checklists.  Pediatrics. 2011;127(3):436-444.PubMedGoogle ScholarCrossref
35.
Wirtschafter  DD, Powers  RJ, Pettit  JS,  et al.  Nosocomial infection reduction in VLBW infants with a statewide quality-improvement model.  Pediatrics. 2011;127(3):419-426.PubMedGoogle ScholarCrossref
36.
Fisher  D, Cochran  KM, Provost  LP,  et al.  Reducing central line–associated bloodstream infections in North Carolina NICUs.  Pediatrics. 2013;132(6):e1664-e1671.PubMedGoogle ScholarCrossref
37.
Miller  MR, Niedner  MF, Huskins  WC,  et al; National Association of Children’s Hospitals and Related Institutions Pediatric Intensive Care Unit Central Line–Associated Bloodstream Infection Quality Transformation Teams.  Reducing PICU central line–associated bloodstream infections: 3-year results.  Pediatrics. 2011;128(5):e1077-e1083.PubMedGoogle ScholarCrossref
38.
Pronovost  P, Needham  D, Berenholtz  S,  et al.  An intervention to decrease catheter-related bloodstream infections in the ICU.  N Engl J Med. 2006;355(26):2725-2732.PubMedGoogle ScholarCrossref
39.
Plsek  PE.  Quality improvement methods in clinical medicine.  Pediatrics. 1999;103(1)(suppl E):203-214.PubMedGoogle Scholar
40.
Eppstein  MJ, Horbar  JD, Buzas  JS, Kauffman  SA.  Searching the clinical fitness landscape.  PLoS One. 2012;7(11):e49901.PubMedGoogle ScholarCrossref
41.
Auerswald  P, Kauffman  S, Lobo  J, Shell  K.  The production recipes approach to modeling technological innovation: an application to learning by doing.  J Econ Dyn Control. 2000;24(3):389-450. doi:10.1016/S0165-1889(98)00091-8Google ScholarCrossref
42.
Schouten  LM, Hulscher  ME, van Everdingen  JJ, Huijsman  R, Grol  RP.  Evidence for the impact of quality improvement collaboratives: systematic review.  BMJ. 2008;336(7659):1491-1494.PubMedGoogle ScholarCrossref
43.
Horbar  JD, Carpenter  JH, Buzas  J,  et al.  Collaborative quality improvement to promote evidence based surfactant for preterm infants: a cluster randomised trial.  BMJ. 2004;329(7473):1004.PubMedGoogle ScholarCrossref
44.
Walsh  M, Laptook  A, Kazzi  SN,  et al; National Institute of Child Health and Human Development Neonatal Research Network.  A cluster-randomized trial of benchmarking and multimodal quality improvement to improve rates of survival free of bronchopulmonary dysplasia for infants with birth weights of less than 1250 grams.  Pediatrics. 2007;119(5):876-890.PubMedGoogle ScholarCrossref
45.
Lee  SK, Aziz  K, Singhal  N,  et al.  Improving the quality of care for infants: a cluster randomized controlled trial.  CMAJ. 2009;181(8):469-476.PubMedGoogle ScholarCrossref
46.
McEvoy  CT, Jain  L, Schmidt  B, Abman  S, Bancalari  E, Aschner  JL.  Bronchopulmonary dysplasia: NHLBI workshop on the primary prevention of chronic lung diseases.  Ann Am Thorac Soc. 2014;11(suppl 3):S146-S153.PubMedGoogle ScholarCrossref
47.
Ioannidis  JP, Horbar  JD, Ovelman  CM,  et al.  Completeness of main outcomes across randomized trials in entire discipline: survey of chronic lung disease outcomes in preterm infants.  BMJ. 2015;350:h72.PubMedGoogle ScholarCrossref
×