These charts illustrate percentiles of risk-adjusted rates for mortality and neonatal morbidities by year. A, Mortality. B, Chronic lung disease. C, Late-onset infection. D, Necrotizing enterocolitis. E, Severe intraventricular hemorrhage. F, Severe retinopathy of prematurity.
These charts illustrate the percentage of neonatal intensive care units reaching the 10th and 25th percentiles for mortality and neonatal morbidities from 2005 by year. A, Mortality. B, Chronic lung disease. C, Late-onset infection. D, Necrotizing enterocolitis. E, Severe intraventricular hemorrhage. F, Severe retinopathy of prematurity.
eTable 1. List of Vermont Oxford Network Members Included in the Analysis
eTable 2. Unadjusted Rates Of Death Before Discharge and Major Morbidities Among Infants 501-1500 Grams at Birth Cared for at Vermont Oxford Network Member Centers in the United States, 2005-2014
Customize your JAMA Network experience by selecting one or more topics from the list below.
Horbar JD, Edwards EM, Greenberg LT, et al. Variation in Performance of Neonatal Intensive Care Units in the United States. JAMA Pediatr. Published online March 01, 2017171(3):e164396. doi:10.1001/jamapediatrics.2016.4396
What proportion of neonatal intensive care units in 2014 achieved the risk-adjusted rates from the best quartile and decile from 2005 for death and serious morbidities?
In this observational study, we calculated the 10th, 25th, 50th, 75th, and 90th percentiles of risk-adjusted neonatal intensive care unit rates for death and serious morbidities from 2005 to 2014 at US Vermont Oxford Network member neonatal intensive care units. Adjusted rates for all outcomes decreased during the study: within 8 years, 75% of neonatal intensive care units achieved the rates from the best quartile in 2005 for all outcomes except chronic lung disease.
These findings provide a novel way to quantify the magnitude and pace of improvement in neonatology.
Hospitals use rates from the best quartile or decile as benchmarks for quality improvement aims, but to what extent these aims are achievable is uncertain.
To determine the proportion of neonatal intensive care units (NICUs) in 2014 that achieved rates for death and major morbidities as low as the shrunken adjusted rates from the best quartile and decile in 2005 and the time it took to achieve those rates.
Design, Setting, and Participants
A total of 408 164 infants with a birth weight of 501 to 1500 g born from January 1, 2005, to December 31, 2014, and cared for at 756 Vermont Oxford Network member NICUs in the United States were evaluated. Logistic regression models with empirical Bayes factors were used to estimate standardized morbidity ratios for each NICU. Each ratio was multiplied by the overall network rate to calculate the 10th, 25th, 50th, 75th, and 90th percentiles of the shrunken adjusted rates for each year. The proportion in 2014 that achieved the 10th and 25th percentile rates from 2005 and the number of years it took for 75% of NICUs to achieve the 2005 rates from the best quartile were estimated.
Main Outcomes and Measures
Death prior to hospital discharge, infection more than 3 days after birth, severe retinopathy of prematurity, severe intraventricular hemorrhage, necrotizing enterocolitis, and chronic lung disease among infants less than 33 weeks’ gestational age at birth.
Of the 756 hospitals, 695 provided data for 2014. The mean unadjusted infant-level rate of death before hospital discharge decreased from 14.0% in 2005 to 10.9% in 2014. In 2014, 689 of 695 NICUs (99.1%; 95% CI, 97.4%-100.0%) achieved the 2005 shrunken adjusted rates from the best quartile for death prior to discharge, 678 of 695 (97.6%; 95% CI, 95.8%-99.6%) for late-onset infection, 558 of 681 (81.9%; 95% CI, 77.2%-86.6%) for severe retinopathy of prematurity, 611 of 693 (88.2%; 95% CI, 81.7%-97.0%) for severe intraventricular hemorrhage, 529 of 696 (76.0%; 95% CI, 71.8%-81.2%) for necrotizing enterocolitis, and 286 of 693 (41.3%; 95% CI, 36.1%-45.6%) for chronic lung disease. It took 3 years before 445 NICUs (75.0%) achieved the 2005 shrunken adjusted rate from the best quartile for death prior to discharge, 5 years to achieve the rate from the best quartile for late-onset infection, 6 years to achieve the rate from the best quartile for severe retinopathy of prematurity and severe intraventricular hemorrhage, and 8 years to achieve the rate from the best quartile for necrotizing enterocolitis.
Conclusions and Relevance
From 2005 to 2014, rates of death prior to discharge and serious morbidities decreased among the NICUs in this study. Within 8 years, 75% of NICUs achieved rates of performance from the best quartile of the 2005 benchmark for all outcomes except chronic lung disease. These findings provide a novel way to quantify the magnitude and pace of improvement in neonatology.
Outcomes for very low-birth-weight infants treated in neonatal intensive care units (NICUs) have improved during the past 25 years.1-3 Despite these improvements, more than 40% of these infants still die during their initial hospitalization or experience 1 or more serious morbidities4 that can result in adverse neurodevelopmental outcomes,5-7 prolonged lengths of stay and increased costs,8 and psychological distress for families.9
The development of evidence-based perinatal and obstetrical care practices10 and application of these practices to clinical care contribute to improved outcomes. Quality improvement teams working in NICUs frequently use the hospital rates at the best quartile or best decile when setting measurable improvement aims. In this study, we address the following question: Can all NICUs perform as well as those at the best quartile or decile? By definition, only 10% or 25% of NICUs can perform in the best decile or best quartile for a given outcome in a given year, but we can ask how many NICUs achieve the rates from the best quartile or best decile from a baseline year in subsequent years and how long it took them to do so.
We used data from Vermont Oxford Network, a voluntary collaboration of health care professionals around the world dedicated to improving the quality, safety, and value of care for newborn infants and their families.11 More than 750 NICUs in the United States participate in the Vermont Oxford Network database, which enrolls nearly 90% of all very low-birth-weight infants born in the United States each year. These data provide a unique opportunity to characterize the variation in outcomes among NICUs. The new measures we describe provide insights into the magnitude and pace of improvements that have occurred in neonatal intensive care and will help teams assess their own performance and set measurable aims for quality improvement.
Quiz Ref IDVermont Oxford Network members submitted standardized data for infants with a birth weight of 401 to 1500 g or a gestational age of 22 weeks 0 days to 29 weeks 6 days who were born in the member hospital or admitted within 28 days of birth without first having been discharged home. This analysis includes infants weighing 501 to 1500 g who were born from January 1, 2005, to December 31, 2014, submitted by 756 hospitals in the United States (eTable 1 in the Supplement). All hospitals that contributed finalized data at any point during the study period were included. Local staff collected data using uniform definitions that did not change during the study period.12 All data underwent automated checks for quality and completeness at the time of submission. The University of Vermont Committee on Human Research approved the use of the Vermont Oxford Network Research Repository for this analysis. Consent for the Research Repository was waived by the University of Vermont Committee on Human Research. All data were deidentified.
Quiz Ref IDMortality was defined as death before hospital discharge. Infants transferred from the reporting hospital to another hospital were tracked for survival status until discharge from that hospital. Infants were classified as having chronic lung disease if they were younger than 33 weeks’ gestational age at birth and received supplemental oxygen at 36 weeks’ postmenstrual age or, if transferred or discharged at 34 to 35 weeks, were receiving supplemental oxygen at discharge. Late-onset infection included recovery of a bacterial pathogen or coagulase-negative Staphylococcus from blood or cerebrospinal fluid obtained more than 3 days after birth or a fungus from a blood specimen obtained more than 3 days after birth. Diagnosis of coagulase-negative staphylococcal infection also required systemic signs of infection and treatment for 5 days or more with intravenous antibiotics. Diagnosis of necrotizing enterocolitis occurred at surgery or postmortem or required at least 1 clinical sign (eg, bilious gastric aspirate or emesis, abdominal distention, or gross or occult blood in the stool) and at least 1 radiographic finding (eg, pneumatosis intestinalis, hepatobiliary gas, or pneumoperitoneum). Intraventricular hemorrhage was diagnosed with cranial imaging within 28 days of birth by using cranial ultrasonography (in 2005) or cranial ultrasonography, magnetic resonance imaging, or computed tomography (from 2006-2014), with severe intraventricular hemorrhage defined as grades 3 and 4 using the classification from Papile et al.13 Diagnosis and staging of retinopathy of prematurity were based on results of retinal examination before discharge, with severe retinopathy of prematurity defined as stages 3 to 5.14
Members completed surveys of hospital characteristics (response rate, 755 [99.9%]). Type of NICU was derived based on responses to whether the hospital was required by state regulation to transfer infants to another hospital for assisted ventilation based on the infant’s characteristics or duration of ventilation required or whether none of 10 surgical procedures was performed at the hospital (omphalocoele repair, ventriculoperitoneal shunt, tracheoesophageal fistula or esophageal atresia repair, bowel resection or renanastomosis, meningomyelocoele repair, patent ductus arteriosus ligation, cardiac catheterization, or cardiac surgery requiring bypass). Teaching hospitals were defined as those having neonatal fellows, pediatric residents, or other residents participating in direct patient care in the NICU.
Members reported observed counts for each outcome and year. We calculated expected counts using logistic regression models adjusted for case mix (sex, gestational age, location of birth, multiple birth, small size for gestational age, birth defects, Apgar score at 1 minute, and mode of delivery).15 Eligible infants from multiple births were included as separate observations. Small size for gestational age was defined within categories of sex, race/ethnicity, and multiple birth as birth weight below the 10th percentile based on smoothed curves from the US Natality Data set, which has counts and rates of births occurring within the United States.16 For chronic lung disease, we adjusted for altitude above 4000 feet.
We used empirical Bayes shrinkage estimators to compute standardized morbidity ratios for each hospital using a hierarchical logistic regression model that assumed a posterior gamma Poisson distribution.17,18 We multiplied each hospital’s standardized morbidity ratio for each year and outcome by the Vermont Oxford Network rate of the outcome for that year to calculate adjusted rates. Quantiles by year represented the hospital-level rates at the 10th, 25th, 50th, 75th, and 90th percentiles. The 95% CIs for the percentage of NICUs that achieved the 10th and 25th percentile rates from 2005 to 2014 were calculated by bootstrap resampling of observed and expected counts.
To evaluate the potential effect of changes in participating hospitals over time, primary analyses were replicated for the 421 hospitals that participated for all 10 years. Since the results were essentially unchanged, we report only the data for the entire population. Analyses were performed using SAS statistical software, version 9.4 (SAS Institute), and R, version 3.3.0 (R Core Team , R Foundation for Statistical Computing).
Of the 756 hospitals that contributed data to the study, 93 (12.3%) had restrictions on assisted ventilation, 225 (29.8%) had no ventilation restrictions and did not perform neonatal surgery, 312 (41.3%) had no ventilation restrictions and performed neonatal surgery except cardiac surgery requiring bypass, and 114 (15.1%) had no ventilation restrictions and performed neonatal surgery, including cardiac surgery requiring bypass. A total of 362 hospitals (47.9%) were teaching hospitals. Overall, 756 hospitals contributed at least 1 year of data, 631 contributed data for at least 5 years, and 421 contributed data for all 10 years.
Mean unadjusted rates for death before hospital discharge and all neonatal morbidities declined from 2005 to 2014 (eTable 2 in the Supplement). Quiz Ref IDThe mean unadjusted infant-level rate of death before hospital discharge decreased from 14.0% in 2005 to 10.9% in 2014, Quiz Ref IDlate-onset infection decreased from 21.9% to 10.1%, severe intraventricular hemorrhage decreased from 9.4% to 7.9%, necrotizing enterocolitis decreased from 7.1% to 5.2%, severe retinopathy of prematurity decreased from 9.8% to 6.2%, and chronic lung disease decreased from 31.6% to 28.6%. During this period, the percentage of infants who were born at the reporting hospital and discharged home increased, while the percentage of infants transferred decreased.
From 2005 to 2014, the shrunken adjusted quantile rates decreased for every outcome (Figure 1 and Table 1). At the 25th percentile, absolute differences in outcome rates from 2005 to 2014 were 9% for late infection, 3% for mortality, 2.2% for chronic lung disease, 2.4% for severe retinopathy of prematurity, 1.4% for necrotizing enterocolitis, and 1.1% for severe intraventricular hemorrhage. Absolute differences from 2005 to 2014 for the 90th percentile were 16% for infection and 6.1% for severe retinopathy of prematurity, indicating that the highest NICU rates for these outcomes dropped substantially during the 10 years of the study. Infection had the largest absolute decreases at every quantile from 2005 to 2014.
By 2014, more than 75% of the 695 NICUs contributing data in 2014 achieved the shrunken adjusted rate from the 25th percentile from 2005 for all measures except chronic lung disease (Figure 2 and Table 2). Neonatal intensive care units achieved the biggest gains in death before hospital discharge and late-onset infection: by 2014, 689 of 695 NICUs (99.1%; 95% CI, 97.4%-100.0%) achieved shrunken adjusted rates of mortality and 678 of 695 (97.6%; 95% CI, 95.8%-99.6%) achieved shrunken adjusted rates of late-onset infection as low as or lower than the rate of the best quartile in 2005, while 684 of 695 NICUs (98.4%; 95% CI, 94.1%-100.0%) achieved shrunken adjusted mortality rates and 632 of 695 (90.9%; 95% CI, 87.1%-95.0%) achieved shrunken adjusted rates of late-onset infection as low as or lower than the rate of the best decile in 2005. For other measures, 611 of 693 NICUs (88.2%; 95% CI, 81.7%-97.0%) achieved the shrunken adjusted rate from the best quartile for severe intraventricular hemorrhage, 558 of 681 (81.9%; 95% CI, 77.2%-86.6%) for severe retinopathy of prematurity, 529 of 696 (76.0%; 95% CI, 71.8%-81.2%) for necrotizing enterocolitis, and 286 of 693 (41.3%; 95% CI, 36.1%-45.6%) for chronic lung disease. A total of 446 of 693 NICUs (64.4%; 95% CI, 59.7%-72.6%) achieved the shrunken adjusted rate for the best decile in 2005 of severe intraventricular hemorrhage, 310 of 681 (45.5%; 95% CI, 40.8%-49.8%) for severe retinopathy of prematurity, 366 of 696 (52.6%; 95% CI, 46.3%-56.9%) for necrotizing enterocolitis, and 118 of 693 (17.0%; 95% CI, 12.0%-21.9%) for chronic lung disease.
It took 3 years before 445 NICUs (75.0%) had achieved the shrunken adjusted rate from the best quartile in 2005 for death prior to discharge, 5 years to achieve the rate from the best quartile for late-onset infection, 6 years to achieve the rate from the best quartile for severe retinopathy of prematurity and severe intraventricular hemorrhage, and 8 years to achieve the rate from the best quartile for necrotizing enterocolitis.
Short-term outcomes for very low-birth-weight infants have improved in recent years.1-3 From 2005 to 2014, unadjusted rates of mortality and 5 neonatal morbidities declined with varying magnitude, from a relative decrease of more than 50% for late-onset infection (from 22% in 2005 to 10% in 2014) to a relative decrease of less than 10% for chronic lung disease (from 32% in 2005 to 29% in 2014).
Variation among NICUs in the magnitude and pace of improvement has not been described. We reported quantiles for shrunken adjusted rates of death before hospital discharge and selected neonatal morbidities from 2005 to 2014 for 756 Vermont Oxford Network member NICUs in the United States. The proportions of NICUs that had achieved rates previously achieved by only the best 10% or 25% of NICUs were high for all outcomes. By 2014, 689 NICUs (99.1%) had achieved the shrunken adjusted rate of the best quartile from 2005 and 684 NICUs (98.4%) had achieved the shrunken adjusted rate of the best decile from 2005 for mortality, while 678 NICUs (97.6%) had achieved the shrunken adjusted rate of the best quartile from 2005 and 632 (90.9%) had achieved the shrunken adjusted rate of the best decile from 2005 for late-onset infection. Neonatal intensive care units were able to make gains in necrotizing enterocolitis, severe intraventricular hemorrhage, and severe retinopathy of prematurity, with 76.0%, 88.2%, and 81.9% of NICUs, respectively, reaching the shrunken adjusted rates of the best quartile from 2005 for these measures in 4 to 8 years. These results illustrate the magnitude and pace of improvement that has occurred over the decade and describe the variation in outcomes that persists among different NICUs. These data provide benchmarks against which NICU teams can assess their own performance, identify focused opportunities for improvement, and set measurable aims.
The metrics we present provide a novel way to quantify the magnitude and pace of improvement for the field of neonatology. The dramatic increases in the proportions of NICUs that had learned to achieve rates that had been achieved by only the best 25% or 10% of units a decade earlier represent the learning curve for the care of very low-birth-weight infants in the United States. The time it took for 75% of NICUs to achieve the shrunken adjusted rate of the best quartile from 2005 highlights the differences in the pace of improvement for the various outcomes that we studied. We are not aware of similar data from other clinical specialties and therefore cannot comment on how the magnitude and pace of improvement in neonatology compares with that of other fields.
We calculated benchmark rates in 2005 and the proportion of NICUs in 2014 below these benchmarks using empirical Bayesian methods applied to individual estimates of NICU standardized morbidity ratios. The estimates for each unit “shrink” toward the overall network mean, with the amount of shrinkage being greater for smaller units, resulting in conservative estimates of the adjusted quantiles. Shrinkage to reduce variation owing to small sample sizes is an accepted method19,20 used by the American College of Surgeons National Surgical Quality Improvement Program21 and other registries.22 Hospitals with fewer infants may appear to perform better through shrinking to the network mean than if they were shrunken to a mean specific to low-volume hospitals.23 There are alternative ways to estimate these benchmarks and proportions, such as squared-error loss ranking procedures or posterior means of an appropriately defined function of the ratios.24 Understanding and comparing the statistical properties of alternative methods, as well as how long a given hospital can sustain a particular quantile rate and the effects of hospital volume, are important areas for future work, particularly if these metrics are to be applied in other clinical fields.
A possible explanation for the improvements we observed is the development and dissemination of evidence-based perinatal and obstetrical care practices and the application of improvement science to translate that evidence into routine practice.25 Use of antenatal corticosteroids,26,27 widespread adoption of less invasive approaches to respiratory support,10 standardized care of very low-birth-weight infants, and improved nutrition management, specifically the recognition of the importance of human milk for feeding preterm infants,28 are examples of evolving practices that can have a measurable effect on these outcomes. The increased adoption of evidence-based practices and their implementation using quality improvement methods may have contributed to the gains in survival without morbidity that we observed.
Since 1995, multidisciplinary teams from more than 550 NICUs have participated in Vermont Oxford Network quality improvement collaboratives. Hundreds more have participated in collaboratives organized by state perinatal organizations29,30 and private neonatology groups.31 Late-onset infection has been a major focus for these efforts, with evidence to suggest that infection in neonatal,32-36 pediatric,37 and adult38 intensive care units can be reduced by participation in quality improvement programs. Neonatology has been at the forefront of applying improvement science to daily care. During the decade under study, many NICU teams have worked together in collaboratives and have tested, implemented, and standardized a variety of potentially better practices designed to reduce the major morbidities we have studied. We refer to them as “potentially better” rather than “better” or “best” practices because a practice is not better or best until it is shown to work in the local context.39 Modeling and simulations based on concepts from complexity science support the suggestion that collaborative learning and learning by doing in the local context are effective strategies for improvement.40,41 The extent to which these efforts have contributed to the improvements we report is uncertain.42
Quiz Ref IDAlthough there were improvements in chronic lung disease, the magnitude of change in rates was smaller than for the other outcomes. Existing interventions for the prevention of chronic lung disease may be less effective than those targeting other outcomes or may be more difficult to reliably implement. There have been 3 cluster randomized clinical trials of quality improvement collaboratives testing interventions that might reduce chronic lung disease.43-45 Although all 3 trials demonstrated changes in practices, only 1 showed a reduction in chronic lung disease.45 Another possibility is that our definition of chronic lung disease is insensitive to detecting changes. Although the definition—receiving supplemental oxygen at 36 weeks’ postnatal age—is widely used in research and clinical reporting,46 it may not capture changes in the more severe cases of chronic lung disease in which infants require either higher levels of supplemental oxygen or continued respiratory assistance. Additional evidence from randomized clinical trials of interventions to prevent chronic lung disease is needed.47
Our study has several potential limitations. First, since this was an uncontrolled observational study, we cannot determine the causes for the improvements reported. Second, the number of hospitals that participated in Vermont Oxford Network increased from 2005 to 2014. We saw nearly identical results when we restricted our analysis to the hospitals that participated in all 10 years, so we do not expect that changes in hospital characteristics contribute to the differences. Third, transferred infants were tracked for survival, but not morbidities, after transfer. The percentage of infants transferred during the study period decreased, which may have resulted in an overestimate of morbidity rates in 2014 relative to 2005 and an underestimate of the proportion of NICUs achieving best quintiles in 2014. Finally, this study includes only members of Vermont Oxford Network and thus may not be generalizable to nonmembers. However, since the Vermont Oxford Network database now enrolls approximately 90% of all very low-birth-weight infants born in the United States each year, it is likely that the results we report are representative of outcomes in the United States.
By 2014, more than 75% of NICUs in the United States had learned to perform as well or better than the best 25% of NICUs performed in 2005 for major morbidities other than chronic lung disease. The challenge is to accelerate the pace of learning and achieve in just a few years what previously took a decade.
Correction: This article was corrected on March 6, 2017, to fix Figure 1.
Accepted for Publication: November 1, 2016.
Corresponding Author: Erika M. Edwards, PhD, Vermont Oxford Network, 33 Kilburn St, Burlington, VT 05401 (firstname.lastname@example.org).
Published Online: January 9, 2017. doi:10.1001/jamapediatrics.2016.4396
Author Contributions: Dr Edwards and Ms Greenberg had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Horbar, Edwards, Soll, Buus-Frank, Buzas.
Acquisition, analysis, or interpretation of data: Horbar, Edwards, Greenberg, Morrow, Buus-Frank, Buzas.
Drafting of the manuscript: Horbar, Edwards, Soll, Buzas.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Edwards, Greenberg, Morrow, Buzas.
Administrative, technical, or material support: Horbar, Buus-Frank.
Study supervision: Horbar, Soll.
Conflict of Interest Disclosures: Drs Horbar, Soll, and Buus-Frank and Mss Greenberg and Morrow are employees of Vermont Oxford Network. Drs Buzas and Edwards are employees of the University of Vermont and receive salary support from Vermont Oxford Network. No other disclosures were reported.
Additional Contributions: We thank the Vermont Oxford Network member hospitals that contributed data used in this study.
Create a personal account or sign in to: