Do differences in infant survival among hospitals and over time affect a prognostic model widely used in extremely preterm birth counseling?
In this prognostic study of most actively treated extremely preterm infants in the United States in 2006 to 2012 and 2013 to 2016, survival increased from 66% to 70%, and model prediction was moderate. The birth hospital and gestational age contributed equally to prediction of survival .
For extremely preterm birth, an area of medicine with substantial variation among hospitals and changing outcomes, prognostic models used in clinical practice may require accounting for local outcomes and periodic updating to remain relevant.
The Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network (NRN) extremely preterm birth outcome model is widely used for prognostication by practitioners caring for families expecting extremely preterm birth. The model provides information on mean outcomes from 1998 to 2003 and does not account for substantial variation in outcomes among US hospitals.
To update and validate the NRN extremely preterm birth outcome model for most extremely preterm infants in the United States.
Design, Setting, and Participants
This prognostic study included 3 observational cohorts from January 1, 2006, to December 31, 2016, at 19 US centers in the NRN (derivation cohort) and 637 US centers in Vermont Oxford Network (VON) (validation cohorts). Actively treated infants born at 22 weeks’ 0 days’ to 25 weeks’ 6 days’ gestation and weighing 401 to 1000 g, including 4176 in the NRN for 2006 to 2012, 45 179 in VON for 2006 to 2012, and 25 969 in VON for 2013 to 2016, were studied. VON cohorts comprised more than 85% of eligible US births. Data analysis was performed from May 1, 2017, to March 31, 2019.
Predictive variables used in the original model, including infant sex, birth weight, plurality, gestational age at birth, and exposure to antenatal corticosteroids.
Main Outcomes and Measures
The main outcome was death before discharge. Secondary outcomes included neurodevelopmental impairment at 18 to 26 months’ corrected age and measures of hospital resource use (days of hospitalization and ventilator use).
Among 4176 actively treated infants in the NRN cohort (48% female; mean [SD] gestational age, 24.2 [0.8] weeks), survival was 63% vs 62% among 3702 infants in the era of the original model (47% female; mean [SD] gestational age, 24.2 [0.8] weeks). In the concurrent (2006-2012) VON cohort, survival was 66% among 45 179 actively treated infants (47% female; mean [SD] gestational age, 24.1 [0.8] weeks) and 70% among 25 969 infants from 2013 to 2016 (48% female; mean [SD] gestational age, 24.1 [0.8] weeks). Model C statistics were 0.74 in the 2006-2012 validation cohort and 0.73 in the 2013-2016 validation cohort. With the use of decision curve analysis to compare the model with a gestational age–only approach to prognostication, the updated model showed a predictive advantage. The birth hospital contributed equally as much to prediction of survival as gestational age (20%) but less than the other factors combined (60%).
Conclusions and Relevance
An updated model using well-known factors to predict survival for extremely preterm infants performed moderately well when applied to large US cohorts. Because survival rates change over time, the model requires periodic updating. The hospital of birth contributed substantially to outcome prediction.
Accurate estimates of prognosis after extremely preterm birth allow practitioners and families to plan for potential outcomes and may guide clinical decision-making.1,2 Since 2008, the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Neonatal Research Network (NRN) has published on its website data describing outcomes of infants born to assist families and practitioners managing extremely preterm birth.3 The website displays outcomes from a model developed using data for infants born at NRN hospitals from 1998 to 2003. The model takes into account 5 factors: an infant’s sex, birth weight, plurality (singleton vs multiple gestation), gestational age at birth, and exposure to antenatal corticosteroids. Prognostic estimates that accounted for these 5 factors were more accurate than those based on gestational age alone.4
The NICHD NRN extremely preterm birth outcome model is widely used to inform perinatal prognostic discussions5 and is cited in guidelines of professional societies.6,7 However, interventions and outcomes for extremely preterm infants have changed since its development.8,9 In addition, studies10-13 have described substantial variation in extremely preterm infant outcomes among US hospitals, including within the NRN.
The aim of this investigation was to update the NICHD NRN extremely preterm birth outcome model to reflect more recent neonatal practices and outcomes. We evaluated the importance of hospital differences in outcomes relative to the other factors included in the model. In addition, we evaluated the updated model using data for most US extremely preterm births from 2006 to 2012 and 2013 to 2016 to assess its performance over time.
In this prognostic study, we included data for inborn infants born at 22 weeks’ 0 days’ to 25 weeks’ 6 days’ gestation and weighing 401 to 1000 g. Infants with major anomalies, with birth weights greater than the 97th percentile for gestational age (raising the possibility that gestational age may have been underestimated14,15), or who were not born at participating hospitals were excluded. These same criteria were used in the development of the original model.4 The institutional review boards at each NRN site approved data collection protocols. Waiver of consent was granted by the institutional review boards at 17 of 19 centers for the collection of in-hospital data and 4 of 19 centers for collection of follow-up data. All data were deidentified. The University of Vermont Institutional Review Board considered this research using the Vermont Oxford Network (VON) deidentified data repository to be not human subjects research. Model derivation, validation, and reporting were conducted according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.16
The new model was derived using data for infants born from April 1, 2006, to December 31, 2012, at 19 centers participating in the NRN. Because outcome data for infants who did not receive any potentially life-sustaining treatment cannot be used to estimate the probability of survival with intensive care,17 the model was derived for infants who received any active treatment after birth. Active treatment was defined as surfactant therapy, endotracheal intubation, ventilatory support (including continuous positive airway pressure, bag-valve-mask ventilation, or mechanical ventilation), epinephrine, or chest compressions.18 This change from the original model, which defined active treatment based solely on administration of mechanical ventilation, reflects changes in practice in the intervening period.4
The updated model was applied to births at 637 US hospitals in the VON database from January 1, 2006, to December 31, 2012, using the derivation cohort inclusion and exclusion criteria. VON included more than 85% of all US infants born at 22 to 25 weeks’ gestation and weighing 400 to 1000 g during the study period.19 Inclusion was not limited by the level or volume of the birth hospital. Although NRN data were derived from hospitals with level III and IV neonatal intensive care units (NICUs), the VON data included infants born at 79 hospitals with level I NICUs (restrictions on assisted ventilatory support), 176 hospitals with level II NICUs (no ventilatory support restrictions and no neonatal surgery), and 382 level III or IV NICUs (no ventilatory support restrictions and perform neonatal surgery). Many infants were transferred to other hospitals during their clinical care. Hospitals participating in both the NRN and VON were excluded from the validation data set for the overlapping years. A second cohort from the VON database comprising infants born from January 1, 2013, to December 31, 2016, from the same hospitals and with the same inclusion and exclusion criteria was used to test the stability of hospital-specific observed and estimated outcomes over time.
Infant data were collected at each hospital using standardized protocols. Gestational age at birth was defined in completed weeks (eg, 23 weeks, representing births at 23 weeks 0 days through 23 weeks 6 days) and determined using standard protocols that prioritized the best obstetric estimate.20 Birth weight was measured in grams and compared with the gestational age– and sex-specific growth curves used in the development of the original model.4,15 Plurality was defined as singleton birth or multiple birth. Exposure to antenatal corticosteroids was defined as a binary variable with exposure including any antenatal receipt of betamethasone or dexamethasone for acceleration of fetal maturation regardless of duration, dose, or timing relative to birth.
Survival was measured as survival to discharge or to 1 year of age if still hospitalized. Data on in-hospital resource use and neurodevelopmental outcomes at 18 to 26 months’ corrected age were available for the NRN derivation cohort and are described in eTable 1 and eTable 2 in the Supplement.
Resource use was described in infant-days of mechanical ventilation per surviving infant and infant-days of hospitalization per surviving infant. Neurodevelopmental outcomes were ascertained by standardized, annually certified examiners who were not blinded to infants’ medical information. They were classified based on cognitive development assessed with the Bayley Scales of Infant and Toddler Development, Third Edition (BSID-III), gross motor function as assessed by the Gross Motor Function Classification System (GMFCS), and the presence of cerebral palsy and hearing and vision impairment. Profound neurodevelopmental impairment was defined as a BSID-III cognitive score that was untestable or a GMFCS level of 5 (on a scale of 0 [normal] to 5 [most impaired]). Moderate-severe neurodevelopmental impairment was defined as a BSID-III cognitive score of less than 85 (ie, >1 SD below the scale mean; mean [SD], 100 ), moderate or severe cerebral palsy, a GMFCS level of 2 to 5, bilateral blindness, or severe hearing impairment that could not be corrected with bilateral amplification.
We compared descriptive characteristics of infants in the derivation and validation cohorts with the characteristics of infants in the cohort used to develop the original model (1998-2003).4 Analysis of the NRN data (L.L.) was conducted using SAS Enterprise Guide, version 7.15 (SAS Institute Inc) and Stata/MP, version 16.0 (StataCorp). Analysis of VON data (L.T.G.) was conducted using R, version 3.3.2 (R Foundation for Statistical Computing). Using the NRN data set, we derived a multivariable logistic mixed model with survival as the dependent outcome; birth weight, gestational age, plurality, and antenatal corticosteroid exposure as independent variables; and hospital of birth as a random intercept. These same factors were included in the original model.4 Neurodevelopmental outcomes and resource use were described for actively treated infants stratified by their probability of survival.
For external validation, the updated model was applied to eligible infants in nonoverlapping hospitals in the 2006 to 2012 VON cohort. We estimated random intercepts for all VON hospitals using k-fold cross-validation to evaluate model fit. Modeling discrimination was assessed using the C statistic. For the outcome of survival, the C statistic represents the probability that for any randomly selected pair of individuals in which one lives and the other dies, the model estimates a higher probability for the one who lives.21,22 Model calibration was assessed by comparing estimated and observed outcomes for the whole sample across several subgroups of infant characteristics.
To complement discrimination and calibration metrics, we also performed a decision curve analysis that compared the predicted outcomes of treating all or no patients by gestational age—the approach recommended in some guidelines23—with treating a targeted group based on model-estimated survival. The decision curve analysis presents various predicted survival rates above which infants might be selected to receive active treatment (ie, threshold probabilities) and presents the predictive net benefit of using that threshold. The predictive net benefit was calculated as follows: [proportion of treated infants who lived] – [(proportion of treated infants who died) × (threshold probability)/(1-threshold probability)].24,25
To evaluate the importance of hospital differences in outcomes relative to the other factors included in the model, we compared observed and model-estimated outcomes with and without accounting for the hospital-specific intercept term in the estimate. We calculated the relative contribution of model variables to outcome estimation using the differences between the log-likelihood value of the full model and of models without each variable.26 To assess the stability of hospital-specific infant outcome estimates over time, the hospital-specific model with intercept terms derived in 2006 to 2012 was applied to the 2013 to 2016 cohort and observed and estimated outcomes were compared.
As a sensitivity analysis, we also compared the outcomes estimated when the model was applied to all infants born in the validation cohorts regardless of whether they were actively treated, as previously performed.4 The results of this analysis estimated the maximum potential survival that might result if all infants in the cohorts were actively treated.4 Data analysis was performed from May 1, 2017, to March 31, 2019.
Among 4176 actively treated infants in the NRN cohort (48% female; mean [SD] gestational age, 24.2 [0.8] weeks), survival was 63% vs 62% among 3702 infants in the era of the original model (47% female; mean [SD] gestational age, 24.2 [0.8] weeks). In the concurrent (2006-2012) VON cohort, survival was 66% among 45 179 actively treated infants (47% female; mean [SD] gestational age, 24.1 [0.8] weeks) and 70% among 25 969 infants from 2013 to 2016 (48% female; mean [SD] gestational age, 24.1 [0.8] weeks). Characteristics of the infants in each cohort are summarized in Table 1.
Odds ratios for each variable included in the updated and original 5-factor models are shown in Figure 1. Except for the increase in the point estimate for survival for infants born at 22 weeks’ compared with 25 weeks’ gestation, there was little change in other model coefficients.
Model-estimated survival outcomes for infants in the 2006 to 2012 NRN and VON validation cohorts by gestational age at birth are shown in eFigure 1 in the Supplement. The mean (SD) estimated probability of survival for actively treated infants born at 22 weeks’ gestation in the NRN was 25% (11%). However, there was considerable variation in the probability of survival for infants born during each gestational age week, with a range of 8% to 61% based on the other factors in the model. Likewise, the mean probability of survival for actively treated infants born at 23 weeks’ gestation in the NRN was 39% (14%), with a range of 9% to 76%.
When accounting for the resources used for infants who died, for each infant who lived with an estimated survival probability of 11% to 20%, 153 infant-days of mechanical ventilation and 280 infant-days of hospitalization were used (eTable 1 in the Supplement). For each infant with a probability of survival greater than 80%, 28 infant-days of mechanical ventilation and 115 infant-days of hospitalization were used per infant who lived.
Of 4176 actively treated infants in the NRN cohort, 3927 (94%) had follow-up outcomes known at 18 to 26 months’ corrected age (eTable 2 in the Supplement). Rates of profound neurodevelopmental impairment were 1% among infants with an estimated survival probability greater than 80% and 11% among infants with an estimated survival probability of 11% to 20%. Rates of moderate-severe neurodevelopmental impairment (including profound impairment) were 21% among infants with an estimated survival probability greater than 80% and 78% among infants with an estimated survival probability of 11% to 20%. No actively treated infants with a probability of survival of 10% or less (n = 6) lived to follow-up.
Estimates of model discrimination and calibration when the model was applied to the 2006 to 2012 VON cohort are given in Table 2. The original 5-factor model for mortality had a C statistic of 0.75 in the previously published derivation cohort of NRN hospitals (1998-2003)4; discrimination was similar in the 2006 to 2012 VON cohort (C statistic, 0.73). By comparison, the updated model incorporating hospital-specific intercepts had a C statistic of 0.74 in the 2006 to 2012 VON cohort.
Compared with the original model and the use of gestational age mean outcomes, the updated model that accounted for hospital of birth was better calibrated to recent outcomes (Figure 2). When model-estimated and observed outcomes were compared, the calibration slope was 0.99 and the intercept was 0.03 (eFigure 2 in the Supplement). Decision curve analysis demonstrated that the updated model provided greater predictive net benefit vs the original model for a wide range of decision thresholds (eFigure 3 in the Supplement). The model that included hospital of birth outperformed both the original model and the updated model that did not account for hospital of birth. This model found a greater predictive net benefit at all gestational ages across a range of threshold probabilities. The predictive net benefit of the updated model was most pronounced at 22 and 23 weeks’ gestation compared with a gestational age–based approach.
As indicated in Table 3, hospital of birth contributed to model estimation of survival equally as much as gestational age (20%). Factors besides hospital of birth and gestational age contributed 60% of model predictive ability. When the model without hospital of birth was applied to individual hospitals in the 2006 to 2012 VON cohort with more than 20 infants during the study period, 125 of 386 (32%) hospitals had an observed hospital survival rate significantly different from the estimated rate (eFigure 4 in the Supplement), suggesting that this model did not adequately account for variation in outcomes across hospitals.
To test whether hospital-specific terms established for the 2006 to 2012 VON cohort could be used prospectively for the same hospitals, the NRN model applied to VON births from 2006 to 2012 was also applied to VON births from 2013 to 2016. For the same 386 VON hospitals, 34 (9%) had observed outcomes significantly different from predicted outcomes (eFigure 5 in the Supplement), with 30 of 34 hospitals having better than predicted outcomes, suggesting an improvement in outcomes over time not accounted for by the model. The C statistic of the model in the 2013 to 2016 cohort was 0.73.
The predicted rate of survival among all infants born in study hospitals, assuming all infants were actively treated (ie, the maximum potential rate), approximated the observed and predicted survival rates among infants who were actively treated (eTable 3 in the Supplement). Of note, rates of both active treatment and infant survival increased from the 2006 to 2012 period to the 2013 to 2016 period. Predicted survival rates also increased during the period in association with increased use of antenatal corticosteroids (Table 1).
The update to the NICHD NRN extremely preterm birth outcome model demonstrated moderate discrimination and calibration when applied to most actively treated extremely preterm infants in the United States. Compared with gestational age–only approaches, the model had a predictive advantage by decision curve analysis. The incorporation of hospital of birth into model estimates contributed substantially to outcome estimation. Although the association of the 5 infant characteristics in the model with infant outcomes remained relatively stable from the original era to the most recent one, outcomes improved during the 2 periods studied, suggesting a need to periodically recalibrate the model, including for hospital-specific outcomes.
Because of variation in hospital outcomes, some practitioners have advocated for using only local statistics for predicting infant survival; however, given the small number of extremely preterm infants born at any given hospital, this approach may result in imprecise estimates and may preclude taking into account important prognostic variables, such as infant sex or antenatal corticosteroid exposure.7,27 In our study, approximately 60% of model estimation of survival resulted from variables besides hospital of birth and gestational age.
Our approach accounted for hospital-specific outcomes while also benefiting from large data sets for deriving and validating the prognostic value of several important infant variables. The contribution of the birth hospital to model prediction may represent differences in practices or case mix among hospitals not accounted for by other model variables. Because prognostic estimates and treatment decisions are made within individual hospitals, our model has been calibrated to individual hospitals as recommended by several methodologists.28-30 Our approach assumes that past hospital-specific neonatal survival rates reliably predict future survival rates, as previously shown in a cohort of US hospitals.31 Our data suggest that outcomes of extremely preterm infants at many US hospitals vary substantially from the mean.
The updated model assigned more accurate probabilities of infant outcomes than the original model when applied to births in recent large US cohorts (Figure 2 and eFigure 2 in the Supplement). However, the improved model calibration should be distinguished from model discrimination as represented by the C statistic, a measure of the ability of the model to separate individuals who develop the outcome of interest from those who do not. The C statistic of the updated model was not substantially different from that observed when deriving the original model4 and in independent validations of that model32,33 and demonstrated the continued difficulty of estimating outcomes for extremely preterm infants at the time of birth. Historically, signs from physical examination, such as fusion of the eyelids, were used to determine viability at extremely preterm gestations, although their utility has not been borne out.34,35 The use of the combined 5 factors in the model was again found to outperform gestational age alone as a tool for predicting survival,4 and the predictive benefits of a model-based vs gestational age–based approach were demonstrated in the decision curve analysis. In the future, other methods, such as the use of biochemical markers of infant maturity, may enhance our ability to estimate infant outcomes.36
Although we attempted to maintain the same form as the original 5-factor model, which has been in use for more than a decade and is familiar to many practitioners, the updated model differs from the original in key ways. The study deriving the original model used mechanical ventilation that required endotracheal intubation as a marker of active treatment for infants born in 1999 to 2003. We defined active treatment using a broader definition that accounted for changes in the use of noninvasive ventilation37; 2.5% of 2394 infants in the derivation cohort did not require endotracheal intubation and mechanical ventilation. We also defined survival as occurring at discharge rather than at follow-up, the time point used in the original model, to validate the model in the VON cohorts, which included data through hospital discharge. These changes may explain the small difference in survival seen between the original 1998 to 2003 cohort and the 2006 to 2012 NRN cohort included in this study. In addition, information on timing of antenatal corticosteroid administration was not available for the cohorts used to update the model; the original model included antenatal corticosteroid exposure only if taking place within 7 days of birth.
The updated model has important limitations, many of which are shared by the original model. First, the model includes birth weight, which can be estimated only imprecisely before birth.38 Second, the best obstetric estimates of gestational age for most pregnancies (except those conceived through in vitro fertilization) have a margin of error of at least 5 days.20 Third, our data do not provide information on reasons for withholding or withdrawing care, and actively treated infants may be different in unmeasured ways from their counterparts despite having similar probabilities of survival based on our model.39 Decisions about whether to provide life support at the margins of viability involve more than statistical calculations and must take into account clinical judgment and the values, beliefs, and concerns of each family.40 Fourth, prognosis is dynamic and can change with new events after birth41; the outcomes here reflect prognostic estimates for infants at the time of birth. Fifth, we did not have data on the causes of hospital variation in outcomes. Further research is needed to identify explanations for the observed variation. Sixth, although we evaluated the performance of the updated model among infants born several years after those whose data were used for model development, we were unable to provide estimates regarding how often the model should be periodically recalibrated to account for changing outcomes over time.
Our final model explicitly incorporated the hospital of birth in outcome estimates, which was possible because VON included data for most eligible extremely preterm infants born in the United States during the period. Hospital-specific survival estimates for each hospital participating (eAppendix in Supplement) in VON are available to that hospital through the VON website.42 The NICHD has published an updated model providing the mean and range of hospital outcomes on their website available to the public.43
The updated NICHD NRN extremely preterm birth outcome model continues to demonstrate the importance of considering more than gestational age when evaluating prognosis at the time of birth. The hospital of birth contributed substantially to outcome prediction. As outcomes and practices continue to change, prognostic models used for extremely preterm birth may require accounting for local outcomes and periodic updating to remain relevant.
Accepted for Publication: December 12, 2019
Corresponding Author: Matthew A. Rysavy, MD, PhD, Stead Family Department of Pediatrics, University of Iowa, 200 Hawkins Dr, Iowa City, IA 52242 (email@example.com).
Published Online: March 2, 2020. doi:10.1001/jamapediatrics.2019.6294
Author Contributions: Dr Li and Ms Greenberg had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Rysavy, Horbar, Bell, Tyson, Carlo, Younge, Walsh, Higgins.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Rysavy, Li.
Critical revision of the manuscript for important intellectual content: Rysavy, Horbar, Bell, Greenberg, Tyson, Patel, Carlo, Younge, Green, Edwards, Hintz, Walsh, Buzas, Das, Higgins.
Statistical analysis: Li, Greenberg, Green, Edwards, Walsh, Buzas, Das.
Obtained funding: Bell, Carlo, Walsh.
Administrative, technical, or material support: Bell, Patel, Walsh, Higgins.
Supervision: Horbar, Bell, Tyson, Carlo, Hintz, Walsh, Higgins.
Conflict of Interest Disclosures: Dr Rysavy reported receiving grants from the National Institutes of Health (NIH) and travel funds from the Vermont Oxford Network (VON) during the conduct of the study. Dr Horbar reported receiving personal fees from VON outside the submitted work and serving as an unpaid member of the Board of Directors of VON. Dr Bell reported receiving grants from the NIH. Dr Tyson reported receiving grants from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) during the conduct of the study and grants from the NICHD outside the submitted work. Dr Patel reported receiving grants from the NIH during the conduct of the study. Dr Carlo served as a paid member of the board of directors of Mednax and received grants from the NIH during the study. Dr Younge reported receiving grants from the NIH during the conduct of the study. Dr Green reported receiving grants from the NICHD during the conduct of the study. Dr Edwards reported receiving grants from VON during the conduct of the study. Dr Hintz reported receiving grants from the NICHD during the conduct of the study. Dr Walsh reported receiving grants from the NIH. Dr Das reported receiving grants from the NICHD and the NIH during the conduct of the study. No other disclosures were reported.
Funding/Support: This study was supported by grants F32 HD098782 (Dr Rysavy), UG1 HD027904, UG1 HD021364 (Dr Walsh), UG1 HD027853, UG1 HD040492, UG1 HD027851, UG1 HD027856, UG1 HD021373, UG1 HD068278, U24 HD095254 (Dr Das), UG1 HD027880, UG1 HD053119, UG1 HD034216 (Dr Carlo), UG1 HD053109 (Dr Bell), UG1 HD053089, UG1 HD068244, UG1 HD068263, UG1 HD040689, UG1 HD053124, UG1 HD021385, and UG1 HD027871 from the NIH and the NICHD, which supported for the Neonatal Research Network’s (NRN’s) Generic Database and Follow-up Study and VON through cooperative agreements.
Role of the Funder/Sponsor: The NICHD staff had input into the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.
Group Members: The following investigators with the NICHD NRN, in addition to those listed as authors, participated in this study and are listed by center. NRN Steering Committee Chair: Michael S. Caplan, MD: University of Chicago, Pritzker School of Medicine (2006-2011); Richard A. Polin, MD: Division of Neonatology, College of Physicians and Surgeons, Columbia University (2011-present). Alpert Medical School of Brown University and Women & Infants Hospital of Rhode Island: Abbot R. Laptook, MD; Angelita M. Hensman, PhD, RNC-NIC; William Oh, MD; Kristin M. Basso, MaT, RN; Elisa Vieira, BSN, RN; Martin Keszler, MD; Betty R. Vohr, MD; Elisabeth C. McGowan, MD; Robert Burke, MD; Melinda Caskey, MD; Katharine Johnson, MD; Barbara Alksninis, PNP; Mary Lenore Keszler, MD; Theresa M. Leach, Med, CAES; Bonnie E. Stephens, MD; Victoria E. Watson, MS, CAS; Suzy Ventura; Andrea M. Knoll. Case Western Reserve University, Rainbow Babies & Children's Hospital: Anna Marie Hibbs, MD; Deanne E. Wilson-Costello, MD; Nancy S. Newman, BA, RN; Bonnie S. Siner, RN; Monika Bhola, MD; Gulgun Yalcinkaya, MD; Harriet G. Friedman, MA. Cincinnati Children's Hospital Medical Center, University Hospital, and Good Samaritan Hospital: Kurt Schibler, MD; Kate Bridges, MD; Barbara Alexander, RN; Cathy Grisby, BSN, CCRC; Jody Hessling, RN; Estelle E. Fischer, MHSA MBA; Lenora D. Jackson, CRC; Kristin Kirker, CRC; Holly L. Mincey, RN, BSN; Greg Muthig, BS; Teresa L. Gratton, PA; Jean J. Steichen, MD; Kimberly Yolton, PhD. Duke University School of Medicine, University Hospital, University of North Carolina, and Duke Regional Hospital: C. Michael Cotten, MD, MHS; Ronald N. Goldberg, MD; Ricki F. Goldstein, MD; Kimberley A. Fisher, PhD, FNP-BC, IBCLC; Kathy J. Auten, MSHS; Katherine A. Foy, RN; Sandra Grimes, RN, BSN; Joanne Finkle, RN, JD; Kathryn E. Gustafson, PhD; Melody B. Lohmeyer, RN, MSN; Matthew M. Laughon, MD MPH; Carl L. Bose, MD; Janice Bernhardt, MS RN; Gennie Bose, RN; Cindy Clark, RN. Emory University, Children’s Healthcare of Atlanta, Grady Memorial Hospital: David P. Carlton, MD; Barbara J. Stoll, MD; Ellen C. Hale, RN, BS, CCRC; Yvonne C. Loggins, RN, BSN; Ira Adams-Chapman, MD; Maureen Mulligan LaRossa, RN; Sheena L. Carter, PhD. NICHD: Stephanie Wilson Archer, MA. Indiana University, University Hospital, Methodist Hospital, Riley Hospital for Children, and Eskenazi Health: Gregory M. Sokol, MD; Brenda B. Poindexter, MD, MS; Anna Maria Dusick, MD (deceased); Dianne E. Herron, RN, CCRC; Abbey C. Hines, PsyD; Carolyn Lytle, MD, MPH; Lucy C. Miller, RN, BSN, CCRC; Heike M. Minnich, PsyD, HSPP; Lu Ann Papile, MD; Leslie Dawn Wilson, BSN, CCRC. McGovern Medical School at The University of Texas Health Science Center at Houston, Children's Memorial Hermann Hospital, and Lyndon Baines Johnson General Hospital/Harris County Hospital District: Kathleen A. Kennedy, MD, MPH; Nora I. Alaniz, BS; Katrina Burson, RN, BSN; Patricia W. Evans, MD; Beverly Foley Harris, RN, BSN; Margarita Jiminez, MD, MPH; Anna E. Lis, RN, BSN; Sarah Martin, RN, BSN; Georgia E. McDavid, RN; Brenda H. Morris, MD; M. Layne Poundstone, RN, BSN; Peggy Robichaux, RN, BSN; Saba Siddiki, MD; Maegan C. Simmons, RN; Patti L. Pierce Tate, RCP; Sharon L. Wright, MT (ASCP). Nationwide Children’s Hospital and the Ohio State University Medical Center: Pablo J. Sánchez, MD; Leif D. Nelin, MD; Sudarshan R. Jadcherla, MD; Patricia Luzader, RN; Christine A. Fortney, PhD, RN; Gail E. Besner; Nehal A. Parikh, MD. RTI International: Marie G. Gantz, PhD; W. Kenneth Poole, PhD (deceased); Dennis Wallace, PhD; Jamie E. Newman, PhD, MPH; Jeanette O’Donnell Auman, BS; Margaret M. Crawford, BS, CCRP; Carolyn M. Petrie Huitema, MS, CCRP; Kristin M. Zaterka-Baxter, RN, BSN, CCRP. Stanford University, Dominican Hospital, El Camino Hospital, and Lucile Packard Children's Hospital: Krisa P. Van Meurs, MD; David K. Stevenson, MD; Marian M. Adams, MD; M. Bethany Ball, BSc, CCRC; Andrew W. Palmquist, RN, BSN; Melinda S. Proud, RCP; Barbara Bentley, PsychD, MSEd; Elizabeth Bruno, PhD; Maria Elena DeAnda, PhD; Anne M. DeBattista, RN, PNP; Beth Earhart, PhD; Jean G. Kohn, MD, MPH; Casey E. Krueger, PhD; Renee P. Pyle, PhD; Heather Taylor, PhD; Hali E. Weiss, MD. Tufts Medical Center, Floating Hospital for Children: Ivan D. Frantz III, MD; John M. Fiascone, MD; Brenda L. MacKinnon, RNC; Ellen Nylen, RN, BSN; Anne Furey, MPH; Elisabeth C. McGowan, MD; Cecelia E. Sibley, PT, MHA; Ana K. Brussa, MS, OTR/L. University of Alabama at Birmingham Health System and Children’s Hospital of Alabama: Namasivayam Ambalavanan, MD; Myriam Peralta-Carcelen, MD, MPH; Monica V. Collins, RN, BSN, MaEd; Shirley S. Cosby, RN, BSN; Fred J. Biasini, PhD; Kathleen G. Nelson, MD; Cryshelle S. Patterson, PhD; Vivien A. Phillips, RN, BSN; Sally Whitley, MA, OTR-L, FAOTA; Carin Kiser, MD; Leigh Ann Smith, CRNP; Richard V. Rector, PhD; Sarah Ryan, PhD; Kristy Domnanovich, PhD; Leslie Rodrigues, PhD; Sheree Chapman York, PT, DPT, MS, PCS. University of Iowa and Mercy Medical Center: Jane E. Brumbaugh, MD; Tarah T. Colaizy, MD, MPH; Dan L. Ellsbury, MD; John A. Widness, MD; Michael J. Acarregui, MD, MBA; Karen J. Johnson, RN, BSN; Donia B. Campbell, RNC-NIC; Diane L. Eastman, RN, CPNP, MA. University of New Mexico Health Sciences Center: Kristi L. Watterberg, MD; Robin K. Ohls, MD; Janell F. Fuller, MD; Conra Backstrom Lacy, RN; Rebecca A. Montman, BSN, RNC; Jean R. Lowe, PhD; Andrea Freeman Duncan, MD; Sandra Brown, BSN; Carol Hartenberger, BSN, MPH. University of Pennsylvania, Hospital of the University of Pennsylvania, Pennsylvania Hospital, and Children's Hospital of Philadelphia: Barbara Schmidt, MD, MSc; Haresh Kirpalani, MB, MSc; Sara B. DeMauro, MD, MSCE; Aasma S. Chaudhary, BS, RRT; Soraya Abbasi, MD; Toni Mancini, RN, BSN, CCRC; Dara Cucinotta. University of Rochester Medical Center, Golisano Children's Hospital, and the University at Buffalo Women's and Children's Hospital of Buffalo: Dale L. Phelps, MD; Gary J. Myers, MD; Linda J. Reubens, RN, CCRC; Erica Burnell, RN; Diane Hust, MS, RN, CS; Julie Babish Johnson, MSW; Rosemary L. Jensen; Emily Kushner, MA; Joan Merzbach, LMSW; Kelley Yost, PhD; Lauren Zwetsch, RN, MS, PNP; Satyan Lakshminrusimha, MD; Anne Marie Reynolds, MD, MPH; Osman Farooq, MD; Ashley Williams, MS, Ed; William A. Zorn, PhD. University of Texas Southwestern Medical Center at Dallas, Parkland Health & Hospital System, and Children's Medical Center Dallas: Pablo J. Sánchez, MD; Roy J. Heyne, MD; Walid A. Salhab, MD; Charles R. Rosenfeld, MD; Luc P. Brion, MD; LiJun Chen, RN, PhD; Alicia Guzman; Melissa H. Leps, RN; Nancy A. Miller, RN; Diana M. Vasil, MSN, BSN, RNC, NIC; Lizette E. Torres, RN; Gaynelle Hensley, RN; Sally S. Adams, MS, RN, CPNP; Linda A. Madden, RN CPNP; Elizabeth Heyne, PsyD PA-C; Janet S. Morgan, RN; Catherine Twell Boatman, MS, CIMI. University of Utah University Hospital, Intermountain Medical Center, LDS Hospital, and Primary Children's Medical: Roger G. Faix, MD; Bradley A. Yoder, MD; Karen A. Osborne, RN BSN CCRC; Shawna Baker, RN; Karie Bird, RN, BSN; Jill Burnett, RNC, BSN; Jennifer J. Jensen, RN, BSN; Cynthia Spencer, RNC, BSN; Mike Steffen, PhD; Kimberlee Weaver-Lewis, RN, MS; Sarah Winter, MD; Karen Zanetti, RN. Wayne State University, University of Michigan, Hutzel Women’s Hospital, Sinai-Grace Hospital, and Children’s Hospital of Michigan: Seetha Shankaran, MD; Sanjay Chawla, MD, Girija Natarajan, MD, Monika Bajaj, MD, Beena Sood, MD, Athina Pappas, MD; John Barks, MD; Rebecca Bara, RN, BSN; Laura A. Goldston, MA; Mary Johnson, RN, BSN; Laura Sumner, RN, BSN; Kara Sawaya, RN, BSN; Kathleen Weingarden, RN, BSN; Mary Christensen, RT; Stephanie Wiggins, MS; Daine F. White, RN; Eunice Woldt, RN, MSN. Yale University, Yale-New Haven Children’s Hospital, and Bridgeport Hospital: Richard A. Ehrenkranz, MD; Harris Jacobs, MD; Christine G. Butler, MD; Patricia Cervone, RN; Sheila Greisman, RN; Monica Konstantino, RN, BSN; JoAnn Poulsen, RN; Janet Taft, RN, BSN; Joanne Williams, RN, BSN; Elaine Romano, MSN.
Disclaimer: The comments and views of the authors do not necessarily represent the views of NICHD, the NIH, the Department of Health and Human Services, or the US government.
RD; National Institute of Child Health and Human Development Neonatal Research Network. Intensive care for extreme prematurity: moving beyond gestational age. N Engl J Med
. 2008;358(16):1672-1681. doi:10.1056/NEJMoa073059
et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network. Causes and timing of death in extremely premature infants from 2000 through 2011. N Engl J Med
. 2015;372(4):331-340. doi:10.1056/NEJMoa1403489
L. Survival for inborn infants weighing 501-1500 grams at birth: Variation among centers. Pediatr Res
. 1990;27(4):245A.Google Scholar
et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network. Neonatal outcomes of extremely preterm infants from the NICHD Neonatal Research Network. Pediatrics
. 2010;126(3):443-456. doi:10.1542/peds.2009-2959
et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network. Individual and center-level factors affecting mortality among extremely low birth weight infants. Pediatrics
. 2013;132(1):e175-e184. doi:10.1542/peds.2012-3707
et al; Fetal/Infant Health Study Group of the Canadian Perinatal Surveillance System. A new and improved population-based Canadian reference for birth weight for gestational age. Pediatrics
. 2001;108(2):E35. doi:10.1542/peds.108.2.e35
KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ
. 2015;350:g7594. doi:10.1136/bmj.g7594
et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network. Between-hospital variation in treatment and outcomes in extremely preterm infants. N Engl J Med
. 2015;372(19):1801-1811. doi:10.1056/NEJMoa1410689
et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ
. 2016;353:i3140. doi:10.1136/bmj.i3140
TH, van Klei
Y. Prediction models for clustered data: comparison of a random intercept and standard regression model. BMC Med Res Methodol
. 2013;13:19. doi:10.1186/1471-2288-13-19
Y, Van Huffel
D, Van Calster
B. Does ignoring clustering in multicenter data influence the performance of prediction models? a simulation study. Stat Methods Med Res
. 2018;27(6):1723-1736. doi:10.1177/0962280216668555
CM, Van Calster
B. Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting. Diagn Progn Res
. 2019;3:6. doi:10.1186/s41512-019-0046-9
LW; Victorian Infant Collaborative Study Group. Predicting death or major neurodevelopmental disability in extremely preterm infants born in Australia. Arch Dis Child Fetal Neonatal Ed
. 2013;98(3):F201-F204. doi:10.1136/archdischild-2012-301628
JE. Infant outcomes after periviable birth: external validation of the neonatal research network estimator with the BEAM trial. Am J Perinatol
. 2016;33(6):569-576.PubMedGoogle Scholar
et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network. Trends in care practices, morbidity, and mortality of extremely preterm neonates, 1993-2012. JAMA
. 2015;314(10):1039-1051. doi:10.1001/jama.2015.10244
et al; Generic Database; Subcommittees of the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network. Outcome trajectories in extremely preterm infants. Pediatrics
. 2012;130(1):e115-e125. doi:10.1542/peds.2011-3693