Assessment of an Updated Neonatal Research Network Extremely Preterm Birth Outcome Model in the Vermont Oxford Network | Neonatology | JAMA Pediatrics | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Odds Ratios for Survival in the Original and Updated Models
Odds Ratios for Survival in the Original and Updated Models

Odds ratios for birth at 22, 23, and 24 weeks’ gestation used birth at 25 weeks’ gestation as the reference category. Odds ratios for birth weight are per 100-g increase. Error bars indicate 95% CIs for survival.

Figure 2.  Observed and Model-Predicted Survival Rates in the Vermont Oxford Network (VON) 2006-2012 Cohort
Observed and Model-Predicted Survival Rates in the Vermont Oxford Network (VON) 2006-2012 Cohort

Observed and model-predicted survival rates in the VON 2006-2012 cohort were compared for infants grouped by birth weight quartile for gestational age (<25%, 25%-75%, or >75%), sex (male or female), plurality (singleton or multiple gestation), and exposure to antenatal steroids (yes or no) as presented in the development of the original model.4

Table 1.  Characteristics of Actively Treated Infants in Each Cohorta
Characteristics of Actively Treated Infants in Each Cohorta
Table 2.  Comparison of the Updated and Original 5-Factor Models in the VON 2006-2012 Cohort
Comparison of the Updated and Original 5-Factor Models in the VON 2006-2012 Cohort
Table 3.  Relative Contribution of Variables to the Multivariable Modela
Relative Contribution of Variables to the Multivariable Modela
1.
Rysavy  MA, Tyson  JE.  The problem and promise of prognosis research.   JAMA Pediatr. 2016;170(5):411-412. doi:10.1001/jamapediatrics.2015.4871 PubMedGoogle ScholarCrossref
2.
Rysavy  MA.  Prognosis as an intervention.   Clin Perinatol. 2018;45(2):231-240. doi:10.1016/j.clp.2018.01.009 PubMedGoogle ScholarCrossref
3.
NICHD Neonatal Research Network (NRN). Extremely preterm birth outcome data. https://www.nichd.nih.gov/about/org/der/branches/ppb/programs/epbo/Pages/epbo_case.aspx. Accessed July 1, 2019.
4.
Tyson  JE, Parikh  NA, Langer  J, Green  C, Higgins  RD; National Institute of Child Health and Human Development Neonatal Research Network.  Intensive care for extreme prematurity: moving beyond gestational age.   N Engl J Med. 2008;358(16):1672-1681. doi:10.1056/NEJMoa073059 PubMedGoogle ScholarCrossref
5.
Myers  P, Laventhal  N, Andrews  B, Lagatta  J, Meadow  W.  Population-based outcomes data for counseling at the margin of gestational viability.   J Pediatr. 2017;181:208-212.e4. doi:10.1016/j.jpeds.2016.10.021 PubMedGoogle ScholarCrossref
6.
American College of Obstetricians and Gynecologists, Society for Maternal-Fetal Medicine.  Obstetric Care Consensus No. 6: periviable birth.   Obstet Gynecol. 2017;130(4):e187-e199. . doi:10.1097/AOG.0000000000002352 PubMedGoogle ScholarCrossref
7.
Cummings  J; Committee on Fetus and Newborn.  Antenatal counseling regarding resuscitation and intensive care before 25 weeks of gestation.   Pediatrics. 2015;136(3):588-595. doi:10.1542/peds.2015-2336 PubMedGoogle ScholarCrossref
8.
Horbar  JD, Carpenter  JH, Badger  GJ,  et al.  Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009.   Pediatrics. 2012;129(6):1019-1026. doi:10.1542/peds.2011-3028 PubMedGoogle ScholarCrossref
9.
Patel  RM, Kandefer  S, Walsh  MC,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Causes and timing of death in extremely premature infants from 2000 through 2011.   N Engl J Med. 2015;372(4):331-340. doi:10.1056/NEJMoa1403489 PubMedGoogle ScholarCrossref
10.
Horbar  JD, Onstad  L.  Survival for inborn infants weighing 501-1500 grams at birth: Variation among centers.   Pediatr Res. 1990;27(4):245A.Google Scholar
11.
Stoll  BJ, Hansen  NI, Bell  EF,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Neonatal outcomes of extremely preterm infants from the NICHD Neonatal Research Network.   Pediatrics. 2010;126(3):443-456. doi:10.1542/peds.2009-2959 PubMedGoogle ScholarCrossref
12.
Alleman  BW, Bell  EF, Li  L,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Individual and center-level factors affecting mortality among extremely low birth weight infants.   Pediatrics. 2013;132(1):e175-e184. doi:10.1542/peds.2012-3707 PubMedGoogle ScholarCrossref
13.
Horbar  JD, Edwards  EM, Greenberg  LT,  et al.  Variation in performance of neonatal intensive care units in the United States.   JAMA Pediatr. 2017;171(3):e164396. doi:10.1001/jamapediatrics.2016.4396 PubMedGoogle Scholar
14.
Kramer  MS, McLean  FH, Boyd  ME, Usher  RH.  The validity of gestational age estimation by menstrual dating in term, preterm, and postterm gestations.   JAMA. 1988;260(22):3306-3308. doi:10.1001/jama.1988.03410220090034 PubMedGoogle ScholarCrossref
15.
Kramer  MS, Platt  RW, Wen  SW,  et al; Fetal/Infant Health Study Group of the Canadian Perinatal Surveillance System.  A new and improved population-based Canadian reference for birth weight for gestational age.   Pediatrics. 2001;108(2):E35. doi:10.1542/peds.108.2.e35 PubMedGoogle Scholar
16.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.   BMJ. 2015;350:g7594. doi:10.1136/bmj.g7594 PubMedGoogle ScholarCrossref
17.
Rysavy  MA, Marlow  N, Doyle  LW,  et al.  Reporting outcomes of extremely preterm births.   Pediatrics. 2016;138(3):e20160689. doi:10.1542/peds.2016-0689 PubMedGoogle Scholar
18.
Rysavy  MA, Li  L, Bell  EF,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Between-hospital variation in treatment and outcomes in extremely preterm infants.   N Engl J Med. 2015;372(19):1801-1811. doi:10.1056/NEJMoa1410689 PubMedGoogle ScholarCrossref
19.
United States Department of Health and Human Services (US DHHS). CDC WONDER online database: natality public-use data 2007-2017. https://wonder.cdc.gov/natality-current.html. Accessed Feb 1, 2019.
20.
Spong  CY.  Defining “term” pregnancy: recommendations from the Defining “Term” Pregnancy Workgroup.   JAMA. 2013;309(23):2445-2446. doi:10.1001/jama.2013.6235 PubMedGoogle ScholarCrossref
21.
Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.   Stat Med. 1996;15(4):361-387. doi:10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4 PubMedGoogle ScholarCrossref
22.
Riley  RD, Ensor  J, Snell  KIE,  et al.  External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges.   BMJ. 2016;353:i3140. doi:10.1136/bmj.i3140 PubMedGoogle ScholarCrossref
23.
Guillén  Ú, Weiss  EM, Munson  D,  et al.  Guidelines for the management of extremely premature deliveries: a systematic review.   Pediatrics. 2015;136(2):343-350. doi:10.1542/peds.2015-0542 PubMedGoogle ScholarCrossref
24.
Vickers  AJ, Van Calster  B, Steyerberg  EW.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.   BMJ. 2016;352:i6. doi:10.1136/bmj.i6 PubMedGoogle ScholarCrossref
25.
Vickers  AJ, Elkin  EB.  Decision curve analysis: a novel method for evaluating prediction models.   Med Decis Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361 PubMedGoogle ScholarCrossref
26.
Puopolo  KM, Draper  D, Wi  S,  et al.  Estimating the probability of neonatal early-onset infection on the basis of maternal risk factors.   Pediatrics. 2011;128(5):e1155-e1163. doi:10.1542/peds.2010-3464 PubMedGoogle ScholarCrossref
27.
Marlow  N.  Keeping up with outcomes for infants born at extremely low gestational ages.   JAMA Pediatr. 2015;169(3):207-208. doi:10.1001/jamapediatrics.2014.3362 PubMedGoogle ScholarCrossref
28.
Bouwmeester  W, Twisk  JW, Kappen  TH, van Klei  WA, Moons  KG, Vergouwe  Y.  Prediction models for clustered data: comparison of a random intercept and standard regression model.   BMC Med Res Methodol. 2013;13:19. doi:10.1186/1471-2288-13-19 PubMedGoogle ScholarCrossref
29.
Wynants  L, Vergouwe  Y, Van Huffel  S, Timmerman  D, Van Calster  B.  Does ignoring clustering in multicenter data influence the performance of prediction models? a simulation study.   Stat Methods Med Res. 2018;27(6):1723-1736. doi:10.1177/0962280216668555 PubMedGoogle ScholarCrossref
30.
Wynants  L, Kent  DM, Timmerman  D, Lundquist  CM, Van Calster  B.  Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting.   Diagn Progn Res. 2019;3:6. doi:10.1186/s41512-019-0046-9 PubMedGoogle ScholarCrossref
31.
Rogowski  JA, Horbar  JD, Staiger  DO, Kenny  M, Carpenter  J, Geppert  J.  Indirect vs direct hospital quality indicators for very low-birth-weight infants.   JAMA. 2004;291(2):202-209. doi:10.1001/jama.291.2.202 PubMedGoogle ScholarCrossref
32.
Boland  RA, Davis  PG, Dawson  JA, Doyle  LW; Victorian Infant Collaborative Study Group.  Predicting death or major neurodevelopmental disability in extremely preterm infants born in Australia.   Arch Dis Child Fetal Neonatal Ed. 2013;98(3):F201-F204. doi:10.1136/archdischild-2012-301628 PubMedGoogle ScholarCrossref
33.
Marrs  CC, Pedroza  C, Mendez-Figueroa  H, Chauhan  SP, Tyson  JE.  Infant outcomes after periviable birth: external validation of the neonatal research network estimator with the BEAM trial.   Am J Perinatol. 2016;33(6):569-576.PubMedGoogle Scholar
34.
Cross  G, Becker  M, Congdon  P.  Prognosis for babies born with fused eyelids.   Arch Dis Child. 1985;60(5):479-480. doi:10.1136/adc.60.5.479 PubMedGoogle ScholarCrossref
35.
Stefano  JL, Morales  M.  Fused eyelids in the extremely premature infant: multivariate analysis of survival and outcome.   Am J Perinatol. 1992;9(2):84-86. doi:10.1055/s-2007-994677 PubMedGoogle ScholarCrossref
36.
Ngo  TTM, Moufarrej  MN, Rasmussen  MH,  et al.  Noninvasive blood tests for fetal development predict gestational age and preterm delivery.   Science. 2018;360(6393):1133-1136. doi:10.1126/science.aar3819 PubMedGoogle ScholarCrossref
37.
Stoll  BJ, Hansen  NI, Bell  EF,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Trends in care practices, morbidity, and mortality of extremely preterm neonates, 1993-2012.   JAMA. 2015;314(10):1039-1051. doi:10.1001/jama.2015.10244 PubMedGoogle ScholarCrossref
38.
Skupski  DW, McCullough  LB, Levene  M, Chervenak  FA.  Improving obstetric estimation of outcomes of extremely premature neonates: an evolving challenge.   J Perinat Med. 2010;38(1):19-22. doi:10.1515/jpm.2010.013 PubMedGoogle ScholarCrossref
39.
Atwell  K, Callander  E, Lindsay  D, Marshall  PB, Morris  SA.  Selection bias and outcomes for preterm neonates.   Pediatrics. 2018;142(1):e20180470. doi:10.1542/peds.2018-0470 PubMedGoogle Scholar
40.
Lantos  JD.  Ethical problems in decision making in the neonatal ICU.   N Engl J Med. 2018;379(19):1851-1860. doi:10.1056/NEJMra1801063 PubMedGoogle ScholarCrossref
41.
Ambalavanan  N, Carlo  WA, Tyson  JE,  et al; Generic Database; Subcommittees of the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Outcome trajectories in extremely preterm infants.   Pediatrics. 2012;130(1):e115-e125. doi:10.1542/peds.2011-3693 PubMedGoogle ScholarCrossref
42.
Vermont Oxford Network. Home page. https://nightingale.vtoxford.org. Accessed January 28, 2020.
43.
Eunice Kennedy Shriver National Institute of Child Health and Human Development. Home page. https://www.nichd.nih.gov/research/supported/EPBO. Accessed January 28, 2020.
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    March 2, 2020

    Assessment of an Updated Neonatal Research Network Extremely Preterm Birth Outcome Model in the Vermont Oxford Network

    Author Affiliations
    • 1Stead Family Department of Pediatrics, University of Iowa, Iowa City
    • 2Vermont Oxford Network, Burlington
    • 3Department of Pediatrics, University of Vermont College of Medicine, Burlington
    • 4Biostatistics and Epidemiology Division, RTI International, Research Triangle Park, North Carolina
    • 5Center for Clinical Research & Evidence-Based Medicine, University of Texas McGovern Medical School, Houston
    • 6Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia
    • 7Department of Pediatrics, University of Alabama, Birmingham
    • 8Department of Pediatrics, Duke University School of Medicine, Durham, North Carolina
    • 9Department of Mathematics and Statistics, College of Engineering and Mathematical Sciences, University of Vermont, Burlington
    • 10Department of Pediatrics, Stanford University School of Medicine, Palo Alto, California
    • 11Department of Pediatrics, Case Western Reserve University, Cleveland, Ohio
    • 12Biostatistics and Epidemiology Division, RTI International, Rockville, Maryland
    • 13Office of Research, George Mason University College of Health and Human Services, Fairfax, Virginia
    JAMA Pediatr. 2020;174(5):e196294. doi:10.1001/jamapediatrics.2019.6294
    Key Points

    Question  Do differences in infant survival among hospitals and over time affect a prognostic model widely used in extremely preterm birth counseling?

    Findings  In this prognostic study of most actively treated extremely preterm infants in the United States in 2006 to 2012 and 2013 to 2016, survival increased from 66% to 70%, and model prediction was moderate. The birth hospital and gestational age contributed equally to prediction of survival .

    Meaning  For extremely preterm birth, an area of medicine with substantial variation among hospitals and changing outcomes, prognostic models used in clinical practice may require accounting for local outcomes and periodic updating to remain relevant.

    Abstract

    Importance  The Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network (NRN) extremely preterm birth outcome model is widely used for prognostication by practitioners caring for families expecting extremely preterm birth. The model provides information on mean outcomes from 1998 to 2003 and does not account for substantial variation in outcomes among US hospitals.

    Objective  To update and validate the NRN extremely preterm birth outcome model for most extremely preterm infants in the United States.

    Design, Setting, and Participants  This prognostic study included 3 observational cohorts from January 1, 2006, to December 31, 2016, at 19 US centers in the NRN (derivation cohort) and 637 US centers in Vermont Oxford Network (VON) (validation cohorts). Actively treated infants born at 22 weeks’ 0 days’ to 25 weeks’ 6 days’ gestation and weighing 401 to 1000 g, including 4176 in the NRN for 2006 to 2012, 45 179 in VON for 2006 to 2012, and 25 969 in VON for 2013 to 2016, were studied. VON cohorts comprised more than 85% of eligible US births. Data analysis was performed from May 1, 2017, to March 31, 2019.

    Exposures  Predictive variables used in the original model, including infant sex, birth weight, plurality, gestational age at birth, and exposure to antenatal corticosteroids.

    Main Outcomes and Measures  The main outcome was death before discharge. Secondary outcomes included neurodevelopmental impairment at 18 to 26 months’ corrected age and measures of hospital resource use (days of hospitalization and ventilator use).

    Results  Among 4176 actively treated infants in the NRN cohort (48% female; mean [SD] gestational age, 24.2 [0.8] weeks), survival was 63% vs 62% among 3702 infants in the era of the original model (47% female; mean [SD] gestational age, 24.2 [0.8] weeks). In the concurrent (2006-2012) VON cohort, survival was 66% among 45 179 actively treated infants (47% female; mean [SD] gestational age, 24.1 [0.8] weeks) and 70% among 25 969 infants from 2013 to 2016 (48% female; mean [SD] gestational age, 24.1 [0.8] weeks). Model C statistics were 0.74 in the 2006-2012 validation cohort and 0.73 in the 2013-2016 validation cohort. With the use of decision curve analysis to compare the model with a gestational age–only approach to prognostication, the updated model showed a predictive advantage. The birth hospital contributed equally as much to prediction of survival as gestational age (20%) but less than the other factors combined (60%).

    Conclusions and Relevance  An updated model using well-known factors to predict survival for extremely preterm infants performed moderately well when applied to large US cohorts. Because survival rates change over time, the model requires periodic updating. The hospital of birth contributed substantially to outcome prediction.

    Introduction

    Accurate estimates of prognosis after extremely preterm birth allow practitioners and families to plan for potential outcomes and may guide clinical decision-making.1,2 Since 2008, the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Neonatal Research Network (NRN) has published on its website data describing outcomes of infants born to assist families and practitioners managing extremely preterm birth.3 The website displays outcomes from a model developed using data for infants born at NRN hospitals from 1998 to 2003. The model takes into account 5 factors: an infant’s sex, birth weight, plurality (singleton vs multiple gestation), gestational age at birth, and exposure to antenatal corticosteroids. Prognostic estimates that accounted for these 5 factors were more accurate than those based on gestational age alone.4

    The NICHD NRN extremely preterm birth outcome model is widely used to inform perinatal prognostic discussions5 and is cited in guidelines of professional societies.6,7 However, interventions and outcomes for extremely preterm infants have changed since its development.8,9 In addition, studies10-13 have described substantial variation in extremely preterm infant outcomes among US hospitals, including within the NRN.

    The aim of this investigation was to update the NICHD NRN extremely preterm birth outcome model to reflect more recent neonatal practices and outcomes. We evaluated the importance of hospital differences in outcomes relative to the other factors included in the model. In addition, we evaluated the updated model using data for most US extremely preterm births from 2006 to 2012 and 2013 to 2016 to assess its performance over time.

    Methods

    In this prognostic study, we included data for inborn infants born at 22 weeks’ 0 days’ to 25 weeks’ 6 days’ gestation and weighing 401 to 1000 g. Infants with major anomalies, with birth weights greater than the 97th percentile for gestational age (raising the possibility that gestational age may have been underestimated14,15), or who were not born at participating hospitals were excluded. These same criteria were used in the development of the original model.4 The institutional review boards at each NRN site approved data collection protocols. Waiver of consent was granted by the institutional review boards at 17 of 19 centers for the collection of in-hospital data and 4 of 19 centers for collection of follow-up data. All data were deidentified. The University of Vermont Institutional Review Board considered this research using the Vermont Oxford Network (VON) deidentified data repository to be not human subjects research. Model derivation, validation, and reporting were conducted according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.16

    Derivation Cohort

    The new model was derived using data for infants born from April 1, 2006, to December 31, 2012, at 19 centers participating in the NRN. Because outcome data for infants who did not receive any potentially life-sustaining treatment cannot be used to estimate the probability of survival with intensive care,17 the model was derived for infants who received any active treatment after birth. Active treatment was defined as surfactant therapy, endotracheal intubation, ventilatory support (including continuous positive airway pressure, bag-valve-mask ventilation, or mechanical ventilation), epinephrine, or chest compressions.18 This change from the original model, which defined active treatment based solely on administration of mechanical ventilation, reflects changes in practice in the intervening period.4

    Validation Cohorts

    The updated model was applied to births at 637 US hospitals in the VON database from January 1, 2006, to December 31, 2012, using the derivation cohort inclusion and exclusion criteria. VON included more than 85% of all US infants born at 22 to 25 weeks’ gestation and weighing 400 to 1000 g during the study period.19 Inclusion was not limited by the level or volume of the birth hospital. Although NRN data were derived from hospitals with level III and IV neonatal intensive care units (NICUs), the VON data included infants born at 79 hospitals with level I NICUs (restrictions on assisted ventilatory support), 176 hospitals with level II NICUs (no ventilatory support restrictions and no neonatal surgery), and 382 level III or IV NICUs (no ventilatory support restrictions and perform neonatal surgery). Many infants were transferred to other hospitals during their clinical care. Hospitals participating in both the NRN and VON were excluded from the validation data set for the overlapping years. A second cohort from the VON database comprising infants born from January 1, 2013, to December 31, 2016, from the same hospitals and with the same inclusion and exclusion criteria was used to test the stability of hospital-specific observed and estimated outcomes over time.

    Predictive Variables

    Infant data were collected at each hospital using standardized protocols. Gestational age at birth was defined in completed weeks (eg, 23 weeks, representing births at 23 weeks 0 days through 23 weeks 6 days) and determined using standard protocols that prioritized the best obstetric estimate.20 Birth weight was measured in grams and compared with the gestational age– and sex-specific growth curves used in the development of the original model.4,15 Plurality was defined as singleton birth or multiple birth. Exposure to antenatal corticosteroids was defined as a binary variable with exposure including any antenatal receipt of betamethasone or dexamethasone for acceleration of fetal maturation regardless of duration, dose, or timing relative to birth.

    Outcomes

    Survival was measured as survival to discharge or to 1 year of age if still hospitalized. Data on in-hospital resource use and neurodevelopmental outcomes at 18 to 26 months’ corrected age were available for the NRN derivation cohort and are described in eTable 1 and eTable 2 in the Supplement.

    Resource use was described in infant-days of mechanical ventilation per surviving infant and infant-days of hospitalization per surviving infant. Neurodevelopmental outcomes were ascertained by standardized, annually certified examiners who were not blinded to infants’ medical information. They were classified based on cognitive development assessed with the Bayley Scales of Infant and Toddler Development, Third Edition (BSID-III), gross motor function as assessed by the Gross Motor Function Classification System (GMFCS), and the presence of cerebral palsy and hearing and vision impairment. Profound neurodevelopmental impairment was defined as a BSID-III cognitive score that was untestable or a GMFCS level of 5 (on a scale of 0 [normal] to 5 [most impaired]). Moderate-severe neurodevelopmental impairment was defined as a BSID-III cognitive score of less than 85 (ie, >1 SD below the scale mean; mean [SD], 100 [15]), moderate or severe cerebral palsy, a GMFCS level of 2 to 5, bilateral blindness, or severe hearing impairment that could not be corrected with bilateral amplification.

    Statistical Analysis

    We compared descriptive characteristics of infants in the derivation and validation cohorts with the characteristics of infants in the cohort used to develop the original model (1998-2003).4 Analysis of the NRN data (L.L.) was conducted using SAS Enterprise Guide, version 7.15 (SAS Institute Inc) and Stata/MP, version 16.0 (StataCorp). Analysis of VON data (L.T.G.) was conducted using R, version 3.3.2 (R Foundation for Statistical Computing). Using the NRN data set, we derived a multivariable logistic mixed model with survival as the dependent outcome; birth weight, gestational age, plurality, and antenatal corticosteroid exposure as independent variables; and hospital of birth as a random intercept. These same factors were included in the original model.4 Neurodevelopmental outcomes and resource use were described for actively treated infants stratified by their probability of survival.

    For external validation, the updated model was applied to eligible infants in nonoverlapping hospitals in the 2006 to 2012 VON cohort. We estimated random intercepts for all VON hospitals using k-fold cross-validation to evaluate model fit. Modeling discrimination was assessed using the C statistic. For the outcome of survival, the C statistic represents the probability that for any randomly selected pair of individuals in which one lives and the other dies, the model estimates a higher probability for the one who lives.21,22 Model calibration was assessed by comparing estimated and observed outcomes for the whole sample across several subgroups of infant characteristics.

    To complement discrimination and calibration metrics, we also performed a decision curve analysis that compared the predicted outcomes of treating all or no patients by gestational age—the approach recommended in some guidelines23—with treating a targeted group based on model-estimated survival. The decision curve analysis presents various predicted survival rates above which infants might be selected to receive active treatment (ie, threshold probabilities) and presents the predictive net benefit of using that threshold. The predictive net benefit was calculated as follows: [proportion of treated infants who lived] – [(proportion of treated infants who died) × (threshold probability)/(1-threshold probability)].24,25

    To evaluate the importance of hospital differences in outcomes relative to the other factors included in the model, we compared observed and model-estimated outcomes with and without accounting for the hospital-specific intercept term in the estimate. We calculated the relative contribution of model variables to outcome estimation using the differences between the log-likelihood value of the full model and of models without each variable.26 To assess the stability of hospital-specific infant outcome estimates over time, the hospital-specific model with intercept terms derived in 2006 to 2012 was applied to the 2013 to 2016 cohort and observed and estimated outcomes were compared.

    As a sensitivity analysis, we also compared the outcomes estimated when the model was applied to all infants born in the validation cohorts regardless of whether they were actively treated, as previously performed.4 The results of this analysis estimated the maximum potential survival that might result if all infants in the cohorts were actively treated.4 Data analysis was performed from May 1, 2017, to March 31, 2019.

    Results

    Among 4176 actively treated infants in the NRN cohort (48% female; mean [SD] gestational age, 24.2 [0.8] weeks), survival was 63% vs 62% among 3702 infants in the era of the original model (47% female; mean [SD] gestational age, 24.2 [0.8] weeks). In the concurrent (2006-2012) VON cohort, survival was 66% among 45 179 actively treated infants (47% female; mean [SD] gestational age, 24.1 [0.8] weeks) and 70% among 25 969 infants from 2013 to 2016 (48% female; mean [SD] gestational age, 24.1 [0.8] weeks). Characteristics of the infants in each cohort are summarized in Table 1.

    Model Derivation

    Odds ratios for each variable included in the updated and original 5-factor models are shown in Figure 1. Except for the increase in the point estimate for survival for infants born at 22 weeks’ compared with 25 weeks’ gestation, there was little change in other model coefficients.

    Model-estimated survival outcomes for infants in the 2006 to 2012 NRN and VON validation cohorts by gestational age at birth are shown in eFigure 1 in the Supplement. The mean (SD) estimated probability of survival for actively treated infants born at 22 weeks’ gestation in the NRN was 25% (11%). However, there was considerable variation in the probability of survival for infants born during each gestational age week, with a range of 8% to 61% based on the other factors in the model. Likewise, the mean probability of survival for actively treated infants born at 23 weeks’ gestation in the NRN was 39% (14%), with a range of 9% to 76%.

    When accounting for the resources used for infants who died, for each infant who lived with an estimated survival probability of 11% to 20%, 153 infant-days of mechanical ventilation and 280 infant-days of hospitalization were used (eTable 1 in the Supplement). For each infant with a probability of survival greater than 80%, 28 infant-days of mechanical ventilation and 115 infant-days of hospitalization were used per infant who lived.

    Of 4176 actively treated infants in the NRN cohort, 3927 (94%) had follow-up outcomes known at 18 to 26 months’ corrected age (eTable 2 in the Supplement). Rates of profound neurodevelopmental impairment were 1% among infants with an estimated survival probability greater than 80% and 11% among infants with an estimated survival probability of 11% to 20%. Rates of moderate-severe neurodevelopmental impairment (including profound impairment) were 21% among infants with an estimated survival probability greater than 80% and 78% among infants with an estimated survival probability of 11% to 20%. No actively treated infants with a probability of survival of 10% or less (n = 6) lived to follow-up.

    Model Validation

    Estimates of model discrimination and calibration when the model was applied to the 2006 to 2012 VON cohort are given in Table 2. The original 5-factor model for mortality had a C statistic of 0.75 in the previously published derivation cohort of NRN hospitals (1998-2003)4; discrimination was similar in the 2006 to 2012 VON cohort (C statistic, 0.73). By comparison, the updated model incorporating hospital-specific intercepts had a C statistic of 0.74 in the 2006 to 2012 VON cohort.

    Compared with the original model and the use of gestational age mean outcomes, the updated model that accounted for hospital of birth was better calibrated to recent outcomes (Figure 2). When model-estimated and observed outcomes were compared, the calibration slope was 0.99 and the intercept was 0.03 (eFigure 2 in the Supplement). Decision curve analysis demonstrated that the updated model provided greater predictive net benefit vs the original model for a wide range of decision thresholds (eFigure 3 in the Supplement). The model that included hospital of birth outperformed both the original model and the updated model that did not account for hospital of birth. This model found a greater predictive net benefit at all gestational ages across a range of threshold probabilities. The predictive net benefit of the updated model was most pronounced at 22 and 23 weeks’ gestation compared with a gestational age–based approach.

    Hospital of Birth

    As indicated in Table 3, hospital of birth contributed to model estimation of survival equally as much as gestational age (20%). Factors besides hospital of birth and gestational age contributed 60% of model predictive ability. When the model without hospital of birth was applied to individual hospitals in the 2006 to 2012 VON cohort with more than 20 infants during the study period, 125 of 386 (32%) hospitals had an observed hospital survival rate significantly different from the estimated rate (eFigure 4 in the Supplement), suggesting that this model did not adequately account for variation in outcomes across hospitals.

    To test whether hospital-specific terms established for the 2006 to 2012 VON cohort could be used prospectively for the same hospitals, the NRN model applied to VON births from 2006 to 2012 was also applied to VON births from 2013 to 2016. For the same 386 VON hospitals, 34 (9%) had observed outcomes significantly different from predicted outcomes (eFigure 5 in the Supplement), with 30 of 34 hospitals having better than predicted outcomes, suggesting an improvement in outcomes over time not accounted for by the model. The C statistic of the model in the 2013 to 2016 cohort was 0.73.

    Sensitivity Analyses

    The predicted rate of survival among all infants born in study hospitals, assuming all infants were actively treated (ie, the maximum potential rate), approximated the observed and predicted survival rates among infants who were actively treated (eTable 3 in the Supplement). Of note, rates of both active treatment and infant survival increased from the 2006 to 2012 period to the 2013 to 2016 period. Predicted survival rates also increased during the period in association with increased use of antenatal corticosteroids (Table 1).

    Discussion

    The update to the NICHD NRN extremely preterm birth outcome model demonstrated moderate discrimination and calibration when applied to most actively treated extremely preterm infants in the United States. Compared with gestational age–only approaches, the model had a predictive advantage by decision curve analysis. The incorporation of hospital of birth into model estimates contributed substantially to outcome estimation. Although the association of the 5 infant characteristics in the model with infant outcomes remained relatively stable from the original era to the most recent one, outcomes improved during the 2 periods studied, suggesting a need to periodically recalibrate the model, including for hospital-specific outcomes.

    Because of variation in hospital outcomes, some practitioners have advocated for using only local statistics for predicting infant survival; however, given the small number of extremely preterm infants born at any given hospital, this approach may result in imprecise estimates and may preclude taking into account important prognostic variables, such as infant sex or antenatal corticosteroid exposure.7,27 In our study, approximately 60% of model estimation of survival resulted from variables besides hospital of birth and gestational age.

    Our approach accounted for hospital-specific outcomes while also benefiting from large data sets for deriving and validating the prognostic value of several important infant variables. The contribution of the birth hospital to model prediction may represent differences in practices or case mix among hospitals not accounted for by other model variables. Because prognostic estimates and treatment decisions are made within individual hospitals, our model has been calibrated to individual hospitals as recommended by several methodologists.28-30 Our approach assumes that past hospital-specific neonatal survival rates reliably predict future survival rates, as previously shown in a cohort of US hospitals.31 Our data suggest that outcomes of extremely preterm infants at many US hospitals vary substantially from the mean.

    The updated model assigned more accurate probabilities of infant outcomes than the original model when applied to births in recent large US cohorts (Figure 2 and eFigure 2 in the Supplement). However, the improved model calibration should be distinguished from model discrimination as represented by the C statistic, a measure of the ability of the model to separate individuals who develop the outcome of interest from those who do not. The C statistic of the updated model was not substantially different from that observed when deriving the original model4 and in independent validations of that model32,33 and demonstrated the continued difficulty of estimating outcomes for extremely preterm infants at the time of birth. Historically, signs from physical examination, such as fusion of the eyelids, were used to determine viability at extremely preterm gestations, although their utility has not been borne out.34,35 The use of the combined 5 factors in the model was again found to outperform gestational age alone as a tool for predicting survival,4 and the predictive benefits of a model-based vs gestational age–based approach were demonstrated in the decision curve analysis. In the future, other methods, such as the use of biochemical markers of infant maturity, may enhance our ability to estimate infant outcomes.36

    Although we attempted to maintain the same form as the original 5-factor model, which has been in use for more than a decade and is familiar to many practitioners, the updated model differs from the original in key ways. The study deriving the original model used mechanical ventilation that required endotracheal intubation as a marker of active treatment for infants born in 1999 to 2003. We defined active treatment using a broader definition that accounted for changes in the use of noninvasive ventilation37; 2.5% of 2394 infants in the derivation cohort did not require endotracheal intubation and mechanical ventilation. We also defined survival as occurring at discharge rather than at follow-up, the time point used in the original model, to validate the model in the VON cohorts, which included data through hospital discharge. These changes may explain the small difference in survival seen between the original 1998 to 2003 cohort and the 2006 to 2012 NRN cohort included in this study. In addition, information on timing of antenatal corticosteroid administration was not available for the cohorts used to update the model; the original model included antenatal corticosteroid exposure only if taking place within 7 days of birth.

    Limitations

    The updated model has important limitations, many of which are shared by the original model. First, the model includes birth weight, which can be estimated only imprecisely before birth.38 Second, the best obstetric estimates of gestational age for most pregnancies (except those conceived through in vitro fertilization) have a margin of error of at least 5 days.20 Third, our data do not provide information on reasons for withholding or withdrawing care, and actively treated infants may be different in unmeasured ways from their counterparts despite having similar probabilities of survival based on our model.39 Decisions about whether to provide life support at the margins of viability involve more than statistical calculations and must take into account clinical judgment and the values, beliefs, and concerns of each family.40 Fourth, prognosis is dynamic and can change with new events after birth41; the outcomes here reflect prognostic estimates for infants at the time of birth. Fifth, we did not have data on the causes of hospital variation in outcomes. Further research is needed to identify explanations for the observed variation. Sixth, although we evaluated the performance of the updated model among infants born several years after those whose data were used for model development, we were unable to provide estimates regarding how often the model should be periodically recalibrated to account for changing outcomes over time.

    Our final model explicitly incorporated the hospital of birth in outcome estimates, which was possible because VON included data for most eligible extremely preterm infants born in the United States during the period. Hospital-specific survival estimates for each hospital participating (eAppendix in Supplement) in VON are available to that hospital through the VON website.42 The NICHD has published an updated model providing the mean and range of hospital outcomes on their website available to the public.43

    Conclusions

    The updated NICHD NRN extremely preterm birth outcome model continues to demonstrate the importance of considering more than gestational age when evaluating prognosis at the time of birth. The hospital of birth contributed substantially to outcome prediction. As outcomes and practices continue to change, prognostic models used for extremely preterm birth may require accounting for local outcomes and periodic updating to remain relevant.

    Back to top
    Article Information

    Accepted for Publication: December 12, 2019

    Corresponding Author: Matthew A. Rysavy, MD, PhD, Stead Family Department of Pediatrics, University of Iowa, 200 Hawkins Dr, Iowa City, IA 52242 (matthew-rysavy@uiowa.edu).

    Published Online: March 2, 2020. doi:10.1001/jamapediatrics.2019.6294

    Author Contributions: Dr Li and Ms Greenberg had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Rysavy, Horbar, Bell, Tyson, Carlo, Younge, Walsh, Higgins.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Rysavy, Li.

    Critical revision of the manuscript for important intellectual content: Rysavy, Horbar, Bell, Greenberg, Tyson, Patel, Carlo, Younge, Green, Edwards, Hintz, Walsh, Buzas, Das, Higgins.

    Statistical analysis: Li, Greenberg, Green, Edwards, Walsh, Buzas, Das.

    Obtained funding: Bell, Carlo, Walsh.

    Administrative, technical, or material support: Bell, Patel, Walsh, Higgins.

    Supervision: Horbar, Bell, Tyson, Carlo, Hintz, Walsh, Higgins.

    Conflict of Interest Disclosures: Dr Rysavy reported receiving grants from the National Institutes of Health (NIH) and travel funds from the Vermont Oxford Network (VON) during the conduct of the study. Dr Horbar reported receiving personal fees from VON outside the submitted work and serving as an unpaid member of the Board of Directors of VON. Dr Bell reported receiving grants from the NIH. Dr Tyson reported receiving grants from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) during the conduct of the study and grants from the NICHD outside the submitted work. Dr Patel reported receiving grants from the NIH during the conduct of the study. Dr Carlo served as a paid member of the board of directors of Mednax and received grants from the NIH during the study. Dr Younge reported receiving grants from the NIH during the conduct of the study. Dr Green reported receiving grants from the NICHD during the conduct of the study. Dr Edwards reported receiving grants from VON during the conduct of the study. Dr Hintz reported receiving grants from the NICHD during the conduct of the study. Dr Walsh reported receiving grants from the NIH. Dr Das reported receiving grants from the NICHD and the NIH during the conduct of the study. No other disclosures were reported.

    Funding/Support: This study was supported by grants F32 HD098782 (Dr Rysavy), UG1 HD027904, UG1 HD021364 (Dr Walsh), UG1 HD027853, UG1 HD040492, UG1 HD027851, UG1 HD027856, UG1 HD021373, UG1 HD068278, U24 HD095254 (Dr Das), UG1 HD027880, UG1 HD053119, UG1 HD034216 (Dr Carlo), UG1 HD053109 (Dr Bell), UG1 HD053089, UG1 HD068244, UG1 HD068263, UG1 HD040689, UG1 HD053124, UG1 HD021385, and UG1 HD027871 from the NIH and the NICHD, which supported for the Neonatal Research Network’s (NRN’s) Generic Database and Follow-up Study and VON through cooperative agreements.

    Role of the Funder/Sponsor: The NICHD staff had input into the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.

    Group Members: The following investigators with the NICHD NRN, in addition to those listed as authors, participated in this study and are listed by center. NRN Steering Committee Chair: Michael S. Caplan, MD: University of Chicago, Pritzker School of Medicine (2006-2011); Richard A. Polin, MD: Division of Neonatology, College of Physicians and Surgeons, Columbia University (2011-present). Alpert Medical School of Brown University and Women & Infants Hospital of Rhode Island: Abbot R. Laptook, MD; Angelita M. Hensman, PhD, RNC-NIC; William Oh, MD; Kristin M. Basso, MaT, RN; Elisa Vieira, BSN, RN; Martin Keszler, MD; Betty R. Vohr, MD; Elisabeth C. McGowan, MD; Robert Burke, MD; Melinda Caskey, MD; Katharine Johnson, MD; Barbara Alksninis, PNP; Mary Lenore Keszler, MD; Theresa M. Leach, Med, CAES; Bonnie E. Stephens, MD; Victoria E. Watson, MS, CAS; Suzy Ventura; Andrea M. Knoll. Case Western Reserve University, Rainbow Babies & Children's Hospital: Anna Marie Hibbs, MD; Deanne E. Wilson-Costello, MD; Nancy S. Newman, BA, RN; Bonnie S. Siner, RN; Monika Bhola, MD; Gulgun Yalcinkaya, MD; Harriet G. Friedman, MA. Cincinnati Children's Hospital Medical Center, University Hospital, and Good Samaritan Hospital: Kurt Schibler, MD; Kate Bridges, MD; Barbara Alexander, RN; Cathy Grisby, BSN, CCRC; Jody Hessling, RN; Estelle E. Fischer, MHSA MBA; Lenora D. Jackson, CRC; Kristin Kirker, CRC; Holly L. Mincey, RN, BSN; Greg Muthig, BS; Teresa L. Gratton, PA; Jean J. Steichen, MD; Kimberly Yolton, PhD. Duke University School of Medicine, University Hospital, University of North Carolina, and Duke Regional Hospital: C. Michael Cotten, MD, MHS; Ronald N. Goldberg, MD; Ricki F. Goldstein, MD; Kimberley A. Fisher, PhD, FNP-BC, IBCLC; Kathy J. Auten, MSHS; Katherine A. Foy, RN; Sandra Grimes, RN, BSN; Joanne Finkle, RN, JD; Kathryn E. Gustafson, PhD; Melody B. Lohmeyer, RN, MSN; Matthew M. Laughon, MD MPH; Carl L. Bose, MD; Janice Bernhardt, MS RN; Gennie Bose, RN; Cindy Clark, RN. Emory University, Children’s Healthcare of Atlanta, Grady Memorial Hospital: David P. Carlton, MD; Barbara J. Stoll, MD; Ellen C. Hale, RN, BS, CCRC; Yvonne C. Loggins, RN, BSN; Ira Adams-Chapman, MD; Maureen Mulligan LaRossa, RN; Sheena L. Carter, PhD. NICHD: Stephanie Wilson Archer, MA. Indiana University, University Hospital, Methodist Hospital, Riley Hospital for Children, and Eskenazi Health: Gregory M. Sokol, MD; Brenda B. Poindexter, MD, MS; Anna Maria Dusick, MD (deceased); Dianne E. Herron, RN, CCRC; Abbey C. Hines, PsyD; Carolyn Lytle, MD, MPH; Lucy C. Miller, RN, BSN, CCRC; Heike M. Minnich, PsyD, HSPP; Lu Ann Papile, MD; Leslie Dawn Wilson, BSN, CCRC. McGovern Medical School at The University of Texas Health Science Center at Houston, Children's Memorial Hermann Hospital, and Lyndon Baines Johnson General Hospital/Harris County Hospital District: Kathleen A. Kennedy, MD, MPH; Nora I. Alaniz, BS; Katrina Burson, RN, BSN; Patricia W. Evans, MD; Beverly Foley Harris, RN, BSN; Margarita Jiminez, MD, MPH; Anna E. Lis, RN, BSN; Sarah Martin, RN, BSN; Georgia E. McDavid, RN; Brenda H. Morris, MD; M. Layne Poundstone, RN, BSN; Peggy Robichaux, RN, BSN; Saba Siddiki, MD; Maegan C. Simmons, RN; Patti L. Pierce Tate, RCP; Sharon L. Wright, MT (ASCP). Nationwide Children’s Hospital and the Ohio State University Medical Center: Pablo J. Sánchez, MD; Leif D. Nelin, MD; Sudarshan R. Jadcherla, MD; Patricia Luzader, RN; Christine A. Fortney, PhD, RN; Gail E. Besner; Nehal A. Parikh, MD. RTI International: Marie G. Gantz, PhD; W. Kenneth Poole, PhD (deceased); Dennis Wallace, PhD; Jamie E. Newman, PhD, MPH; Jeanette O’Donnell Auman, BS; Margaret M. Crawford, BS, CCRP; Carolyn M. Petrie Huitema, MS, CCRP; Kristin M. Zaterka-Baxter, RN, BSN, CCRP. Stanford University, Dominican Hospital, El Camino Hospital, and Lucile Packard Children's Hospital: Krisa P. Van Meurs, MD; David K. Stevenson, MD; Marian M. Adams, MD; M. Bethany Ball, BSc, CCRC; Andrew W. Palmquist, RN, BSN; Melinda S. Proud, RCP; Barbara Bentley, PsychD, MSEd; Elizabeth Bruno, PhD; Maria Elena DeAnda, PhD; Anne M. DeBattista, RN, PNP; Beth Earhart, PhD; Jean G. Kohn, MD, MPH; Casey E. Krueger, PhD; Renee P. Pyle, PhD; Heather Taylor, PhD; Hali E. Weiss, MD. Tufts Medical Center, Floating Hospital for Children: Ivan D. Frantz III, MD; John M. Fiascone, MD; Brenda L. MacKinnon, RNC; Ellen Nylen, RN, BSN; Anne Furey, MPH; Elisabeth C. McGowan, MD; Cecelia E. Sibley, PT, MHA; Ana K. Brussa, MS, OTR/L. University of Alabama at Birmingham Health System and Children’s Hospital of Alabama: Namasivayam Ambalavanan, MD; Myriam Peralta-Carcelen, MD, MPH; Monica V. Collins, RN, BSN, MaEd; Shirley S. Cosby, RN, BSN; Fred J. Biasini, PhD; Kathleen G. Nelson, MD; Cryshelle S. Patterson, PhD; Vivien A. Phillips, RN, BSN; Sally Whitley, MA, OTR-L, FAOTA; Carin Kiser, MD; Leigh Ann Smith, CRNP; Richard V. Rector, PhD; Sarah Ryan, PhD; Kristy Domnanovich, PhD; Leslie Rodrigues, PhD; Sheree Chapman York, PT, DPT, MS, PCS. University of Iowa and Mercy Medical Center: Jane E. Brumbaugh, MD; Tarah T. Colaizy, MD, MPH; Dan L. Ellsbury, MD; John A. Widness, MD; Michael J. Acarregui, MD, MBA; Karen J. Johnson, RN, BSN; Donia B. Campbell, RNC-NIC; Diane L. Eastman, RN, CPNP, MA. University of New Mexico Health Sciences Center: Kristi L. Watterberg, MD; Robin K. Ohls, MD; Janell F. Fuller, MD; Conra Backstrom Lacy, RN; Rebecca A. Montman, BSN, RNC; Jean R. Lowe, PhD; Andrea Freeman Duncan, MD; Sandra Brown, BSN; Carol Hartenberger, BSN, MPH. University of Pennsylvania, Hospital of the University of Pennsylvania, Pennsylvania Hospital, and Children's Hospital of Philadelphia: Barbara Schmidt, MD, MSc; Haresh Kirpalani, MB, MSc; Sara B. DeMauro, MD, MSCE; Aasma S. Chaudhary, BS, RRT; Soraya Abbasi, MD; Toni Mancini, RN, BSN, CCRC; Dara Cucinotta. University of Rochester Medical Center, Golisano Children's Hospital, and the University at Buffalo Women's and Children's Hospital of Buffalo: Dale L. Phelps, MD; Gary J. Myers, MD; Linda J. Reubens, RN, CCRC; Erica Burnell, RN; Diane Hust, MS, RN, CS; Julie Babish Johnson, MSW; Rosemary L. Jensen; Emily Kushner, MA; Joan Merzbach, LMSW; Kelley Yost, PhD; Lauren Zwetsch, RN, MS, PNP; Satyan Lakshminrusimha, MD; Anne Marie Reynolds, MD, MPH; Osman Farooq, MD; Ashley Williams, MS, Ed; William A. Zorn, PhD. University of Texas Southwestern Medical Center at Dallas, Parkland Health & Hospital System, and Children's Medical Center Dallas: Pablo J. Sánchez, MD; Roy J. Heyne, MD; Walid A. Salhab, MD; Charles R. Rosenfeld, MD; Luc P. Brion, MD; LiJun Chen, RN, PhD; Alicia Guzman; Melissa H. Leps, RN; Nancy A. Miller, RN; Diana M. Vasil, MSN, BSN, RNC, NIC; Lizette E. Torres, RN; Gaynelle Hensley, RN; Sally S. Adams, MS, RN, CPNP; Linda A. Madden, RN CPNP; Elizabeth Heyne, PsyD PA-C; Janet S. Morgan, RN; Catherine Twell Boatman, MS, CIMI. University of Utah University Hospital, Intermountain Medical Center, LDS Hospital, and Primary Children's Medical: Roger G. Faix, MD; Bradley A. Yoder, MD; Karen A. Osborne, RN BSN CCRC; Shawna Baker, RN; Karie Bird, RN, BSN; Jill Burnett, RNC, BSN; Jennifer J. Jensen, RN, BSN; Cynthia Spencer, RNC, BSN; Mike Steffen, PhD; Kimberlee Weaver-Lewis, RN, MS; Sarah Winter, MD; Karen Zanetti, RN. Wayne State University, University of Michigan, Hutzel Women’s Hospital, Sinai-Grace Hospital, and Children’s Hospital of Michigan: Seetha Shankaran, MD; Sanjay Chawla, MD, Girija Natarajan, MD, Monika Bajaj, MD, Beena Sood, MD, Athina Pappas, MD; John Barks, MD; Rebecca Bara, RN, BSN; Laura A. Goldston, MA; Mary Johnson, RN, BSN; Laura Sumner, RN, BSN; Kara Sawaya, RN, BSN; Kathleen Weingarden, RN, BSN; Mary Christensen, RT; Stephanie Wiggins, MS; Daine F. White, RN; Eunice Woldt, RN, MSN. Yale University, Yale-New Haven Children’s Hospital, and Bridgeport Hospital: Richard A. Ehrenkranz, MD; Harris Jacobs, MD; Christine G. Butler, MD; Patricia Cervone, RN; Sheila Greisman, RN; Monica Konstantino, RN, BSN; JoAnn Poulsen, RN; Janet Taft, RN, BSN; Joanne Williams, RN, BSN; Elaine Romano, MSN.

    Disclaimer: The comments and views of the authors do not necessarily represent the views of NICHD, the NIH, the Department of Health and Human Services, or the US government.

    References
    1.
    Rysavy  MA, Tyson  JE.  The problem and promise of prognosis research.   JAMA Pediatr. 2016;170(5):411-412. doi:10.1001/jamapediatrics.2015.4871 PubMedGoogle ScholarCrossref
    2.
    Rysavy  MA.  Prognosis as an intervention.   Clin Perinatol. 2018;45(2):231-240. doi:10.1016/j.clp.2018.01.009 PubMedGoogle ScholarCrossref
    3.
    NICHD Neonatal Research Network (NRN). Extremely preterm birth outcome data. https://www.nichd.nih.gov/about/org/der/branches/ppb/programs/epbo/Pages/epbo_case.aspx. Accessed July 1, 2019.
    4.
    Tyson  JE, Parikh  NA, Langer  J, Green  C, Higgins  RD; National Institute of Child Health and Human Development Neonatal Research Network.  Intensive care for extreme prematurity: moving beyond gestational age.   N Engl J Med. 2008;358(16):1672-1681. doi:10.1056/NEJMoa073059 PubMedGoogle ScholarCrossref
    5.
    Myers  P, Laventhal  N, Andrews  B, Lagatta  J, Meadow  W.  Population-based outcomes data for counseling at the margin of gestational viability.   J Pediatr. 2017;181:208-212.e4. doi:10.1016/j.jpeds.2016.10.021 PubMedGoogle ScholarCrossref
    6.
    American College of Obstetricians and Gynecologists, Society for Maternal-Fetal Medicine.  Obstetric Care Consensus No. 6: periviable birth.   Obstet Gynecol. 2017;130(4):e187-e199. . doi:10.1097/AOG.0000000000002352 PubMedGoogle ScholarCrossref
    7.
    Cummings  J; Committee on Fetus and Newborn.  Antenatal counseling regarding resuscitation and intensive care before 25 weeks of gestation.   Pediatrics. 2015;136(3):588-595. doi:10.1542/peds.2015-2336 PubMedGoogle ScholarCrossref
    8.
    Horbar  JD, Carpenter  JH, Badger  GJ,  et al.  Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009.   Pediatrics. 2012;129(6):1019-1026. doi:10.1542/peds.2011-3028 PubMedGoogle ScholarCrossref
    9.
    Patel  RM, Kandefer  S, Walsh  MC,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Causes and timing of death in extremely premature infants from 2000 through 2011.   N Engl J Med. 2015;372(4):331-340. doi:10.1056/NEJMoa1403489 PubMedGoogle ScholarCrossref
    10.
    Horbar  JD, Onstad  L.  Survival for inborn infants weighing 501-1500 grams at birth: Variation among centers.   Pediatr Res. 1990;27(4):245A.Google Scholar
    11.
    Stoll  BJ, Hansen  NI, Bell  EF,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Neonatal outcomes of extremely preterm infants from the NICHD Neonatal Research Network.   Pediatrics. 2010;126(3):443-456. doi:10.1542/peds.2009-2959 PubMedGoogle ScholarCrossref
    12.
    Alleman  BW, Bell  EF, Li  L,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Individual and center-level factors affecting mortality among extremely low birth weight infants.   Pediatrics. 2013;132(1):e175-e184. doi:10.1542/peds.2012-3707 PubMedGoogle ScholarCrossref
    13.
    Horbar  JD, Edwards  EM, Greenberg  LT,  et al.  Variation in performance of neonatal intensive care units in the United States.   JAMA Pediatr. 2017;171(3):e164396. doi:10.1001/jamapediatrics.2016.4396 PubMedGoogle Scholar
    14.
    Kramer  MS, McLean  FH, Boyd  ME, Usher  RH.  The validity of gestational age estimation by menstrual dating in term, preterm, and postterm gestations.   JAMA. 1988;260(22):3306-3308. doi:10.1001/jama.1988.03410220090034 PubMedGoogle ScholarCrossref
    15.
    Kramer  MS, Platt  RW, Wen  SW,  et al; Fetal/Infant Health Study Group of the Canadian Perinatal Surveillance System.  A new and improved population-based Canadian reference for birth weight for gestational age.   Pediatrics. 2001;108(2):E35. doi:10.1542/peds.108.2.e35 PubMedGoogle Scholar
    16.
    Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.   BMJ. 2015;350:g7594. doi:10.1136/bmj.g7594 PubMedGoogle ScholarCrossref
    17.
    Rysavy  MA, Marlow  N, Doyle  LW,  et al.  Reporting outcomes of extremely preterm births.   Pediatrics. 2016;138(3):e20160689. doi:10.1542/peds.2016-0689 PubMedGoogle Scholar
    18.
    Rysavy  MA, Li  L, Bell  EF,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Between-hospital variation in treatment and outcomes in extremely preterm infants.   N Engl J Med. 2015;372(19):1801-1811. doi:10.1056/NEJMoa1410689 PubMedGoogle ScholarCrossref
    19.
    United States Department of Health and Human Services (US DHHS). CDC WONDER online database: natality public-use data 2007-2017. https://wonder.cdc.gov/natality-current.html. Accessed Feb 1, 2019.
    20.
    Spong  CY.  Defining “term” pregnancy: recommendations from the Defining “Term” Pregnancy Workgroup.   JAMA. 2013;309(23):2445-2446. doi:10.1001/jama.2013.6235 PubMedGoogle ScholarCrossref
    21.
    Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.   Stat Med. 1996;15(4):361-387. doi:10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4 PubMedGoogle ScholarCrossref
    22.
    Riley  RD, Ensor  J, Snell  KIE,  et al.  External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges.   BMJ. 2016;353:i3140. doi:10.1136/bmj.i3140 PubMedGoogle ScholarCrossref
    23.
    Guillén  Ú, Weiss  EM, Munson  D,  et al.  Guidelines for the management of extremely premature deliveries: a systematic review.   Pediatrics. 2015;136(2):343-350. doi:10.1542/peds.2015-0542 PubMedGoogle ScholarCrossref
    24.
    Vickers  AJ, Van Calster  B, Steyerberg  EW.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.   BMJ. 2016;352:i6. doi:10.1136/bmj.i6 PubMedGoogle ScholarCrossref
    25.
    Vickers  AJ, Elkin  EB.  Decision curve analysis: a novel method for evaluating prediction models.   Med Decis Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361 PubMedGoogle ScholarCrossref
    26.
    Puopolo  KM, Draper  D, Wi  S,  et al.  Estimating the probability of neonatal early-onset infection on the basis of maternal risk factors.   Pediatrics. 2011;128(5):e1155-e1163. doi:10.1542/peds.2010-3464 PubMedGoogle ScholarCrossref
    27.
    Marlow  N.  Keeping up with outcomes for infants born at extremely low gestational ages.   JAMA Pediatr. 2015;169(3):207-208. doi:10.1001/jamapediatrics.2014.3362 PubMedGoogle ScholarCrossref
    28.
    Bouwmeester  W, Twisk  JW, Kappen  TH, van Klei  WA, Moons  KG, Vergouwe  Y.  Prediction models for clustered data: comparison of a random intercept and standard regression model.   BMC Med Res Methodol. 2013;13:19. doi:10.1186/1471-2288-13-19 PubMedGoogle ScholarCrossref
    29.
    Wynants  L, Vergouwe  Y, Van Huffel  S, Timmerman  D, Van Calster  B.  Does ignoring clustering in multicenter data influence the performance of prediction models? a simulation study.   Stat Methods Med Res. 2018;27(6):1723-1736. doi:10.1177/0962280216668555 PubMedGoogle ScholarCrossref
    30.
    Wynants  L, Kent  DM, Timmerman  D, Lundquist  CM, Van Calster  B.  Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting.   Diagn Progn Res. 2019;3:6. doi:10.1186/s41512-019-0046-9 PubMedGoogle ScholarCrossref
    31.
    Rogowski  JA, Horbar  JD, Staiger  DO, Kenny  M, Carpenter  J, Geppert  J.  Indirect vs direct hospital quality indicators for very low-birth-weight infants.   JAMA. 2004;291(2):202-209. doi:10.1001/jama.291.2.202 PubMedGoogle ScholarCrossref
    32.
    Boland  RA, Davis  PG, Dawson  JA, Doyle  LW; Victorian Infant Collaborative Study Group.  Predicting death or major neurodevelopmental disability in extremely preterm infants born in Australia.   Arch Dis Child Fetal Neonatal Ed. 2013;98(3):F201-F204. doi:10.1136/archdischild-2012-301628 PubMedGoogle ScholarCrossref
    33.
    Marrs  CC, Pedroza  C, Mendez-Figueroa  H, Chauhan  SP, Tyson  JE.  Infant outcomes after periviable birth: external validation of the neonatal research network estimator with the BEAM trial.   Am J Perinatol. 2016;33(6):569-576.PubMedGoogle Scholar
    34.
    Cross  G, Becker  M, Congdon  P.  Prognosis for babies born with fused eyelids.   Arch Dis Child. 1985;60(5):479-480. doi:10.1136/adc.60.5.479 PubMedGoogle ScholarCrossref
    35.
    Stefano  JL, Morales  M.  Fused eyelids in the extremely premature infant: multivariate analysis of survival and outcome.   Am J Perinatol. 1992;9(2):84-86. doi:10.1055/s-2007-994677 PubMedGoogle ScholarCrossref
    36.
    Ngo  TTM, Moufarrej  MN, Rasmussen  MH,  et al.  Noninvasive blood tests for fetal development predict gestational age and preterm delivery.   Science. 2018;360(6393):1133-1136. doi:10.1126/science.aar3819 PubMedGoogle ScholarCrossref
    37.
    Stoll  BJ, Hansen  NI, Bell  EF,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Trends in care practices, morbidity, and mortality of extremely preterm neonates, 1993-2012.   JAMA. 2015;314(10):1039-1051. doi:10.1001/jama.2015.10244 PubMedGoogle ScholarCrossref
    38.
    Skupski  DW, McCullough  LB, Levene  M, Chervenak  FA.  Improving obstetric estimation of outcomes of extremely premature neonates: an evolving challenge.   J Perinat Med. 2010;38(1):19-22. doi:10.1515/jpm.2010.013 PubMedGoogle ScholarCrossref
    39.
    Atwell  K, Callander  E, Lindsay  D, Marshall  PB, Morris  SA.  Selection bias and outcomes for preterm neonates.   Pediatrics. 2018;142(1):e20180470. doi:10.1542/peds.2018-0470 PubMedGoogle Scholar
    40.
    Lantos  JD.  Ethical problems in decision making in the neonatal ICU.   N Engl J Med. 2018;379(19):1851-1860. doi:10.1056/NEJMra1801063 PubMedGoogle ScholarCrossref
    41.
    Ambalavanan  N, Carlo  WA, Tyson  JE,  et al; Generic Database; Subcommittees of the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network.  Outcome trajectories in extremely preterm infants.   Pediatrics. 2012;130(1):e115-e125. doi:10.1542/peds.2011-3693 PubMedGoogle ScholarCrossref
    42.
    Vermont Oxford Network. Home page. https://nightingale.vtoxford.org. Accessed January 28, 2020.
    43.
    Eunice Kennedy Shriver National Institute of Child Health and Human Development. Home page. https://www.nichd.nih.gov/research/supported/EPBO. Accessed January 28, 2020.
    ×