[Skip to Content]
[Skip to Content Landing]
Figure 1.
Children’s Intracranial Injury Decision Aid Risk Score
Children’s Intracranial Injury Decision Aid Risk Score

The graph shows the risk for the composite outcome at different point levels.

Figure 2.
Risk Score vs Emergency Department Disposition
Risk Score vs Emergency Department Disposition

The proportion of patients who, at each level of the Children’s Intracranial Injury Decision Aid (CHIIDA) risk score, were sent to high- vs low-acuity emergency department dispositions based on clinical assessment alone. The open circles indicate a low-acuity disposition (ie, home, general inpatient, or short-stay/observation unit). The darker (solid) circles indicate a high-acuity disposition (intensive care unit or the operating room). The R2 statistic of the CHIIDA score vs disposition was 0.38, indicating a relatively weak correlation.

Table 1.  
Population Demographic Characteristics, Clinical Characteristics, Computed Tomographic Findings, and Emergency Department Disposition
Population Demographic Characteristics, Clinical Characteristics, Computed Tomographic Findings, and Emergency Department Disposition
Table 2.  
Final Multivariate Model Predicting the Need for Intensive Care Unit Admission Among Children With GCS 13-15 Head Injuries and Intracranial Injurya
Final Multivariate Model Predicting the Need for Intensive Care Unit Admission Among Children With GCS 13-15 Head Injuries and Intracranial Injurya
Table 3.  
Performance Characteristics of Different Risk Cutoffs in Predicting the Composite Outcome
Performance Characteristics of Different Risk Cutoffs in Predicting the Composite Outcome
1.
Mannix  R, O’Brien  MJ, Meehan  WP  III.  The epidemiology of outpatient visits for minor head injury: 2005 to 2009.  Neurosurgery. 2013;73(1):129-134.PubMedGoogle ScholarCrossref
2.
National Center for Injury Prevention and Control.  Report to Congress on Mild Traumatic Brain Injury in the United States: Steps to Prevent a Serious Public Health Problem. Atlanta, GA: National Center for Injury Prevention and Control; 2003.
3.
Centers for Disease Control and Prevention. Injury prevention & control: traumatic brain injury & concussion. http://www.cdc.gov/traumaticbraininjury/severe.html. Accessed April 5, 2016.
4.
Koepsell  TD, Rivara  FP, Vavilala  MS,  et al.  Incidence and descriptive epidemiologic features of traumatic brain injury in King County, Washington.  Pediatrics. 2011;128(5):946-954.PubMedGoogle ScholarCrossref
5.
Bowman  SM, Bird  TM, Aitken  ME, Tilford  JM.  Trends in hospitalizations associated with pediatric traumatic brain injuries.  Pediatrics. 2008;122(5):988-993.PubMedGoogle ScholarCrossref
6.
Mannix  R, Meehan  WP, Monuteaux  MC, Bachur  RG.  Computed tomography for minor head injury: variation and trends in major United States pediatric emergency departments.  J Pediatr. 2012;160(1):136-9.e1.PubMedGoogle ScholarCrossref
7.
Di  F, Gao  Q, Xiang  J,  et al.  Clinical practice experiences in diagnosis and treatment of traumatic brain injury in children: a survey among clinicians at 9 large hospitals in China.  PLoS One. 2015;10(11):e0142983.PubMedGoogle ScholarCrossref
8.
Kuppermann  N, Holmes  JF, Dayan  PS,  et al; Pediatric Emergency Care Applied Research Network (PECARN).  Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study.  Lancet. 2009;374(9696):1160-1170.PubMedGoogle ScholarCrossref
9.
Yeates  KO, Taylor  HG, Rusin  J,  et al.  Longitudinal trajectories of postconcussive symptoms in children with mild traumatic brain injuries and their relationship to acute clinical status.  Pediatrics. 2009;123(3):735-743.PubMedGoogle ScholarCrossref
10.
Rivara  FP, Koepsell  TD, Wang  J,  et al.  Disability 3, 12, and 24 months after traumatic brain injury among children and adolescents.  Pediatrics. 2011;128(5):e1129-e1138.PubMedGoogle ScholarCrossref
11.
Wang  HE, Yealy  DM.  Distribution of specialized care centers in the United States.  Ann Emerg Med. 2012;60(5):632-637.e7.PubMedGoogle ScholarCrossref
12.
Colville  G, Darkins  J, Hesketh  J, Bennett  V, Alcock  J, Noyes  J.  The impact on parents of a child’s admission to intensive care: integration of qualitative findings from a cross-sectional study.  Intensive Crit Care Nurs. 2009;25(2):72-79.PubMedGoogle ScholarCrossref
13.
Balluffi  A, Kassam-Adams  N, Kazak  A, Tucker  M, Dominguez  T, Helfaer  M.  Traumatic stress in parents of children admitted to the pediatric intensive care unit.  Pediatr Crit Care Med. 2004;5(6):547-553.PubMedGoogle ScholarCrossref
14.
Boran  BOB, Boran  P, Barut  N, Akgun  C, Celikoglu  E, Bozbuga  M.  Evaluation of mild head injury in a pediatric population.  Pediatr Neurosurg. 2006;42(4):203-207.PubMedGoogle ScholarCrossref
15.
Simon  B, Letourneau  P, Vitorino  E, McCall  J.  Pediatric minor head trauma: indications for computed tomographic scanning revisited.  J Trauma. 2001;51(2):231-237.PubMedGoogle ScholarCrossref
16.
Greenberg  JK, Stoev  IT, Park  TS,  et al.  Management of children with mild traumatic brain injury and intracranial hemorrhage.  J Trauma Acute Care Surg. 2014;76(4):1089-1095.PubMedGoogle ScholarCrossref
17.
Quayle  KS, Powell  EC, Mahajan  P,  et al.  Epidemiology of blunt head trauma in children in US emergency departments.  N Engl J Med. 2014;371(20):1945-1947.PubMedGoogle ScholarCrossref
18.
Burns  EC, Burns  B, Newgard  CD,  et al.  Pediatric minor traumatic brain injury with intracranial hemorrhage: identifying low-risk patients who may not benefit from ICU admission [published online October 28, 2016].  Pediatr Emerg Care. doi:10.1097/PEC.0000000000000950PubMedGoogle Scholar
19.
Pediatric Emergency Care Applied Research Network. Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study. http://www.pecarn.org/studyDatasets/StudyDetails?studyID=4. Revised September 28, 2015. Accessed October 22, 2016.
20.
Landis  JR, Koch  GG.  The measurement of observer agreement for categorical data.  Biometrics. 1977;33(1):159-174.PubMedGoogle ScholarCrossref
21.
van Buuren  S, Boshuizen  HC, Knook  DL.  Multiple imputation of missing blood pressure covariates in survival analysis.  Stat Med. 1999;18(6):681-694.PubMedGoogle ScholarCrossref
22.
Van Buuren  S, Groothuis-Oudshoorn  K.  Mice: multivariate imputation by chained equations in R.  J Stat Softw. 2011;45(3). doi:10.18637/jss.v045.i03Google Scholar
23.
R Core Team. R Foundation for Statistical Computing. R: a language and environment for statistical computing. http://www.R-project.org/. Accessed October 10, 2016.
24.
Rubin  DB, Schenker  N.  Multiple imputation in health-care databases: an overview and some applications.  Stat Med. 1991;10(4):585-598.PubMedGoogle ScholarCrossref
25.
Hanley  JA, McNeil  BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve.  Radiology. 1982;143(1):29-36.PubMedGoogle ScholarCrossref
26.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138.PubMedGoogle ScholarCrossref
27.
Altman  DG, Vergouwe  Y, Royston  P, Moons  KG.  Prognosis and prognostic research: validating a prognostic model.  BMJ. 2009;338:b605.PubMedGoogle ScholarCrossref
28.
Wells  BJ, Jain  A, Arrigain  S, Yu  C, Rosenkrans  WA  Jr, Kattan  MW.  Predicting 6-year mortality risk in patients with type 2 diabetes.  Diabetes Care. 2008;31(12):2301-2306.PubMedGoogle ScholarCrossref
29.
Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use: The Framingham Study risk score functions.  Stat Med. 2004;23(10):1631-1660.PubMedGoogle ScholarCrossref
30.
rms: Regression Modeling Strategies [computer program]. R package version 4.4-1. Vienna, Austria: R Foundation for Statistical Computing; 2015.
31.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model For Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD Statement.  BMC Med. 2015;13(1):1.PubMedGoogle ScholarCrossref
32.
Bell  MJ, Wisniewski  SR.  Severe traumatic brain injury in children: a vision for the future.  Intensive Care Med. 2016;42(10):1618-1620.PubMedGoogle ScholarCrossref
33.
University Hospitals Rainbow Babies & Children's Hospital. Patient pricing information. http://www.uhhospitals.org/rainbow/patients-and-visitors/billing-insurance-and-medical-records/patient-pricing-information. Accessed April 9, 2016.
34.
Nationwide Children's. Patient price information list. http://www.nationwidechildrens.org/price-information-list. Accessed April 9, 2016.
35.
Halpern  NA, Pastores  SM.  Critical care medicine in the United States 2000-2005: an analysis of bed numbers, occupancy rates, payer mix, and costs.  Crit Care Med. 2010;38(1):65-71.PubMedGoogle ScholarCrossref
36.
Shudy  M, de Almeida  ML, Ly  S,  et al.  Impact of pediatric critical illness and injury on families: a systematic literature review.  Pediatrics. 2006;118(suppl 3):S203-S218.PubMedGoogle ScholarCrossref
37.
Wang  MYG, Griffith  P, Sterling  J, McComb  JG, Levy  ML.  A prospective population-based study of pediatric trauma patients with mild alterations in consciousness (Glasgow Coma Scale score of 13-14).  Neurosurgery. 2000;46(5):1093-1099.PubMedGoogle ScholarCrossref
38.
Holsti  M, Kadish  HA, Sill  BL, Firth  SD, Nelson  DS.  Pediatric closed head injuries treated in an observation unit.  Pediatr Emerg Care. 2005;21(10):639-644.PubMedGoogle ScholarCrossref
39.
Atabaki  SM, Stiell  IG, Bazarian  JJ,  et al.  A clinical decision rule for cranial computed tomography in minor pediatric head trauma.  Arch Pediatr Adolesc Med. 2008;162(5):439-445.PubMedGoogle ScholarCrossref
40.
Kanzaria  HK, Brook  RH, Probst  MA, Harris  D, Berry  SH, Hoffman  JR.  Emergency physician perceptions of shared decision-making.  Acad Emerg Med. 2015;22(4):399-405.PubMedGoogle ScholarCrossref
41.
Hess  EP, Grudzen  CR, Thomson  R, Raja  AS, Carpenter  CR.  Shared decision-making in the emergency department: respecting patient autonomy when seconds count.  Acad Emerg Med. 2015;22(7):856-864.PubMedGoogle ScholarCrossref
42.
Bullock  MR, Chesnut  R, Ghajar  J,  et al; Surgical Management of Traumatic Brain Injury Author Group.  Surgical management of depressed cranial fractures.  Neurosurgery. 2006;58(3)(suppl):S56-S60, discussion Si-iv.PubMedGoogle Scholar
43.
Hunter  B, Carpenter  CR. The development of clinical prediction rules. In: Wilson  MP, Guluma  KZ, Hayden  S, eds.  Doing Research in Emergency and Acute Care: Making Order Out of Chaos. Chichester, West Sussex, England: Wiley-Blackwell; 2015:139-148.
Original Investigation
April 2017

Development and Internal Validation of a Clinical Risk Score for Treating Children With Mild Head Trauma and Intracranial Injury

Author Affiliations
  • 1Department of Neurosurgery, Washington University School of Medicine in St Louis, St Louis, Missouri
  • 2Department of Surgery, Washington University School of Medicine in St Louis, St Louis, Missouri
  • 3Division of Emergency Medicine, Department of Neurology, Washington University School of Medicine in St Louis, St Louis, Missouri
  • 4Department of Pediatrics, Washington University School of Medicine in St Louis, St Louis, Missouri
  • 5Alvin J Siteman Cancer Center, Washington University School of Medicine in St Louis, St Louis, Missouri
  • 6Prevention Research Center, Washington University School of Medicine in St Louis, St Louis, Missouri
JAMA Pediatr. 2017;171(4):342-349. doi:10.1001/jamapediatrics.2016.4520
Key Points

Question  Can routine clinical and imaging variables predict the need for intensive care unit admission among children with mild traumatic brain injury and intracranial injury?

Findings  This analysis found that the presence of midline shift, depressed skull fracture, epidural hematoma, and lower Glasgow Coma Scale score are associated with the need for intensive care unit admission in children with mild traumatic brain injury and intracranial injury. The negative predictive value of having none of these risk factors was 98.8%.

Meaning  Use of these risk criteria can help guide the need for intensive care unit admission in children with mild traumatic brain injury and intracranial injury.

Abstract

Importance  The appropriate treatment of children with mild traumatic brain injury (mTBI) and intracranial injury (ICI) on computed tomographic imaging remains unclear. Evidence-based risk assessments may improve patient safety and reduce resource use.

Objective  To derive a risk score predicting the need for intensive care unit observation in children with mTBI and ICI.

Design, Setting, and Participants  This retrospective analysis of the prospective Pediatric Emergency Care Applied Research Network (PECARN) head injury cohort study included patients enrolled in 25 North American emergency departments from 2004 to 2006. We included patients younger than 18 years with mTBI (Glasgow Coma Scale [GCS] score, 13-15) and ICI on computed tomography. The data analysis was conducted from May 2015 to October 2016.

Main Outcomes and Measures  The primary outcome was the composite of neurosurgical intervention, intubation for more than 24 hours for TBI, or death from TBI. Multivariate logistic regression was used to predict the outcome. The C statistic was used to quantify discrimination, and model performance was internally validated using 10-fold cross-validation. Based on this modeling, the Children’s Intracranial Injury Decision Aid score was created.

Results  Among 15 162 children with GCS 13 to 15 head injuries who received head computed tomographic imaging in the emergency department, 839 (5.5%) had ICI. The median ages of those with and without a composite outcome were 7 and 5 years, respectively. Among those patients with ICI, 8.7% (n = 73) experienced the primary outcome, including 8.3% (n = 70) who had a neurosurgical intervention. The only clinical variable significantly associated with outcome was GCS score (odds ratio [OR], 3.4; 95% CI, 1.5-7.4 for GCS score 13 vs 15). Significant radiologic predictors included midline shift (OR, 6.8; 95% CI, 3.4-13.8), depressed skull fracture (OR, 6.5; 95% CI, 3.7-11.4), and epidural hematoma (OR, 3.4; 95% CI, 1.8-6.2). The model C statistic was 0.84 (95% CI, 0.79-0.88); the 10-fold cross-validated C statistic was 0.83. Based on this modeling, we developed the Children’s Intracranial Injury Decision Aid score, which ranged from 0 to 24 points. The negative predictive value of having 0 points (ie, none of these risk factors) was 98.8% (95% CI, 97.3%-99.6%).

Conclusions and Relevance  Lower GCS score, midline shift, depressed skull fracture, and epidural hematoma are key risk factors for needing intensive care unit–level care in children with mTBI and ICI. Based on these results, the Children’s Intracranial Injury Decision Aid score is a potentially novel tool to risk stratify this population, thereby aiding management decisions.

Introduction

Traumatic brain injury (TBI) is one of the most common and potentially most devastating diagnoses affecting children in the United States, leading to almost 600 000 emergency department (ED) visits each year.1 Children with mild TBI (mTBI)—those with Glasgow Coma Scale (GCS) scores of 13 to 152,3—comprise the most common TBI subgroup, constituting more than 90% of new pediatric TBI cases4 and approximately one-third of the 50 000 to 60 000 TBI hospitalizations each year.5 Despite the prevalence and potentially significant harms affecting children with mTBI, there is significant variability and often uncertainty in the treatment of these patients.6,7

In 2009, the Pediatric Emergency Care Applied Research Network (PECARN) derived and validated clinical decision tools for identifying which children with minor head injuries are at risk for clinically important TBI.8 While this study helped identify which children with minor head injuries should receive computed tomographic (CT) imaging, among those with intracranial injury (ICI) identified on CT, there is limited evidence guiding appropriate care pathways. Such patients have often been categorized as having complicated mTBI, and multiple studies indicate that the long-term longitudinal course of this group differs from those with mTBI and normal CT findings.9,10 However, owing to persistent uncertainty regarding the risk for acute neurological decline in these patients, the appropriate initial level of care remains unclear. In particular, intensive care unit (ICU) observation may increase the likelihood of detecting early neurological worsening but is also associated with a variety of costs, including financial strain, resource burden,11 and emotional stress.12,13

While early evidence suggested ICU admission was necessary for children with complicated mTBI due to a high risk for neurosurgical intervention,14-16 recent findings indicated that most of these patients may be safely observed on a general ward.16,17 However, this evidence has been based on retrospectively collected single-center data, limiting those results.16,18 Consequently, there remains an important need to identify which patients with complicated mTBI are safe for ward observation.

The goal of this study was to use prospective, multicenter data to develop a generalizable clinical decision tool to risk stratify the need for ICU admission among children with complicated mTBI.

Methods
Study Population

This study involved a secondary analysis of the prospective PECARN cohort study of children with mTBI. The details of the population have been published elsewhere8; briefly, this observational study enrolled children and teens younger than 18 years who experienced nonpenetrating head trauma and presented to 1 of 25 North American EDs from 2004 to 2006. Patients were followed up with standardized telephone surveys of guardians and/or medical record review 7 to 90 days post-ED visit to ensure no outcomes were missed.8

All data were analyzed from a deidentified, public-use data set. Consequently, the Washington University in St Louis Human Research Protection Office deemed this study not subject to institutional review board review. For the initial data collection, the study was approved by the human subjects research committee at each site, and either a waiver of consent or verbal consent was used in the initial data collection. No additional consent was obtained for this subsequent analysis of the deidentified data. The data analysis was conducted from May 2015 to October 2016.

From the PECARN data set, we included children who had an ED CT scan that showed ICI, defined as intracranial hemorrhage, cerebral edema, skull diastasis, midline shift, pneumocephalus, depressed skull fracture (depressed by at least the width of the skull), traumatic infarction, diffuse axonal injury, herniation, shear injury, or sigmoid sinus thrombosis. The PECARN Study excluded children with trivial injury history or presentation (eg, running into stationary objects), as well as those with penetrating TBI, preexisting comorbid neurological disease, and bleeding disorders. The complete list of inclusion and exclusion criteria has been published previously.8 The sample size for this study was based on the available data from the PECARN cohort.

Selection of Predictor Variables

To derive a risk model, we evaluated the clinical and radiologic variables listed in eTable 1 in the Supplement. Predefined imaging variables were abstracted from radiology reports, with investigators able to add additional findings if relevant.19 Site principal investigators approved all imaging findings and quality checks were performed for unusual or inconsistent findings. While the PECARN investigators required a fracture to be depressed by at least the width of the skull to be sufficient for a diagnosis of ICI, the original data set did not otherwise distinguish depressed from nondepressed fractures. Because primary CT images were not available for review, we defined the variable for depressed skull fracture by reviewing radiologist impressions from CT scan reports for any mention of fracture depression or displacement in patients with known skull fracture.

The GCS scores were recorded at the time the patient was first evaluated by the ED team as part of routine standard of care. Injury mechanism was stratified a priori into severe (eg, rollover motor vehicle collision), mild (eg, ground-level falls), and moderate (all other mechanisms) categories, as previously described.8 All of the predictor variables evaluated (except scalp hematoma) had κ statistics of 0.5 or more, constituting at least moderate reliability by traditional measures.8,20

Outcome Variable

The primary study outcome was the composite of neurosurgical intervention (eg, intracranial pressure monitor placement and hematoma evacuation), intubation for more than 24 hours for head trauma, or death. These events were chosen because they indicated a significant objective worsening in a patient who initially appeared to have a minor head injury and indicated a strong need for critical care observation. Separately, we also evaluated the frequency of hospitalization for 2 or more nights due to TBI.

Statistical Analysis

Several variables in the data set had missing data. For dichotomous variables with 5% or less missing data, we assumed “0” (ie, not present) values for missing data. For categorical variables with missing data and all variables with 6% or more with missing data, we performed multiple imputation with 5 imputed data sets, using previously described approaches for missing data exploration and imputation.21-23 We encountered several pairs of variables with missing data that required conditional imputation not offered in standard statistical software. For instance, for a patient with known headache but missing headache severity, headache severity could only be imputed as mild, moderate, or severe, but not as no headache. Consequently, we developed novel postprocessing functions using the MICE package in R,22,23 which are described in the eAppendix in the Supplement.

To develop a risk model predicting the composite outcome, we used multivariate logistic regression. We first conducted univariate regression analyses and entered variables with P < .20 into the multivariate model. We used forward selection to retain variables with P < .05 in more than 50% of the imputed data sets. We then used Rubin’s rules to combine results across the multiple imputed data sets for a unified inference.24 For the final model, the variable for amnesia was removed because of its inverse correlation with the composite outcome (ie, amnesia was associated with lower risk for the outcome) because that finding countered basic clinical intuition—as determined by the research team—and appeared to be an artifact of the study data set and the likely difficulty assessing that variable.

Model performance was evaluated by examining both discrimination and calibration. Discrimination was assessed using the C statistic, which represented the area under the receiver operating characteristic curve.25 Bootstrap methods were used to obtain the bias-corrected confidence interval of the C statistic. Calibration was evaluated statistically using the calibration-in-the-large and a calibration slope. The calibration-in-the-large, which is the intercept of the calibration plot, indicates the degree to which predictions are systematically too low or too high and should be near 0 in a well-calibrated model.26 The calibration slope should be 1 in a perfectly calibrated model, reflecting predictions on the 45° line of observed vs expected outcomes.26 In addition to these statistics, we evaluated calibration graphically using a calibration plot of observed vs expected probabilities. Finally, we internally validated model performance using 10-fold cross-validation.27,28 Using this approach, the study sample was divided into deciles and models were developed using 90% of the sample and then tested using the remaining 10%. This process was repeated 10 times, and the C statistic was averaged across the 10 repetitions to obtain a mean result.

While the study data set lacked a variable for non–TBI-related indications for ICU admission, we tested the C statistic of the final model among a restricted subpopulation of children that lacked any significant noncranial injuries.

To create a clinically usable risk score from our final multivariate model, we used previously described methods29 to assign integer point values to each variable in the model. We assessed the performance of different score cutoffs by evaluating sensitivity, specificity, positive and negative predictive values, and the positive and negative likelihood ratios. Ninety-five percent CIs were calculated using exact methods.

Treatment Based on Risk Score

To investigate how patients’ risk score related to their clinical treatment, we plotted the distribution of ED disposition vs risk score. In this comparison, we dichotomized disposition into low acuity (ie, home, general inpatient, or short stay/observation) or high acuity (operating room or ICU). For simplicity, we excluded the small number of patients with missing disposition or patients sent to other dispositions. We used the R2 statistic to quantify the relationship between disposition and risk score.

All analyses were conducted using R statistical software and associated packages (R Foundation for Statistical Computing)22,23,30 and with SAS version 9.4 (SAS Institute Inc). Two-tailed P values less than .05 were used to define statistically significant outcomes. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guidelines were followed in reporting this study’s methods and results.31

Results

There were 42 735 children with mTBI (GCS score, 13-15) in the PECARN data set, and 15 162 received a head CT scan in the ED. Within that group, 839 children had ICI and constituted the study cohort. Most children (n = 611, 72.8%) had a GCS score of 15, and 65.4% (n = 549) were 2 years of age or older. The most common abnormal CT findings included nondepressed skull fracture (n = 362, 43.1%), contusion/intraparenchymal hematoma (n = 267, 31.8%) and subdural hematoma (n = 207, 24.7%). Detailed population demographics, clinical presentations, and CT findings are shown in Table 1.

From the ED, 38.4% of patients (n = 322) with complicated mTBI were admitted to a general ward and 8.6% (n = 72) were admitted to a short-stay unit, while a slightly smaller number (n = 309, 36.8%) were admitted to the ICU. A small number of patients (n = 68, 8.1%) were sent home, while the remainder were sent directly to the operating room (n = 29, 3.5%), transferred to another hospital (n = 20, 2.4%), or did not have their disposition specified (n = 19, 2.3%). In total, 73 patients (8.7%) experienced the primary outcome, including 70 (8.3%) who had neurosurgical intervention and 11 (1.3%) who were intubated for more than 24 hours for TBI. The frequency of different types of neurosurgical interventions is shown in eTable 2 in the Supplement. No patients died of TBI. Half (50.2%) of the population required hospitalization for 2 or more nights due to TBI.

Risk Model Creation

The univariate statistical significance for all variables with P values less than .20 are shown in Table 2. Several clinical variables, including GCS score, presence of a neurological deficit, and severe headache, were significantly associated with the composite outcome. Likewise, multiple imaging variables—including epidural hematoma, depressed skull fracture, and midline shift—were associated with significantly increased risk, whereas subarachnoid hemorrhage was associated with lower risk.

The results of the multivariate analysis are shown in Table 2. The only clinical variable retained in the final model was GCS score (odds ratio [OR], 3.4; 95% CI, 1.5-7.4 for GCS score 13 vs 15). Among the imaging findings, the presence of an epidural hematoma (OR, 3.4; 95% CI, 1.8-6.2), midline shift (OR, 6.8; 95% CI, 3.4-13.8), and depressed skull fracture (OR, 6.5; 95% CI, 3.7-11.4) were all associated with significantly increased risk. The model C statistic was 0.84 (95% CI, 0.79-0.88). The 10-fold cross-validated C statistic was 0.83. Among the subpopulation of children with no significant noncranial injuries, the C statistic was 0.85 (95% CI, 0.80-0.90).

The calibration-in-the-large statistic was −0.08, indicating low systemic overprediction or underprediction. The calibration slope was 0.95, indicating close agreement between predicted and observed event rates. Examination of the calibration plot revealed that the model was well calibrated at low to moderate risk levels, but overpredicted risk for patients with approximately 50% or higher risk for the outcome (eFigure in the Supplement).

Children’s Intracranial Injury Decision Aid Risk Score

Based on the multivariate risk model, we developed the Children’s Intracranial Injury Decision Aid (CHIIDA) score to predict the need for ICU admission (Box). Each variable in the model was assigned a point value ranging from 2 to 7, and each patient’s score could range from 0 to 24. As shown in Figure 1, the predicted risk of the composite outcome ranged from 2.6% for patients with 0 points to a maximum of 92.5% for patients with 24 points.

Box Section Ref ID
Box.

CHIIDA Point Values for the Different Risk Factors

  • Depressed skull fracture, 7 points

  • Midline shift, 7 points

  • Epidural hematoma, 5 points

  • Glasgow Coma Scale score of 13, 5 points

  • Glasgow Coma Scale score of 14, 2 points

Abbreviation: CHIIDA, Children’s Intracranial Injury Decision Aid.

Using a cutoff criterion of more than 0 points to admit to the ICU had a sensitivity of 93.2% (95% CI, 84.7%-97.7%) and a negative predictive value of 98.8% (95% CI, 97.3%-99.6%) (Table 3). Using this cutoff would have avoided ICU admission in 51.3% of patients. Using a cutoff criterion of more than 2 points had a sensitivity of 86.3% (95% CI, 76.3%-93.2%) and a negative predictive value of 98.2% (95% CI, 96.7%-99.1%). Using this cutoff would have avoided ICU admission in 65.4% of patients.

The proportion of patients with high vs low acuity ED dispositions at each level of the CHIIDA score is shown in Figure 2. As shown in Figure 2, for most risk levels, there was significant variability in disposition, with the level of care often not corresponding to patients’ evidence-based predicted risk. The R2 statistic of the point score vs disposition was 0.38, indicating that the CHIIDA score explained less than 40% of the variability in disposition decision.

Discussion

In this study, we describe the development of the CHIIDA score, a novel clinical decision tool for predicting the post–head CT need for ICU admission among pediatric patients with complicated mTBI. Using data from a multicenter prospective study, we identified GCS score as the sole independent clinical predictor of needing ICU care, emphasizing the importance of accurately assessing this metric. By comparison, the presence of midline shift, epidural hematoma, and depressed skull fracture were all significant radiologic predictors. The final model had high discrimination and calibration and effectively identified low-risk populations that could typically avoid ICU admission. In addition, we demonstrated that current post-CT management practices are highly variable and often not related to patients’ evidence-based risk, further emphasizing the importance of the CHIIDA score.

While there are ongoing large-scale efforts to evaluate best practices among children with severe TBI—including the Approaches and Decisions in Acute TBI (ADAPT) Trial32—the post-CT treatment of patients with mTBI has not received the same attention. The most compelling reason to risk stratify children with complicated mTBI is to ensure that those at high risk for serious events receive appropriate attention and monitoring. The relatively weak association—ie, low R2—between ED disposition and CHIIDA score suggests that factors such as individual gestalt impression and institutional culture may be having a significant influence on disposition decisions. Most importantly, this weak association indicates that many high-risk children may be receiving insufficient attention.

Although appropriate ICU monitoring for high-risk children remains paramount, universal ICU admission, independent of patient risk, is also associated with a variety of costs. While exact figures vary, by some estimates, ICU stays are associated with several thousand dollars per day in increased charges compared with a general ward.33,34 In addition, pediatric ICU beds are a limited resource—approximately 2000 total beds in the United States—and potentially unnecessary admissions may limit access for other children in greater need.11,35 Finally, pediatric ICU admission is often a major emotional burden for families that can potentially be avoided in many circumstances.12,13,36

A large part of the practice variability demonstrated in this study may result from divergent results in the existing literature. For example, previous studies have shown that 0% to 43% of children with complicated mTBI required neurosurgical intervention, likely reflecting the retrospective, single-center nature and variable inclusion criteria across these analyses.14-18,37-39 Likewise, to our knowledge, the only other study that identified independent predictors of outcome among children with complicated mTBI was a single-center analysis with only 29 outcomes,18 emphasizing the value of using multicenter data.

Leveraging the large-scale PECARN data set, we developed the CHIIDA score to aid patient treatment. While external validation is still needed and individual decisions should consider institutional experience, physician judgment, and family comfort, we recommend that nearly all children with 0 points and many children with less than 3 points be admitted to a general ward. Using these cutoffs, the negative predictive value of a major event is 98% to 99%, and 50% to 65% of patients could avoid ICU admission. In contrast, patients at higher risk levels should likely be admitted to an ICU or higher-acuity stepdown unit in most circumstances. By following this evidence-based framework, the CHIIDA score may both reduce resource use and improve patient safety, potentially limiting the practice variations demonstrated in this study. Other possible benefits, such as facilitating shared decision making40,41 and helping guide hospital transfer practices, may be explored in future work.

Limitations

Despite these advances, this study had several limitations, which serve as directions for future research. First, because primary CT images were not available for review, we were unable to grade imaging findings based on size or severity (eg, large vs small hemorrhage and degree of skull fracture depression). Future analyses should evaluate the extent to which more detailed radiologic parameters improve risk prediction and clinician acceptance. Second, despite the large sample size, there were relatively few patients at the highest risk levels, potentially explaining the lower model calibration for that group. However, given the almost certain need for ICU observation among these patients, precise model calibration is likely less important in this subgroup. Third, some aspects of surgical decision making, such as deciding whether to elevate a depressed skull fracture, involved clinical judgment and may have varied among surgeons, complicating risk predictions.42 Fourth, owing to limitations in the study data set, we were unable to determine surgical indications, such as an expanding hematoma vs clinical decline. Finally, this tool will need to be validated in future studies to verify its prognostic accuracy and evaluate key implementation outcomes, such as physician acceptance.43

Conclusions

Using data from a large, prospectively collected, multicenter data set, we found that lower GCS score, midline shift, depressed skull fracture, and epidural hematoma are independent predictors of needing ICU-level care in children with complicated mTBI. These factors served as the basis for developing the CHIIDA score, a decision tool to aid physicians treating these patients.

Back to top
Article Information

Accepted for Publication: November 16, 2016.

Corresponding Author: Jacob K. Greenberg, MD, MSCI, St Louis Children’s Hospital, Washington University School of Medicine, 1 Children’s Pl, Ste 4S20, St Louis, MO 63110 (greenbergj@wudosis.wustl.edu).

Published Online: February 13, 2017. doi:10.1001/jamapediatrics.2016.4520

Author Contributions: Dr Greenberg had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Greenberg, Carpenter, Lumba-Brown, Keller, Pineda, Brownson, Limbrick.

Acquisition, analysis, or interpretation of data: Greenberg, Yan, Carpenter, Pineda, Limbrick.

Drafting of the manuscript: Greenberg, Yan, Carpenter, Lumba-Brown.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Greenberg, Yan.

Obtained funding: Greenberg, Limbrick.

Administrative, technical, or material support: Carpenter, Lumba-Brown, Brownson, Limbrick.

Study supervision: Carpenter, Lumba-Brown, Limbrick.

Conflict of Interest Disclosures: None reported.

Funding/Support: This work was supported by grant FPP-1501 from the Washington University School of Medicine Faculty Practice Plan and the St Louis Children’s Hospital Foundation. The Pediatric Emergency Care Applied Research Network was funded by the Health Resources Services Administration/Maternal Child Health Bureau /Emergency Medical Services for Children.

Role of the Funder/Sponsor: The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: This article was prepared using the “Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study (TBI study)” data set obtained from the University of Utah School of Medicine and does not necessarily reflect the opinions or views of the TBI Trial investigators or the Health Resources Services Administration/Maternal Child Health Bureau/Emergency Medical Services for Children.

Additional Contributions: We thank Margaret Olsen, PhD, MPH (Department of Internal Medicine, Washington University in St Louis, St Louis, Missouri), for her insightful comments and suggestions related to this study. She did not receive compensation.

References
1.
Mannix  R, O’Brien  MJ, Meehan  WP  III.  The epidemiology of outpatient visits for minor head injury: 2005 to 2009.  Neurosurgery. 2013;73(1):129-134.PubMedGoogle ScholarCrossref
2.
National Center for Injury Prevention and Control.  Report to Congress on Mild Traumatic Brain Injury in the United States: Steps to Prevent a Serious Public Health Problem. Atlanta, GA: National Center for Injury Prevention and Control; 2003.
3.
Centers for Disease Control and Prevention. Injury prevention & control: traumatic brain injury & concussion. http://www.cdc.gov/traumaticbraininjury/severe.html. Accessed April 5, 2016.
4.
Koepsell  TD, Rivara  FP, Vavilala  MS,  et al.  Incidence and descriptive epidemiologic features of traumatic brain injury in King County, Washington.  Pediatrics. 2011;128(5):946-954.PubMedGoogle ScholarCrossref
5.
Bowman  SM, Bird  TM, Aitken  ME, Tilford  JM.  Trends in hospitalizations associated with pediatric traumatic brain injuries.  Pediatrics. 2008;122(5):988-993.PubMedGoogle ScholarCrossref
6.
Mannix  R, Meehan  WP, Monuteaux  MC, Bachur  RG.  Computed tomography for minor head injury: variation and trends in major United States pediatric emergency departments.  J Pediatr. 2012;160(1):136-9.e1.PubMedGoogle ScholarCrossref
7.
Di  F, Gao  Q, Xiang  J,  et al.  Clinical practice experiences in diagnosis and treatment of traumatic brain injury in children: a survey among clinicians at 9 large hospitals in China.  PLoS One. 2015;10(11):e0142983.PubMedGoogle ScholarCrossref
8.
Kuppermann  N, Holmes  JF, Dayan  PS,  et al; Pediatric Emergency Care Applied Research Network (PECARN).  Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study.  Lancet. 2009;374(9696):1160-1170.PubMedGoogle ScholarCrossref
9.
Yeates  KO, Taylor  HG, Rusin  J,  et al.  Longitudinal trajectories of postconcussive symptoms in children with mild traumatic brain injuries and their relationship to acute clinical status.  Pediatrics. 2009;123(3):735-743.PubMedGoogle ScholarCrossref
10.
Rivara  FP, Koepsell  TD, Wang  J,  et al.  Disability 3, 12, and 24 months after traumatic brain injury among children and adolescents.  Pediatrics. 2011;128(5):e1129-e1138.PubMedGoogle ScholarCrossref
11.
Wang  HE, Yealy  DM.  Distribution of specialized care centers in the United States.  Ann Emerg Med. 2012;60(5):632-637.e7.PubMedGoogle ScholarCrossref
12.
Colville  G, Darkins  J, Hesketh  J, Bennett  V, Alcock  J, Noyes  J.  The impact on parents of a child’s admission to intensive care: integration of qualitative findings from a cross-sectional study.  Intensive Crit Care Nurs. 2009;25(2):72-79.PubMedGoogle ScholarCrossref
13.
Balluffi  A, Kassam-Adams  N, Kazak  A, Tucker  M, Dominguez  T, Helfaer  M.  Traumatic stress in parents of children admitted to the pediatric intensive care unit.  Pediatr Crit Care Med. 2004;5(6):547-553.PubMedGoogle ScholarCrossref
14.
Boran  BOB, Boran  P, Barut  N, Akgun  C, Celikoglu  E, Bozbuga  M.  Evaluation of mild head injury in a pediatric population.  Pediatr Neurosurg. 2006;42(4):203-207.PubMedGoogle ScholarCrossref
15.
Simon  B, Letourneau  P, Vitorino  E, McCall  J.  Pediatric minor head trauma: indications for computed tomographic scanning revisited.  J Trauma. 2001;51(2):231-237.PubMedGoogle ScholarCrossref
16.
Greenberg  JK, Stoev  IT, Park  TS,  et al.  Management of children with mild traumatic brain injury and intracranial hemorrhage.  J Trauma Acute Care Surg. 2014;76(4):1089-1095.PubMedGoogle ScholarCrossref
17.
Quayle  KS, Powell  EC, Mahajan  P,  et al.  Epidemiology of blunt head trauma in children in US emergency departments.  N Engl J Med. 2014;371(20):1945-1947.PubMedGoogle ScholarCrossref
18.
Burns  EC, Burns  B, Newgard  CD,  et al.  Pediatric minor traumatic brain injury with intracranial hemorrhage: identifying low-risk patients who may not benefit from ICU admission [published online October 28, 2016].  Pediatr Emerg Care. doi:10.1097/PEC.0000000000000950PubMedGoogle Scholar
19.
Pediatric Emergency Care Applied Research Network. Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study. http://www.pecarn.org/studyDatasets/StudyDetails?studyID=4. Revised September 28, 2015. Accessed October 22, 2016.
20.
Landis  JR, Koch  GG.  The measurement of observer agreement for categorical data.  Biometrics. 1977;33(1):159-174.PubMedGoogle ScholarCrossref
21.
van Buuren  S, Boshuizen  HC, Knook  DL.  Multiple imputation of missing blood pressure covariates in survival analysis.  Stat Med. 1999;18(6):681-694.PubMedGoogle ScholarCrossref
22.
Van Buuren  S, Groothuis-Oudshoorn  K.  Mice: multivariate imputation by chained equations in R.  J Stat Softw. 2011;45(3). doi:10.18637/jss.v045.i03Google Scholar
23.
R Core Team. R Foundation for Statistical Computing. R: a language and environment for statistical computing. http://www.R-project.org/. Accessed October 10, 2016.
24.
Rubin  DB, Schenker  N.  Multiple imputation in health-care databases: an overview and some applications.  Stat Med. 1991;10(4):585-598.PubMedGoogle ScholarCrossref
25.
Hanley  JA, McNeil  BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve.  Radiology. 1982;143(1):29-36.PubMedGoogle ScholarCrossref
26.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138.PubMedGoogle ScholarCrossref
27.
Altman  DG, Vergouwe  Y, Royston  P, Moons  KG.  Prognosis and prognostic research: validating a prognostic model.  BMJ. 2009;338:b605.PubMedGoogle ScholarCrossref
28.
Wells  BJ, Jain  A, Arrigain  S, Yu  C, Rosenkrans  WA  Jr, Kattan  MW.  Predicting 6-year mortality risk in patients with type 2 diabetes.  Diabetes Care. 2008;31(12):2301-2306.PubMedGoogle ScholarCrossref
29.
Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use: The Framingham Study risk score functions.  Stat Med. 2004;23(10):1631-1660.PubMedGoogle ScholarCrossref
30.
rms: Regression Modeling Strategies [computer program]. R package version 4.4-1. Vienna, Austria: R Foundation for Statistical Computing; 2015.
31.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model For Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD Statement.  BMC Med. 2015;13(1):1.PubMedGoogle ScholarCrossref
32.
Bell  MJ, Wisniewski  SR.  Severe traumatic brain injury in children: a vision for the future.  Intensive Care Med. 2016;42(10):1618-1620.PubMedGoogle ScholarCrossref
33.
University Hospitals Rainbow Babies & Children's Hospital. Patient pricing information. http://www.uhhospitals.org/rainbow/patients-and-visitors/billing-insurance-and-medical-records/patient-pricing-information. Accessed April 9, 2016.
34.
Nationwide Children's. Patient price information list. http://www.nationwidechildrens.org/price-information-list. Accessed April 9, 2016.
35.
Halpern  NA, Pastores  SM.  Critical care medicine in the United States 2000-2005: an analysis of bed numbers, occupancy rates, payer mix, and costs.  Crit Care Med. 2010;38(1):65-71.PubMedGoogle ScholarCrossref
36.
Shudy  M, de Almeida  ML, Ly  S,  et al.  Impact of pediatric critical illness and injury on families: a systematic literature review.  Pediatrics. 2006;118(suppl 3):S203-S218.PubMedGoogle ScholarCrossref
37.
Wang  MYG, Griffith  P, Sterling  J, McComb  JG, Levy  ML.  A prospective population-based study of pediatric trauma patients with mild alterations in consciousness (Glasgow Coma Scale score of 13-14).  Neurosurgery. 2000;46(5):1093-1099.PubMedGoogle ScholarCrossref
38.
Holsti  M, Kadish  HA, Sill  BL, Firth  SD, Nelson  DS.  Pediatric closed head injuries treated in an observation unit.  Pediatr Emerg Care. 2005;21(10):639-644.PubMedGoogle ScholarCrossref
39.
Atabaki  SM, Stiell  IG, Bazarian  JJ,  et al.  A clinical decision rule for cranial computed tomography in minor pediatric head trauma.  Arch Pediatr Adolesc Med. 2008;162(5):439-445.PubMedGoogle ScholarCrossref
40.
Kanzaria  HK, Brook  RH, Probst  MA, Harris  D, Berry  SH, Hoffman  JR.  Emergency physician perceptions of shared decision-making.  Acad Emerg Med. 2015;22(4):399-405.PubMedGoogle ScholarCrossref
41.
Hess  EP, Grudzen  CR, Thomson  R, Raja  AS, Carpenter  CR.  Shared decision-making in the emergency department: respecting patient autonomy when seconds count.  Acad Emerg Med. 2015;22(7):856-864.PubMedGoogle ScholarCrossref
42.
Bullock  MR, Chesnut  R, Ghajar  J,  et al; Surgical Management of Traumatic Brain Injury Author Group.  Surgical management of depressed cranial fractures.  Neurosurgery. 2006;58(3)(suppl):S56-S60, discussion Si-iv.PubMedGoogle Scholar
43.
Hunter  B, Carpenter  CR. The development of clinical prediction rules. In: Wilson  MP, Guluma  KZ, Hayden  S, eds.  Doing Research in Emergency and Acute Care: Making Order Out of Chaos. Chichester, West Sussex, England: Wiley-Blackwell; 2015:139-148.
×