The graph shows the risk for the composite outcome at different point levels.
The proportion of patients who, at each level of the Children’s Intracranial Injury Decision Aid (CHIIDA) risk score, were sent to high- vs low-acuity emergency department dispositions based on clinical assessment alone. The open circles indicate a low-acuity disposition (ie, home, general inpatient, or short-stay/observation unit). The darker (solid) circles indicate a high-acuity disposition (intensive care unit or the operating room). The R2 statistic of the CHIIDA score vs disposition was 0.38, indicating a relatively weak correlation.
eAppendix. Imputation Methods.
eTable 1. A list of the Variables Evaluated as Potential Predictors of the Need for ICU Admission.
eTable 2. A List of the Neurosurgical Interventions That Occurred in the Study Cohort.
eFigure. Calibration Curve Showing Observed vs Expected Probabilities Across Risk Levels.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Greenberg JK, Yan Y, Carpenter CR, et al. Development and Internal Validation of a Clinical Risk Score for Treating Children With Mild Head Trauma and Intracranial Injury. JAMA Pediatr. 2017;171(4):342–349. doi:10.1001/jamapediatrics.2016.4520
Can routine clinical and imaging variables predict the need for intensive care unit admission among children with mild traumatic brain injury and intracranial injury?
This analysis found that the presence of midline shift, depressed skull fracture, epidural hematoma, and lower Glasgow Coma Scale score are associated with the need for intensive care unit admission in children with mild traumatic brain injury and intracranial injury. The negative predictive value of having none of these risk factors was 98.8%.
Use of these risk criteria can help guide the need for intensive care unit admission in children with mild traumatic brain injury and intracranial injury.
The appropriate treatment of children with mild traumatic brain injury (mTBI) and intracranial injury (ICI) on computed tomographic imaging remains unclear. Evidence-based risk assessments may improve patient safety and reduce resource use.
To derive a risk score predicting the need for intensive care unit observation in children with mTBI and ICI.
Design, Setting, and Participants
This retrospective analysis of the prospective Pediatric Emergency Care Applied Research Network (PECARN) head injury cohort study included patients enrolled in 25 North American emergency departments from 2004 to 2006. We included patients younger than 18 years with mTBI (Glasgow Coma Scale [GCS] score, 13-15) and ICI on computed tomography. The data analysis was conducted from May 2015 to October 2016.
Main Outcomes and Measures
The primary outcome was the composite of neurosurgical intervention, intubation for more than 24 hours for TBI, or death from TBI. Multivariate logistic regression was used to predict the outcome. The C statistic was used to quantify discrimination, and model performance was internally validated using 10-fold cross-validation. Based on this modeling, the Children’s Intracranial Injury Decision Aid score was created.
Among 15 162 children with GCS 13 to 15 head injuries who received head computed tomographic imaging in the emergency department, 839 (5.5%) had ICI. The median ages of those with and without a composite outcome were 7 and 5 years, respectively. Among those patients with ICI, 8.7% (n = 73) experienced the primary outcome, including 8.3% (n = 70) who had a neurosurgical intervention. The only clinical variable significantly associated with outcome was GCS score (odds ratio [OR], 3.4; 95% CI, 1.5-7.4 for GCS score 13 vs 15). Significant radiologic predictors included midline shift (OR, 6.8; 95% CI, 3.4-13.8), depressed skull fracture (OR, 6.5; 95% CI, 3.7-11.4), and epidural hematoma (OR, 3.4; 95% CI, 1.8-6.2). The model C statistic was 0.84 (95% CI, 0.79-0.88); the 10-fold cross-validated C statistic was 0.83. Based on this modeling, we developed the Children’s Intracranial Injury Decision Aid score, which ranged from 0 to 24 points. The negative predictive value of having 0 points (ie, none of these risk factors) was 98.8% (95% CI, 97.3%-99.6%).
Conclusions and Relevance
Lower GCS score, midline shift, depressed skull fracture, and epidural hematoma are key risk factors for needing intensive care unit–level care in children with mTBI and ICI. Based on these results, the Children’s Intracranial Injury Decision Aid score is a potentially novel tool to risk stratify this population, thereby aiding management decisions.
Traumatic brain injury (TBI) is one of the most common and potentially most devastating diagnoses affecting children in the United States, leading to almost 600 000 emergency department (ED) visits each year.1 Children with mild TBI (mTBI)—those with Glasgow Coma Scale (GCS) scores of 13 to 152,3—comprise the most common TBI subgroup, constituting more than 90% of new pediatric TBI cases4 and approximately one-third of the 50 000 to 60 000 TBI hospitalizations each year.5 Despite the prevalence and potentially significant harms affecting children with mTBI, there is significant variability and often uncertainty in the treatment of these patients.6,7
In 2009, the Pediatric Emergency Care Applied Research Network (PECARN) derived and validated clinical decision tools for identifying which children with minor head injuries are at risk for clinically important TBI.8 While this study helped identify which children with minor head injuries should receive computed tomographic (CT) imaging, among those with intracranial injury (ICI) identified on CT, there is limited evidence guiding appropriate care pathways. Such patients have often been categorized as having complicated mTBI, and multiple studies indicate that the long-term longitudinal course of this group differs from those with mTBI and normal CT findings.9,10 However, owing to persistent uncertainty regarding the risk for acute neurological decline in these patients, the appropriate initial level of care remains unclear. In particular, intensive care unit (ICU) observation may increase the likelihood of detecting early neurological worsening but is also associated with a variety of costs, including financial strain, resource burden,11 and emotional stress.12,13
While early evidence suggested ICU admission was necessary for children with complicated mTBI due to a high risk for neurosurgical intervention,14-16 recent findings indicated that most of these patients may be safely observed on a general ward.16,17 However, this evidence has been based on retrospectively collected single-center data, limiting those results.16,18 Consequently, there remains an important need to identify which patients with complicated mTBI are safe for ward observation.
The goal of this study was to use prospective, multicenter data to develop a generalizable clinical decision tool to risk stratify the need for ICU admission among children with complicated mTBI.
This study involved a secondary analysis of the prospective PECARN cohort study of children with mTBI. The details of the population have been published elsewhere8; briefly, this observational study enrolled children and teens younger than 18 years who experienced nonpenetrating head trauma and presented to 1 of 25 North American EDs from 2004 to 2006. Patients were followed up with standardized telephone surveys of guardians and/or medical record review 7 to 90 days post-ED visit to ensure no outcomes were missed.8
All data were analyzed from a deidentified, public-use data set. Consequently, the Washington University in St Louis Human Research Protection Office deemed this study not subject to institutional review board review. For the initial data collection, the study was approved by the human subjects research committee at each site, and either a waiver of consent or verbal consent was used in the initial data collection. No additional consent was obtained for this subsequent analysis of the deidentified data. The data analysis was conducted from May 2015 to October 2016.
From the PECARN data set, we included children who had an ED CT scan that showed ICI, defined as intracranial hemorrhage, cerebral edema, skull diastasis, midline shift, pneumocephalus, depressed skull fracture (depressed by at least the width of the skull), traumatic infarction, diffuse axonal injury, herniation, shear injury, or sigmoid sinus thrombosis. The PECARN Study excluded children with trivial injury history or presentation (eg, running into stationary objects), as well as those with penetrating TBI, preexisting comorbid neurological disease, and bleeding disorders. The complete list of inclusion and exclusion criteria has been published previously.8 The sample size for this study was based on the available data from the PECARN cohort.
To derive a risk model, we evaluated the clinical and radiologic variables listed in eTable 1 in the Supplement. Predefined imaging variables were abstracted from radiology reports, with investigators able to add additional findings if relevant.19 Site principal investigators approved all imaging findings and quality checks were performed for unusual or inconsistent findings. While the PECARN investigators required a fracture to be depressed by at least the width of the skull to be sufficient for a diagnosis of ICI, the original data set did not otherwise distinguish depressed from nondepressed fractures. Because primary CT images were not available for review, we defined the variable for depressed skull fracture by reviewing radiologist impressions from CT scan reports for any mention of fracture depression or displacement in patients with known skull fracture.
The GCS scores were recorded at the time the patient was first evaluated by the ED team as part of routine standard of care. Injury mechanism was stratified a priori into severe (eg, rollover motor vehicle collision), mild (eg, ground-level falls), and moderate (all other mechanisms) categories, as previously described.8 All of the predictor variables evaluated (except scalp hematoma) had κ statistics of 0.5 or more, constituting at least moderate reliability by traditional measures.8,20
The primary study outcome was the composite of neurosurgical intervention (eg, intracranial pressure monitor placement and hematoma evacuation), intubation for more than 24 hours for head trauma, or death. These events were chosen because they indicated a significant objective worsening in a patient who initially appeared to have a minor head injury and indicated a strong need for critical care observation. Separately, we also evaluated the frequency of hospitalization for 2 or more nights due to TBI.
Several variables in the data set had missing data. For dichotomous variables with 5% or less missing data, we assumed “0” (ie, not present) values for missing data. For categorical variables with missing data and all variables with 6% or more with missing data, we performed multiple imputation with 5 imputed data sets, using previously described approaches for missing data exploration and imputation.21-23 We encountered several pairs of variables with missing data that required conditional imputation not offered in standard statistical software. For instance, for a patient with known headache but missing headache severity, headache severity could only be imputed as mild, moderate, or severe, but not as no headache. Consequently, we developed novel postprocessing functions using the MICE package in R,22,23 which are described in the eAppendix in the Supplement.
To develop a risk model predicting the composite outcome, we used multivariate logistic regression. We first conducted univariate regression analyses and entered variables with P < .20 into the multivariate model. We used forward selection to retain variables with P < .05 in more than 50% of the imputed data sets. We then used Rubin’s rules to combine results across the multiple imputed data sets for a unified inference.24 For the final model, the variable for amnesia was removed because of its inverse correlation with the composite outcome (ie, amnesia was associated with lower risk for the outcome) because that finding countered basic clinical intuition—as determined by the research team—and appeared to be an artifact of the study data set and the likely difficulty assessing that variable.
Model performance was evaluated by examining both discrimination and calibration. Discrimination was assessed using the C statistic, which represented the area under the receiver operating characteristic curve.25 Bootstrap methods were used to obtain the bias-corrected confidence interval of the C statistic. Calibration was evaluated statistically using the calibration-in-the-large and a calibration slope. The calibration-in-the-large, which is the intercept of the calibration plot, indicates the degree to which predictions are systematically too low or too high and should be near 0 in a well-calibrated model.26 The calibration slope should be 1 in a perfectly calibrated model, reflecting predictions on the 45° line of observed vs expected outcomes.26 In addition to these statistics, we evaluated calibration graphically using a calibration plot of observed vs expected probabilities. Finally, we internally validated model performance using 10-fold cross-validation.27,28 Using this approach, the study sample was divided into deciles and models were developed using 90% of the sample and then tested using the remaining 10%. This process was repeated 10 times, and the C statistic was averaged across the 10 repetitions to obtain a mean result.
While the study data set lacked a variable for non–TBI-related indications for ICU admission, we tested the C statistic of the final model among a restricted subpopulation of children that lacked any significant noncranial injuries.
To create a clinically usable risk score from our final multivariate model, we used previously described methods29 to assign integer point values to each variable in the model. We assessed the performance of different score cutoffs by evaluating sensitivity, specificity, positive and negative predictive values, and the positive and negative likelihood ratios. Ninety-five percent CIs were calculated using exact methods.
To investigate how patients’ risk score related to their clinical treatment, we plotted the distribution of ED disposition vs risk score. In this comparison, we dichotomized disposition into low acuity (ie, home, general inpatient, or short stay/observation) or high acuity (operating room or ICU). For simplicity, we excluded the small number of patients with missing disposition or patients sent to other dispositions. We used the R2 statistic to quantify the relationship between disposition and risk score.
All analyses were conducted using R statistical software and associated packages (R Foundation for Statistical Computing)22,23,30 and with SAS version 9.4 (SAS Institute Inc). Two-tailed P values less than .05 were used to define statistically significant outcomes. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guidelines were followed in reporting this study’s methods and results.31
There were 42 735 children with mTBI (GCS score, 13-15) in the PECARN data set, and 15 162 received a head CT scan in the ED. Within that group, 839 children had ICI and constituted the study cohort. Most children (n = 611, 72.8%) had a GCS score of 15, and 65.4% (n = 549) were 2 years of age or older. The most common abnormal CT findings included nondepressed skull fracture (n = 362, 43.1%), contusion/intraparenchymal hematoma (n = 267, 31.8%) and subdural hematoma (n = 207, 24.7%). Detailed population demographics, clinical presentations, and CT findings are shown in Table 1.
From the ED, 38.4% of patients (n = 322) with complicated mTBI were admitted to a general ward and 8.6% (n = 72) were admitted to a short-stay unit, while a slightly smaller number (n = 309, 36.8%) were admitted to the ICU. A small number of patients (n = 68, 8.1%) were sent home, while the remainder were sent directly to the operating room (n = 29, 3.5%), transferred to another hospital (n = 20, 2.4%), or did not have their disposition specified (n = 19, 2.3%). In total, 73 patients (8.7%) experienced the primary outcome, including 70 (8.3%) who had neurosurgical intervention and 11 (1.3%) who were intubated for more than 24 hours for TBI. The frequency of different types of neurosurgical interventions is shown in eTable 2 in the Supplement. No patients died of TBI. Half (50.2%) of the population required hospitalization for 2 or more nights due to TBI.
The univariate statistical significance for all variables with P values less than .20 are shown in Table 2. Several clinical variables, including GCS score, presence of a neurological deficit, and severe headache, were significantly associated with the composite outcome. Likewise, multiple imaging variables—including epidural hematoma, depressed skull fracture, and midline shift—were associated with significantly increased risk, whereas subarachnoid hemorrhage was associated with lower risk.
The results of the multivariate analysis are shown in Table 2. The only clinical variable retained in the final model was GCS score (odds ratio [OR], 3.4; 95% CI, 1.5-7.4 for GCS score 13 vs 15). Among the imaging findings, the presence of an epidural hematoma (OR, 3.4; 95% CI, 1.8-6.2), midline shift (OR, 6.8; 95% CI, 3.4-13.8), and depressed skull fracture (OR, 6.5; 95% CI, 3.7-11.4) were all associated with significantly increased risk. The model C statistic was 0.84 (95% CI, 0.79-0.88). The 10-fold cross-validated C statistic was 0.83. Among the subpopulation of children with no significant noncranial injuries, the C statistic was 0.85 (95% CI, 0.80-0.90).
The calibration-in-the-large statistic was −0.08, indicating low systemic overprediction or underprediction. The calibration slope was 0.95, indicating close agreement between predicted and observed event rates. Examination of the calibration plot revealed that the model was well calibrated at low to moderate risk levels, but overpredicted risk for patients with approximately 50% or higher risk for the outcome (eFigure in the Supplement).
Based on the multivariate risk model, we developed the Children’s Intracranial Injury Decision Aid (CHIIDA) score to predict the need for ICU admission (Box). Each variable in the model was assigned a point value ranging from 2 to 7, and each patient’s score could range from 0 to 24. As shown in Figure 1, the predicted risk of the composite outcome ranged from 2.6% for patients with 0 points to a maximum of 92.5% for patients with 24 points.
Depressed skull fracture, 7 points
Midline shift, 7 points
Epidural hematoma, 5 points
Glasgow Coma Scale score of 13, 5 points
Glasgow Coma Scale score of 14, 2 points
Abbreviation: CHIIDA, Children’s Intracranial Injury Decision Aid.
Using a cutoff criterion of more than 0 points to admit to the ICU had a sensitivity of 93.2% (95% CI, 84.7%-97.7%) and a negative predictive value of 98.8% (95% CI, 97.3%-99.6%) (Table 3). Using this cutoff would have avoided ICU admission in 51.3% of patients. Using a cutoff criterion of more than 2 points had a sensitivity of 86.3% (95% CI, 76.3%-93.2%) and a negative predictive value of 98.2% (95% CI, 96.7%-99.1%). Using this cutoff would have avoided ICU admission in 65.4% of patients.
The proportion of patients with high vs low acuity ED dispositions at each level of the CHIIDA score is shown in Figure 2. As shown in Figure 2, for most risk levels, there was significant variability in disposition, with the level of care often not corresponding to patients’ evidence-based predicted risk. The R2 statistic of the point score vs disposition was 0.38, indicating that the CHIIDA score explained less than 40% of the variability in disposition decision.
In this study, we describe the development of the CHIIDA score, a novel clinical decision tool for predicting the post–head CT need for ICU admission among pediatric patients with complicated mTBI. Using data from a multicenter prospective study, we identified GCS score as the sole independent clinical predictor of needing ICU care, emphasizing the importance of accurately assessing this metric. By comparison, the presence of midline shift, epidural hematoma, and depressed skull fracture were all significant radiologic predictors. The final model had high discrimination and calibration and effectively identified low-risk populations that could typically avoid ICU admission. In addition, we demonstrated that current post-CT management practices are highly variable and often not related to patients’ evidence-based risk, further emphasizing the importance of the CHIIDA score.
While there are ongoing large-scale efforts to evaluate best practices among children with severe TBI—including the Approaches and Decisions in Acute TBI (ADAPT) Trial32—the post-CT treatment of patients with mTBI has not received the same attention. The most compelling reason to risk stratify children with complicated mTBI is to ensure that those at high risk for serious events receive appropriate attention and monitoring. The relatively weak association—ie, low R2—between ED disposition and CHIIDA score suggests that factors such as individual gestalt impression and institutional culture may be having a significant influence on disposition decisions. Most importantly, this weak association indicates that many high-risk children may be receiving insufficient attention.
Although appropriate ICU monitoring for high-risk children remains paramount, universal ICU admission, independent of patient risk, is also associated with a variety of costs. While exact figures vary, by some estimates, ICU stays are associated with several thousand dollars per day in increased charges compared with a general ward.33,34 In addition, pediatric ICU beds are a limited resource—approximately 2000 total beds in the United States—and potentially unnecessary admissions may limit access for other children in greater need.11,35 Finally, pediatric ICU admission is often a major emotional burden for families that can potentially be avoided in many circumstances.12,13,36
A large part of the practice variability demonstrated in this study may result from divergent results in the existing literature. For example, previous studies have shown that 0% to 43% of children with complicated mTBI required neurosurgical intervention, likely reflecting the retrospective, single-center nature and variable inclusion criteria across these analyses.14-18,37-39 Likewise, to our knowledge, the only other study that identified independent predictors of outcome among children with complicated mTBI was a single-center analysis with only 29 outcomes,18 emphasizing the value of using multicenter data.
Leveraging the large-scale PECARN data set, we developed the CHIIDA score to aid patient treatment. While external validation is still needed and individual decisions should consider institutional experience, physician judgment, and family comfort, we recommend that nearly all children with 0 points and many children with less than 3 points be admitted to a general ward. Using these cutoffs, the negative predictive value of a major event is 98% to 99%, and 50% to 65% of patients could avoid ICU admission. In contrast, patients at higher risk levels should likely be admitted to an ICU or higher-acuity stepdown unit in most circumstances. By following this evidence-based framework, the CHIIDA score may both reduce resource use and improve patient safety, potentially limiting the practice variations demonstrated in this study. Other possible benefits, such as facilitating shared decision making40,41 and helping guide hospital transfer practices, may be explored in future work.
Despite these advances, this study had several limitations, which serve as directions for future research. First, because primary CT images were not available for review, we were unable to grade imaging findings based on size or severity (eg, large vs small hemorrhage and degree of skull fracture depression). Future analyses should evaluate the extent to which more detailed radiologic parameters improve risk prediction and clinician acceptance. Second, despite the large sample size, there were relatively few patients at the highest risk levels, potentially explaining the lower model calibration for that group. However, given the almost certain need for ICU observation among these patients, precise model calibration is likely less important in this subgroup. Third, some aspects of surgical decision making, such as deciding whether to elevate a depressed skull fracture, involved clinical judgment and may have varied among surgeons, complicating risk predictions.42 Fourth, owing to limitations in the study data set, we were unable to determine surgical indications, such as an expanding hematoma vs clinical decline. Finally, this tool will need to be validated in future studies to verify its prognostic accuracy and evaluate key implementation outcomes, such as physician acceptance.43
Using data from a large, prospectively collected, multicenter data set, we found that lower GCS score, midline shift, depressed skull fracture, and epidural hematoma are independent predictors of needing ICU-level care in children with complicated mTBI. These factors served as the basis for developing the CHIIDA score, a decision tool to aid physicians treating these patients.
Accepted for Publication: November 16, 2016.
Corresponding Author: Jacob K. Greenberg, MD, MSCI, St Louis Children’s Hospital, Washington University School of Medicine, 1 Children’s Pl, Ste 4S20, St Louis, MO 63110 (email@example.com).
Published Online: February 13, 2017. doi:10.1001/jamapediatrics.2016.4520
Author Contributions: Dr Greenberg had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Greenberg, Carpenter, Lumba-Brown, Keller, Pineda, Brownson, Limbrick.
Acquisition, analysis, or interpretation of data: Greenberg, Yan, Carpenter, Pineda, Limbrick.
Drafting of the manuscript: Greenberg, Yan, Carpenter, Lumba-Brown.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Greenberg, Yan.
Obtained funding: Greenberg, Limbrick.
Administrative, technical, or material support: Carpenter, Lumba-Brown, Brownson, Limbrick.
Study supervision: Carpenter, Lumba-Brown, Limbrick.
Conflict of Interest Disclosures: None reported.
Funding/Support: This work was supported by grant FPP-1501 from the Washington University School of Medicine Faculty Practice Plan and the St Louis Children’s Hospital Foundation. The Pediatric Emergency Care Applied Research Network was funded by the Health Resources Services Administration/Maternal Child Health Bureau /Emergency Medical Services for Children.
Role of the Funder/Sponsor: The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: This article was prepared using the “Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study (TBI study)” data set obtained from the University of Utah School of Medicine and does not necessarily reflect the opinions or views of the TBI Trial investigators or the Health Resources Services Administration/Maternal Child Health Bureau/Emergency Medical Services for Children.
Additional Contributions: We thank Margaret Olsen, PhD, MPH (Department of Internal Medicine, Washington University in St Louis, St Louis, Missouri), for her insightful comments and suggestions related to this study. She did not receive compensation.
Create a personal account or sign in to: