[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Download PDF
Figure 1.
Flow Diagram of Patients
Flow Diagram of Patients

aThe research ethics board for 1 of the 9 sites did not permit the collection of reasons for meeting exclusion criteria due to provincial regulations. Therefore, the total for “other” includes not specified along with missing.

Figure 2.
Receiver Operating Characteristic Curves
Receiver Operating Characteristic Curves

PPCS indicates persistent postconcussive symptoms. The area under the curve was 0.71 (95% CI, 0.69-0.74) for the derivation cohort and 0.68 (95% CI, 0.65-0.72) for the validation cohort.

Table 1.  
Baseline Patient Characteristicsa
Baseline Patient Characteristicsa
Table 2.  
Mechanism and Types of Injuries Sustained and Medications Used to Treat Patients
Mechanism and Types of Injuries Sustained and Medications Used to Treat Patients
Table 3.  
Demographic and Medical History Variables of Patients With Persistent Postconcussive Symptoms (PPCS) at 28 Days in the Derivation Cohort
Demographic and Medical History Variables of Patients With Persistent Postconcussive Symptoms (PPCS) at 28 Days in the Derivation Cohort
Table 4.  
Medical History, Injury, and Assessment Score Variables of Patients With Persistent Postconcussive Symptoms (PPCS) at 28 Days in the Derivation Cohort
Medical History, Injury, and Assessment Score Variables of Patients With Persistent Postconcussive Symptoms (PPCS) at 28 Days in the Derivation Cohort
Table 5.  
Selected Predictor Variables for Multivariable Model of Persistent Postconcussive Symptoms (PPCS) at 28 Days in the Derivation Cohorta
Selected Predictor Variables for Multivariable Model of Persistent Postconcussive Symptoms (PPCS) at 28 Days in the Derivation Cohorta
Table 6.  
Risk Categories for Persistent Postconcussive Symptoms (PPCS) in the Derivation Cohorta
Risk Categories for Persistent Postconcussive Symptoms (PPCS) in the Derivation Cohorta
Table 7.  
Physicians’ Prediction at Time of Emergency Department Visit for Probability of Persistent Postconcussive Symptoms (PPCS) at 28 Days in the Derivation Cohort
Physicians’ Prediction at Time of Emergency Department Visit for Probability of Persistent Postconcussive Symptoms (PPCS) at 28 Days in the Derivation Cohort
1.
National Center for Injury Prevention. Report to Congress on Mild Traumatic Brain Injury in the United States: Steps to Prevent a Serious Public Health Problem. Atlanta, GA: National Center for Injury Prevention; 2003.
2.
Langlois  JA, Rutland-Brown  W, Wald  MM.  The epidemiology and impact of traumatic brain injury. J Head Trauma Rehabil. 2006;21(5):375-378.
PubMedArticle
3.
Bakhos  LL, Lockhart  GR, Myers  R, Linakis  JG.  Emergency department visits for concussion in young child athletes. Pediatrics. 2010;126(3):e550-e556.
PubMedArticle
4.
Gilchrist  J, Thomas  K, Xu  L,  et al.  Nonfatal traumatic brain injuries related to sports and recreation activities among persons aged ≤19 years. MMWR Morb Mortal Wkly Rep. 2011;60(39):1337-1342.
PubMed
5.
Babcock  L, Byczkowski  T, Wade  SL,  et al.  Predicting postconcussion syndrome after mild traumatic brain injury in children and adolescents who present to the emergency department. JAMA Pediatr. 2013;167(2):156-161.
PubMedArticle
6.
Barlow  M, Schlabach  D, Peiffer  J, Cook  C.  Differences in change scores and the predictive validity of three commonly used measures following concussion in the middle school and high school aged population. Int J Sports Phys Ther. 2011;6(3):150-157.
PubMed
7.
Ayr  LK, Yeates  KO, Taylor  HG, Browne  M.  Dimensions of postconcussive symptoms in children with mild traumatic brain injuries. J Int Neuropsychol Soc. 2009;15(1):19-30.
PubMedArticle
8.
Yeates  KO, Kaizar  E, Rusin  J,  et al.  Reliable change in postconcussive symptoms and its functional consequences among children with mild traumatic brain injury. Arch Pediatr Adolesc Med. 2012;166(7):615-622.
PubMedArticle
9.
Zemek  RL, Farion  KJ, Sampson  M, McGahern  C.  Prognosticators of persistent symptoms following pediatric concussion. JAMA Pediatr. 2013;167(3):259-265.
PubMedArticle
10.
Lau  B, Lovell  MR, Collins  MW, Pardini  J.  Neurocognitive and symptom predictors of recovery in high school athletes. Clin J Sport Med. 2009;19(3):216-221.
PubMedArticle
11.
Chrisman  SP, Rivara  FP, Schiff  MA,  et al.  Risk factors for concussive symptoms 1 week or longer in high school athletes. Brain Inj. 2013;27(1):1-9.
PubMedArticle
12.
Meehan  WP  III, Mannix  RC, Stracciolini  A,  et al.  Symptom severity predicts prolonged recovery after sport-related concussion, but age and amnesia do not. J Pediatr. 2013;163(3):721-725.
PubMedArticle
13.
Morgan  CD, Zuckerman  SL, Lee  YM,  et al.  Predictors of postconcussion syndrome after sports-related concussion in young athletes. J Neurosurg Pediatr. 2015;15(6):589-598.
PubMedArticle
14.
Barlow  KM, Crawford  S, Stevenson  A,  et al.  Epidemiology of postconcussion syndrome in pediatric mild traumatic brain injury. Pediatrics. 2010;126(2):e374-e381.
PubMedArticle
15.
Committee on Sports-Related Concussions in Youth; Board on Children, Youth, and Families; Institute of Medicine; National Research Council. In: Graham R, Rivara FP, Ford MA, Spicer CM, eds. Sports-Related Concussions in Youth: Improving the Science, Changing the Culture. Washington, DC: National Academies Press; 2013.
16.
Zemek  R, Osmond  MH, Barrowman  N,  et al.  Predicting and Preventing Postconcussive Problems in Paediatrics (5P) study [published correction appears in BMJ Open. 2013;3(8):e003550corr1]. BMJ Open. 2013;3(8):1-10.
PubMedArticle
17.
Bouwmeester  W, Zuithoff  NPA, Mallett  S,  et al.  Reporting and methods in clinical prediction research. PLoS Med. 2012;9(5):1-12.
PubMedArticle
18.
McCrory  P, Meeuwisse  WH, Aubry  M,  et al.  Consensus statement on concussion in sport. Br J Sports Med. 2013;47(5):250-258.
PubMedArticle
19.
Harris  PA, Taylor  R, Thielke  R,  et al.  Research electronic data capture (REDCap). J Biomed Inform. 2009;42(2):377-381.
PubMedArticle
20.
Gioia  GA, Collins  M, Isquith  PK.  Improving identification and diagnosis of mild traumatic brain injury with evidence. J Head Trauma Rehabil. 2008;23(4):230-242.
PubMedArticle
21.
Gioia  GA, Schneider  JC, Vaughan  CG, Isquith  PK.  Which symptom assessments and approaches are uniquely appropriate for paediatric concussion? Br J Sports Med. 2009;43(suppl 1):i13-i22.
PubMedArticle
22.
Sady  MD, Vaughan  CG, Gioia  GA.  Psychometric characteristics of the Postconcussion Symptom Inventory in children and adolescents. Arch Clin Neuropsychol. 2014;29(4):348-363.
PubMedArticle
23.
 Child SCAT3. Br J Sports Med. 2013;47(5):263.
PubMed
24.
Stiell  IG, Wells  GA.  Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999;33(4):437-447.
PubMedArticle
25.
Marcano Belisario  JS, Jamsek  J, Huckvale  K,  et al.  Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev. 2015;7:MR000042.
PubMed
26.
Steindel  SJ.  International Classification of Diseases, 10th edition, clinical modification and procedure coding system. J Am Med Inform Assoc. 2010;17(3):274-282.
PubMedArticle
27.
Osmond  MH, Klassen  TP, Wells  GA,  et al.  CATCH. CMAJ. 2010;182(4):341-348.
PubMedArticle
28.
Zemek  R, Clarkin  C, Farion  KJ,  et al.  Parental anxiety at initial acute presentation is not associated with prolonged symptoms following pediatric concussion. Acad Emerg Med. 2013;20(10):1041-1049.
PubMedArticle
29.
Zemek  R, Eady  K, Moreau  K,  et al.  Canadian pediatric emergency physician knowledge of concussion diagnosis and initial management. CJEM. 2015;17(2):115-122.
PubMed
30.
Harrell  F  Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York, NY: Springer; 2010.
31.
Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use. Stat Med. 2004;23(10):1631-1660.
PubMedArticle
32.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models. Epidemiology. 2010;21(1):128-138.
PubMedArticle
33.
DeLong  ER, DeLong  DM, Clarke-Pearson  DL.  Comparing the areas under two or more correlated receiver operating characteristic curves. Biometrics. 1988;44(3):837-845.
PubMedArticle
34.
Meehan  WP  III, Mannix  R, Monuteaux  MC,  et al.  Early symptom burden predicts recovery after sport-related concussion. Neurology. 2014;83(24):2204-2210.
PubMedArticle
35.
Scopaz  KA, Hatzenbuehler  JR.  Risk modifiers for concussion and prolonged recovery. Sports Health. 2013;5(6):537-541.
PubMedArticle
36.
Burke  MJ, Fralick  M, Nejatbakhsh  N,  et al.  In search of evidence-based treatment for concussion. Brain Inj. 2015;29(3):300-305.
PubMedArticle
37.
Zemek  R, Eady  K, Moreau  K,  et al.  Knowledge of paediatric concussion among front-line primary care providers. Paediatr Child Health. 2014;19(9):475-480.
PubMed
38.
Gravel  J, D’Angelo  A, Carrière  B,  et al.  Interventions provided in the acute phase for mild traumatic brain injury. Syst Rev. 2013;2:63.
PubMedArticle
39.
Rivara  FP, Graham  R.  Sports-related concussions in youth. JAMA. 2014;311(3):239-240.
PubMedArticle
40.
Zemek  R, Duval  S, Dematteo  C. Guidelines for Diagnosing and Managing Pediatric Concussion. Toronto: Ontario Neurotrauma Foundation; 2014.
41.
Lau  BC, Kontos  AP, Collins  MW,  et al.  Which on-field signs/symptoms predict protracted recovery from sport-related concussion among high school football players? Am J Sports Med. 2011;39(11):2311-2318.
PubMedArticle
42.
Carson  JD, Lawrence  DW, Kraft  SA,  et al.  Premature return to play and return to learn after a sport-related concussion. Can Fam Physician. 2014;60(6):e310, e312-e315.
PubMed
43.
Marshall  SW, Guskiewicz  KM, Shankar  V,  et al.  Epidemiology of sports-related concussion in seven US high school and collegiate sports. Inj Epidemiol. 2015;2(1):13.Article
Original Investigation
March 8, 2016

Clinical Risk Score for Persistent Postconcussion Symptoms Among Children With Acute Concussion in the ED

Author Affiliations
  • 1Department of Pediatrics, Children’s Hospital of Eastern Ontario, University of Ottawa, Ottawa, Canada
  • 2Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada
  • 3Department of Pediatrics, Alberta Children’s Hospital, Alberta Children’s Hospital Research Institute, University of Calgary, Calgary, Canada
  • 4Department of Pediatrics, Hospital Ste Justine, University of Montreal, Montreal, Quebec, Canada
  • 5Department of Pediatrics, Montreal Children’s Hospital, McGill University Health Center, Montreal, Quebec, Canada
  • 6Department of Pediatrics, Children’s Hospital of Western Ontario, Western University, London, Canada
  • 7Department of Pediatrics, Hospital for Sick Children, Toronto, Ontario, Canada
  • 8Department of Pediatrics, Manitoba Children’s Hospital, Winnipeg, Canada
  • 9Department of Pediatrics, Stollery Children’s Hospital, Edmonton, Alberta, Canada
  • 10Department of Pediatrics, IWK Health Centre, Halifax, Nova Scotia, Canada
  • 11Department of Pediatrics, Alberta Children’s Hospital, Calgary, Canada
  • 12Department of Pediatrics and Clinical Neurosciences, Alberta’s Children’s Hospital, University of Calgary, Calgary, Canada
  • 13Sport Injury Prevention Research Centre, Faculty of Kinesiology, University of Calgary, Calgary, Alberta, Canada
  • 14Department of Neuropsychology, Children’s National Health System, George Washington University School of Medicine, Rockville, Maryland
  • 15Sports Concussion Clinic, Boston Children’s Hospital, Boston, Massachusetts
  • 16Ste Justine Research Center, University of Montreal, Montreal, Quebec, Canada
  • 17Department of Kinesiology, University of Ottawa, Ottawa, Ontario, Canada
  • 18Department of Psychology, Children’s Hospital of Eastern Ontario, Ottawa, Canada
  • 19Alberta Children’s Hospital Research Institute, University of Calgary, Calgary, Canada
  • 20Department of Psychology, Alberta Children’s Hospital Research Institute, University of Calgary, Calgary, Canada
  • 21Department of Neurosurgery, Children’s Hospital of Eastern Ontario, Ottawa, Canada
  • 22Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, Ontario, Canada
  • 23Department of Neurology, Stollery Children’s Hospital, Edmonton, Alberta, Canada
  • 24School of Rehabilitation Science, McMaster University, Hamilton, Ontario, Canada
JAMA. 2016;315(10):1014-1025. doi:10.1001/jama.2016.1203
Abstract

Importance  Approximately one-third of children experiencing acute concussion experience ongoing somatic, cognitive, and psychological or behavioral symptoms, referred to as persistent postconcussion symptoms (PPCS). However, validated and pragmatic tools enabling clinicians to identify patients at risk for PPCS do not exist.

Objective  To derive and validate a clinical risk score for PPCS among children presenting to the emergency department.

Design, Setting, and Participants  Prospective, multicenter cohort study (Predicting and Preventing Postconcussive Problems in Pediatrics [5P]) enrolled young patients (aged 5-<18 years) who presented within 48 hours of an acute head injury at 1 of 9 pediatric emergency departments within the Pediatric Emergency Research Canada (PERC) network from August 2013 through September 2014 (derivation cohort) and from October 2014 through June 2015 (validation cohort). Participants completed follow-up 28 days after the injury.

Exposures  All eligible patients had concussions consistent with the Zurich consensus diagnostic criteria.

Main Outcomes and Measures  The primary outcome was PPCS risk score at 28 days, which was defined as 3 or more new or worsening symptoms using the patient-reported Postconcussion Symptom Inventory compared with recalled state of being prior to the injury.

Results  In total, 3063 patients (median age, 12.0 years [interquartile range, 9.2-14.6 years]; 1205 [39.3%] girls) were enrolled (n = 2006 in the derivation cohort; n = 1057 in the validation cohort) and 2584 of whom (n = 1701 [85%] in the derivation cohort; n = 883 [84%] in the validation cohort) completed follow-up at 28 days after the injury. Persistent postconcussion symptoms were present in 801 patients (31.0%) (n = 510 [30.0%] in the derivation cohort and n = 291 [33.0%] in the validation cohort). The 12-point PPCS risk score model for the derivation cohort included the variables of female sex, age of 13 years or older, physician-diagnosed migraine history, prior concussion with symptoms lasting longer than 1 week, headache, sensitivity to noise, fatigue, answering questions slowly, and 4 or more errors on the Balance Error Scoring System tandem stance. The area under the curve was 0.71 (95% CI, 0.69-0.74) for the derivation cohort and 0.68 (95% CI, 0.65-0.72) for the validation cohort.

Conclusions and Relevance  A clinical risk score developed among children presenting to the emergency department with concussion and head injury within the previous 48 hours had modest discrimination to stratify PPCS risk at 28 days. Before this score is adopted in clinical practice, further research is needed for external validation, assessment of accuracy in an office setting, and determination of clinical utility.

Introduction

Concussion is a serious public health epidemic.1,2 Rates have doubled during the last decade3 with an estimated 750 000 pediatric acute concussion visits to emergency departments (EDs) occurring annually in the United States.1,4 Although many children experience symptom resolution within 2 weeks, approximately 33% experience ongoing somatic, cognitive, psychological, behavioral symptoms, or a combination of these symptoms.5,6 Symptoms persisting beyond 28 days are referred to as persistent postconcussion symptoms (PPCS)7 and can have serious adverse effects, resulting in school absenteeism, impaired academic performance, depressed mood, loss of social activities, and lower quality of life.8

Validated and pragmatic tools to identify children at high risk of developing PPCS do not exist.9 Adolescent age, female sex, and physician-diagnosed history of migraine have been associated with PPCS in children5,10; however, prior studies have had significant limitations. Retrospective studies are limited by poor data quality, missing data, minimal use of validated symptom scoring scales, and lack of standardized acute evaluation.5,6,911

Additional limitations include small sample sizes,6,12 recruitment beyond the acute injury period,13,14 and inconsistent definition and measurement of PPCS.9 Studies including elite adolescent athletes and adults dominate the literature, limiting applicability to subsets of children. The Institute of Medicine and the National Research Council emphasized the need for a large, prospective study to quantify PPCS risk in children and youth and to establish “objective, sensitive, and specific metrics and markers of concussion diagnosis, prognosis, and recovery in youth.”15

The Predicting and Preventing Postconcussive Problems in Pediatrics (5P) study was designed to derive and validate a clinical risk score to stratify PPCS risk occurring after acute concussion in children and youth using readily available clinical features.

Methods
Study Design

The 5P was a prospective, multicenter cohort study.16 Participants were recruited from 9 pediatric emergency departments within the Pediatric Emergency Research Canada (PERC) network. Enrollment occurred from August 2013 through September 2014 (derivation cohort) and from October 2014 through June 2015 (validation cohort) (Figure 1). The study complied with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement17 and was approved by the ethics committees of the PERC participating institutions. Written consent and assent was obtained from all participants and their parents or guardians as appropriate. The trial protocol appears in Supplement 1.

Study Population

Quiz Ref ID Eligible patients were aged 5 years through younger than 18 years, presented to a participating ED with a head injury within the preceding 48 hours, and met concussion diagnostic criteria consistent with the fourth Zurich consensus statement.18 Concussion was defined as a complex pathophysiological process caused by a direct blow to the head, face, neck, or elsewhere on the body with an impulsive force transmitted to the head (which may or may not have involved loss of consciousness), resulting in a brain injury with 1 or more symptoms in 1 or more of the following clinical domains: somatic, cognitive, emotional or behavioral, or sleep (eTable 1 in Supplement 2).18 Patients were excluded for (1) a Glasgow Coma Scale score of 13 or less, (2) a structural abnormality on neuroimaging (if performed), (3) a neurosurgical intervention, (4) intubation or intensive care unit admission, (5) multisystem injury requiring hospitalization, (6) procedural sedation, (7) severe preexisting neurological developmental delay resulting in communication difficulties, (8) intoxication, (9) absence of trauma as primary event, (10) previously enrolled in this same study, (11) insurmountable language barrier, or (12) the inability to follow-up by telephone or email.

Study Protocol

Procedures were identical for the derivation and validation phases of the study. Prior to study initiation, participating site ED physicians and research staff were trained on data collection methods using standardized training sessions during site visits by the principal investigator and the national coordinator. Trained research assistants completed standardized assessments of all patients as described in the published protocol.16 Data were collected and managed using research electronic data capture.19

Patients and parents provided information on demographics, history, and injury characteristics using the Acute Concussion Evaluation inventory.20 Patients and parents quantified state of being prior to the injury and current symptoms using the Postconcussion Symptom Inventory (eFigure 1 in Supplement 2).21,22 Cognition, physical examination, and balance were assessed using the third edition of the Child-Sport Concussion Assessment Tool.23 At enrollment, concussion; developmental, neurological, and psychiatric history; therapies received during the ED visit; discharge instructions; and treating physician prognostication of PPCS risk with predicted symptom duration were prospectively collected. Blinded, independent second raters in a convenience subset of 10% of patients duplicated data collection to assess reliability.24

Participants completed electronic follow-up surveys at 7, 14, and 28 days after the injury, including the patient-reported Postconcussion Symptom Inventory16; electronic capture was not expected to affect reporting.25 Patients opting for web-based follow-up received email reminders 24 hours following each survey deadline; research assistants telephoned nonresponders and those opting for telephone follow-up up to 5 times to complete measures orally.

Primary Outcome Measure

Quiz Ref IDThe primary outcome measure, PPCS, was defined in keeping with the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) definition of postconcussion syndrome, which requires persistence beyond 4 weeks of at least 3 symptoms compared with state of being prior to the injury.26 In the study, an individual symptom was defined as a positive difference between the patient-reported current minus the perceived preinjury symptom rating; both were completed 28 days after the injury.26

Secondary Outcome Measure

Physician performance on prediction of PPCS was measured and compared with PPCS risk score performance. A risk assessment tool should outperform clinician accuracy to be relevant.24 Treating physicians completed standardized surveys, which included the following question: “How likely is this patient to develop persistent symptoms beyond 1 month?” (response options: 0%-10%, 11%-20%, 21%-30%, 31%-50%, 51%-70%, 71%-90%, and 91%-100%).

Statistical Analysis

Forty-six variables were selected a priori for assessment based on a national planning meeting, recent systematic reviews, previous studies, and clinical experience.16 Factors occurring after the ED assessment (eg, compliance with recommendations regarding rest or exertion) were omitted because this would reduce the face validity of a predictive score. At a subsequent consensus meeting, the total parent-reported Postconcussion Symptom Inventory score obtained during the ED visit was separated into its 20 individual components and each was analyzed as independent candidate variable scores.

Based on the pilot study, it was estimated that 25% of participants would experience PPCS when applying ICD-10 criteria.9 Including 10 events per each candidate predictor variable,17 345 cases of PPCS would be required after screening for acceptable interrater agreement, assuming a dropout rate of 25% for the a priori selected variables.27 To obtain 345 cases of PPCS, 1380 patients with new concussion had to be enrolled. Factoring a loss to follow-up rate of 23% based on pilot data,28 the final derivation cohort sample size required was 1792 patients. To capture potential seasonal variability in PPCS rates, a 1-year enrollment period was required.

Based on a survey of PERC members,29 90% sensitivity was targeted to predict PPCS. To validate PPCS risk score with clinically acceptable confidence bounds (95% CI, 85%-95%), 200 patients with PPCS were required in a separate validation cohort. Assuming a rate of PPCS of 25%, 800 patients with new concussion had to be enrolled. With a loss to follow-up rate of 15% based on the derivation phase, the required validation sample size was 920 patients.

Descriptive statistics were used to summarize baseline characteristics. The differences between children with and without PPCS were assessed using the χ2 test or the Fisher exact test as appropriate. Emphasizing clinical relevance and face validity, predictors with continuous outcomes were categorized or dichotomized. Interrater agreement was assessed for all candidate variables using the κ statistic; those variables with acceptable reliability (κ ≥0.6) remained eligible for the multivariable analysis.24 Missing data were handled via list-wise deletion.

All reliable variables associated with PPCS (P < .20) were entered into a multivariable model using forward stepwise binary logistic regression analysis (P = .05 included but P = .10 removed). Variables in the regression model were assessed for co-linearity using the variance inflation factor.

The risk score was evaluated as a diagnostic test calculating sensitivity, specificity, and positive and negative likelihood ratios. The final model was validated internally using bootstrap resampling.30 A risk score for the final multivariable model was derived using the model by Sullivan et al,31 in which points were assigned to each predictor variable with point totals corresponding to risk estimate. High- and low-risk cut points for the PPCS risk score were determined by consensus at a team meeting following the derivation phase.

Temporal validation was performed using a separate independent cohort in the same institutions from which the derivation data were collected (ie, no data from the validation cohort were used to derive the risk score, and no data from the derivation cohort were used to validate). Validation performance was evaluated with correlated receiver operating characteristic analysis and test characteristics. Score calibration was assessed using the Hosmer-Lemeshow test and graphically using a calibration plot.32

Physicians’ prediction was analyzed by logistic regression to predict PPCS. The accuracy of the validated risk stratification score was compared with that of physicians’ predictions using the receiver operating characteristic analysis by DeLong et al.33

Statistical analyses were performed using SPSS Statistics versions 21 and 23 (SPSS Inc) and R version 3.0.2 (R Foundation for Statistical Computing). Two-sided P values of less than .05 were considered statistically significant.

Results
Patient Characteristics

There was complete assessment of the primary outcome of PPCS for 1701 of 2006 participants (84.8%) in the derivation cohort and 883 of 1057 participants (83.5%) in the validation cohort (Figure 1). The median age for both cohorts was 12.0 years (interquartile range, 9.2-14.6 years). The baseline patient characteristics appear in Table 1. Details about the injuries sustained and the types of medications used appear in Table 2.

The characteristics of patients with missing primary outcome data appear in eTable 2 in Supplement 2. The type of treatments provided in the ED appear in eTable 3.

Derivation Cohort
Bivariable Analysis

There were 510 participants (30.0%) who met the criteria of having PPCS in the derivation cohort. Forty-seven potential predictor variables were associated with PPCS in the bivariable analysis (Table 3, Table 4, and eTable 4 in Supplement 2). There were 294 patients (15%) who had blinded duplicate assessments (research assistant only: n = 145 [7%]; physician only: n = 92 [5%]; both research assistant and physician: n = 57 [3%]). Excellent overall interrater agreement was demonstrated (median κ = 0.97 [κ interquartile range, 0.75-0.99]).

Multivariable Analysis

The final multivariable model included (1) age, (2) sex, (3) prior concussion with symptom duration of longer than 1 week, (4) physician-diagnosed migraine history, (5) headache, (6) sensitivity to noise, (7) fatigue, (8) answering questions slowly, and (9) abnormal tandem stance (Table 5). The area under the curve (AUC) was 0.71 (95% CI, 0.69 to 0.74; eFigure 2 in Supplement 2). All variables had a variance inflation factor of less than 2.5, indicating a lack of multicollinearity between predictors.

Bootstrapping analysis (ie, resampling the model 1000 times) revealed a mean overoptimism value of 0.01 (95% CI, –0.02 to 0.03) and a corrected AUC of 0.70. In the final derivation model, 94.3% (1604/1701) of the participants with primary outcome data had complete data on all 9 predictor variables included in the multivariable model. The PPCS risk score derived from the multivariable model (score range, 0 to 12) linearly corresponded to risk estimate. Three cutoff points were selected to stratify PPCS risk (low risk: ≤3 points; medium risk: 4-8 points; and high risk: ≥9 points; Table 6).

Validation Cohort

There were 291 patients (33.0%) who met the criteria of having PPCS. The AUC for the model was 0.68 (95% CI, 0.65-0.72). For patients not at low risk (≤3 points), the sensitivity was 93.5% (95% CI, 90.0%-95.8%), specificity was 18.1% (95% CI, 15.2%-21.4%), and the negative likelihood ratio was 0.36 (95% CI, 0.23-0.58); the negative predictive value was 84.9% (95% CI, 77.6%-90.1%) and the positive predictive value was 35.9% (95% CI, 32.6%-39.5%).

For high-risk patients (≥9 points), the specificity was 93.4% (95% CI, 91.1%-95.1%), sensitivity was 20.3% (95% CI, 16.1%-25.3%), and the positive likelihood ratio was 3.00 (95% CI, 2.06-4.37); the negative predictive value was 70.4% (95% CI, 67.1%-73.5%) and the positive predictive value was 59.6% (95% CI, 50.3%-69.3%).

Validation test characteristics for all point values appear in eTable 5 in Supplement 2. The posttest probabilities for the 3 risk strata (low, medium, and high) appear in eTable 6. The Hosmer-Lemeshow test indicated goodness of fit for the model (P = .50). The calibration plot of observed frequency compared with the predicted probability of PPCS showed an intercept of 0.07 and a slope of 0.90, suggesting acceptable calibration (eFigure 3).

The data for physicians’ prediction at time of emergency department visit for probability of PPCS at 28 days appear in Table 7. A model with 9 variables from the risk score as well as physicians’ prediction had an AUC of 0.68 (95% CI, 0.63-0.73), whereas physicians’ prediction alone had an AUC of 0.55 (95% CI, 0.50-0.59; Figure 2). Thus, in the validation cohort, the addition of the derived prediction model to the physicians’ judgment alone resulted in an incremental C statistic improvement of 0.13 (95% CI, 0.07-0.20; P < .001).

Discussion

Quiz Ref IDA PPCS clinical risk score derived in a large, diverse cohort of children presenting to the ED with concussion within 48 hours of head injury was significantly better than physician judgment in predicting future PPCS, although the discrimination of the risk score model was modest (AUC of 0.71). The PPCS risk score incorporates 9 clinical variables containing information from demographics, history, initial symptoms, cognitive complaints, and physical examination. Evaluation in an independent validation cohort demonstrated good test characteristic retention.

Even though prior research found an association between prolonged recovery and total postinjury symptom burden score (22 items using a 7-point scale),34 such a complex scale is a barrier to adoption by acute care clinicians.24 We instead analyzed individual symptoms, resulting in a final model that includes 4 early symptoms and signs in the PPCS risk score. Several final model variables have been associated with PPCS, including headache, answering questions slowly, and sensitivity to noise.9,11,35 Female sex and older age are associated with prolonged recovery in children and adults.9,35

Although the clinical utility of the PPCS risk score will need to be assessed in an externally validated implementation study prior to adoption into routine practice, the risk stratification score has the potential to individualize concussion care through optimal symptom management and appropriate follow-up.9,16 Therefore, future research needs to determine if the moderate test characteristics of the PPCS risk score allow for clinicians to confidently provide reassurance, alter management plans, or both. Future clinical benefits might include identifying high-risk individuals for further screening, prioritization for specialized concussion evaluations, and initiation of emerging treatments to prevent PPCS.36

Strengths of this study include standardized assessment of predictor and outcome variables using validated scales in a cohort with acute concussions (exclusion of presentations >48 hours after injury). Moreover, a large, cross-country, multisite validation cohort confirmed good predictive performance of the risk score model used in the derivation cohort. Inclusion of participants from a wide age range and spectrum of injuries and those with behavioral, learning, and psychological problems enhances generalizability.

There were minor baseline differences between the derivation and validation cohorts (eg, age group, learning disabilities, depression, and type of sports played). These differences increase the generalizability of our model. Therefore, we believe our findings are applicable to the assessment of children seeking acute care following a concussive event.

Quiz Ref IDDespite collective agreement across guidelines that initial management should include physical and cognitive rest followed by graduated return to normal activities, wide practice variation exists.29,37 The lack of evidence for initial management of concussion (including protocols regarding timing of returning to activity) is a crucial issue in the field of pediatric concussion, and results of this study should be applied in urgently needed future comparative clinical trials.38 The PPCS risk score will benefit concussion care research by providing a tool for the targeted selection of patients in greatest need of intervention.9 Selection of pediatric patients at high risk for PPCS may optimize research recruitment by offering more efficient and cost-effective enrollment strategies, or may be used to stratify participants in clinical trials according to PPCS risk.

Because no objective criterion standards for concussion or PPCS diagnoses exist (ie, no readily available biomarkers or imaging modalities),39 the PPCS risk prediction score may be less precise than prediction studies for other diseases. Nonetheless, the outcome measures used in this study generated the best-available evidence through the use of validated tools. In addition, the definitions we used aligned with current concussion guidelines and ICD-10 standards.18,26,40 The PPCS risk model demonstrated only modest ability to discriminate patients who will and will not have PPCS, resulting in erroneous categorization. Test characteristics could be further refined through inclusion of biomarkers, genetic data, or advanced neuroimaging techniques. The pragmatic, generalizable PPCS risk model does not require expensive and painful testing, and could therefore be used to triage initial management.

There are several limitations to this study. Selection bias may limit generalizability. The patients with concussion in our derivation and validation cohorts may have higher PPCS rates and different risk characteristics than those patients with less severe injuries who may have not sought pediatric ED care. Nonetheless, the study included a heterogeneous population recruited through the use of a large number of study sites with great geographical variation.

Even with inclusion of concussions sustained by a variety of mechanisms, some of which may have involved higher forces than those generally seen in sports (eg, motor vehicle collisions), we observed similar rates of PPCS and loss of consciousness as the rates in the sideline assessment and outpatient literature.4143 Because it is possible that the PPCS risk score may not perform as well in different populations, validation should occur in other clinical settings, such as non–tertiary care EDs, primary care, and sideline assessments.

In addition, the sample was limited to participants without observable lesions on imaging; therefore, it may not be representative of a more complicated spectrum of mild traumatic brain injury. However, because the presence of an intracranial lesion on standard imaging no longer meets the current concussion definition,18,40 outcome prediction in this population is beyond the study’s scope. Future research should seek to determine the performance of this PPCS risk assessment tool in a nuanced population of patients with mild traumatic brain injury.

Although other injuries might have contributed to ongoing symptoms, patients with multisystem injuries requiring hospitalization were excluded. An additional limitation is that measures of socioeconomic status or family functioning were not included, which are variables that could have an effect on follow-up care and resources thus affecting symptom burden with alteration of PPCS risk.

Even though the study had missing data, it was limited in scope. Baseline characteristics were similar between those with and without the primary outcome in both the derivation and validation cohorts, and most enrolled participants had no missing data.

Given the wide age range, potential variation exists in respondent type (parent vs patient); however the Postconcussion Symptom Inventory has good parent and self-report correlation.22 In addition, without a control group, we cannot definitively attribute ongoing symptoms to the acute injury. However, literature examining PPCS incidence in patients with head injury compared with controls (eg, orthopedic injury) has yielded similar rates in the head injury group as in our cohort.9

Conclusions

Quiz Ref IDA clinical risk score developed among children presenting to the ED with concussion and head injury within the previous 48 hours had modest discrimination to stratify PPCS risk at 28 days. Before this score is adopted in clinical practice, further research is needed for external validation, assessment of accuracy in an office setting, and determination of clinical utility.

Back to top
Article Information

Corresponding Author: Roger Zemek, MD, Children’s Hospital of Eastern Ontario Research Institute, 401 Smyth Rd, Room R-139, Ottawa, ON K1H 8L1, Canada (rzemek@cheo.on.ca).

Correction: This article was corrected online June 21, 2016, to fix incorrect wording in the Results section of the text and data transcription errors in Tables 3 and 4 that occurred when these Tables were split.

Author Contributions: Drs Zemek and Barrowman had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Zemek, Barrowman, Freedman, Gravel, Gagnon, McGahern, Sangha, Boutis, Craig, Burns, Farion, Mikrogianakis, Barlow, Giola, Meehan, Kamil, Hoshizaki, Anderson, Klassen, Richer, DeMatteo, Osmond.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Zemek, Barrowman, Aglipay, Burns, Farion, Mikrogianakis, Grool, Brooks, Yeates, Keightley, Richer, Osmond.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Zemek, Barrowman, Aglipay, Yeates, Keightley.

Obtained funding: Zemek, Freedman, Gravel, Gagnon, Boutis, Farion, Barlow, Richer, Osmond.

Administrative, technical, or material support: Zemek, Gravel, Gagnon, McGahern, Sangha, Boutis, Craig, Farion, Mikrogianakis, Barlow, Giola, Meehan, Beauchamp, Kamil, Grool, Anderson, Brooks, Vassilyadi.

Study supervision: Zemek, Freedman, Gagnon, McGahern, Sangha, Boutis, Craig, Dubrovsky, Hoshizaki, Anderson, Klassen, Keightley, Osmond.

Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Gioia reported being the author of the Postconcussion Symptom Inventory (PCSI); the PCSI is freely available and he receives no financial benefit for its use. Dr Meehan reported receiving royalties from ABC-Clio publishing for Kids, Sports, and Concussion: A Guide for Coaches and Parents and from Wolters-Kluwer for being an author for UpToDate; being under contract with ABC-Clio publishing and Springer International for future books; and receiving research funding from the National Football League Players Association and the National Hockey League Alumni Association through the Corey C. Griffin Pro-Am Tournament. No other disclosures were reported.

Funding/Support: This study was supported by operating grant 126197 from the Canadian Institutes of Health Research; grant TM1:#127047 from the Canadian Institutes of Health Research–Ontario Neurotrauma Foundation Mild Traumatic Brain Injury Team; and planning grant MRP: #119829 from the Canadian Institutes of Health Research. Dr Freedman is supported by the Alberta Children’s Hospital Foundation Professorship in Child Health and Wellness.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Group Information: Pediatric Emergency Research Canada Concussion (PERC) Team members: Jialing Xie, MD, and Jennifer Chatfield, MN (Alberta Children’s Hospital, Calgary, Canada); Nadia Dow, BA (Stollery Children’s Hospital, Edmonton, Alberta, Canada); Rena Papadimitropoulos, MSc, and Tracey Levesque, RN (Winnipeg Children’s Hospital, Winnipeg, Manitoba, Canada); Cindy Langford, RN (Children’s Hospital of Western Ontario, London, Canada); Tinh Trung Tran, NRP (Hospital for Sick Children, Toronto, Ontario, Canada); Candice McGahern, BA (Children’s Hospital of Eastern Ontario, Ottawa, Canada); Vanessa DiGirolamo, RT, and Joanna Mazza, BSc (Montreal Children’s Hospital, Montreal, Quebec, Canada); Maryse Lagacé, RN, and Ramona Cook, RN (Centre Hospitalier Universitaire Sainte Justine, Montreal, Quebec, Canada); and Eleanor Fitzpatrick, MN, Jessica MacIntyre, MSc, and Jill Moore, MSc (IWK Children’s Hospital, Halifax, Nova Scotia, Canada).

Previous Presentation: The derivation cohort results were presented at the Pediatric Academic Societies Annual Meeting; April 26, 2015; San Diego, California.

Additional Contributions: We thank the parents and children who participated in this study. We appreciate the collaboration of the treating physicians and the research assistants at each site. We are grateful to Pediatric Emergency Research Canada Network for making this study possible.

References
1.
National Center for Injury Prevention. Report to Congress on Mild Traumatic Brain Injury in the United States: Steps to Prevent a Serious Public Health Problem. Atlanta, GA: National Center for Injury Prevention; 2003.
2.
Langlois  JA, Rutland-Brown  W, Wald  MM.  The epidemiology and impact of traumatic brain injury. J Head Trauma Rehabil. 2006;21(5):375-378.
PubMedArticle
3.
Bakhos  LL, Lockhart  GR, Myers  R, Linakis  JG.  Emergency department visits for concussion in young child athletes. Pediatrics. 2010;126(3):e550-e556.
PubMedArticle
4.
Gilchrist  J, Thomas  K, Xu  L,  et al.  Nonfatal traumatic brain injuries related to sports and recreation activities among persons aged ≤19 years. MMWR Morb Mortal Wkly Rep. 2011;60(39):1337-1342.
PubMed
5.
Babcock  L, Byczkowski  T, Wade  SL,  et al.  Predicting postconcussion syndrome after mild traumatic brain injury in children and adolescents who present to the emergency department. JAMA Pediatr. 2013;167(2):156-161.
PubMedArticle
6.
Barlow  M, Schlabach  D, Peiffer  J, Cook  C.  Differences in change scores and the predictive validity of three commonly used measures following concussion in the middle school and high school aged population. Int J Sports Phys Ther. 2011;6(3):150-157.
PubMed
7.
Ayr  LK, Yeates  KO, Taylor  HG, Browne  M.  Dimensions of postconcussive symptoms in children with mild traumatic brain injuries. J Int Neuropsychol Soc. 2009;15(1):19-30.
PubMedArticle
8.
Yeates  KO, Kaizar  E, Rusin  J,  et al.  Reliable change in postconcussive symptoms and its functional consequences among children with mild traumatic brain injury. Arch Pediatr Adolesc Med. 2012;166(7):615-622.
PubMedArticle
9.
Zemek  RL, Farion  KJ, Sampson  M, McGahern  C.  Prognosticators of persistent symptoms following pediatric concussion. JAMA Pediatr. 2013;167(3):259-265.
PubMedArticle
10.
Lau  B, Lovell  MR, Collins  MW, Pardini  J.  Neurocognitive and symptom predictors of recovery in high school athletes. Clin J Sport Med. 2009;19(3):216-221.
PubMedArticle
11.
Chrisman  SP, Rivara  FP, Schiff  MA,  et al.  Risk factors for concussive symptoms 1 week or longer in high school athletes. Brain Inj. 2013;27(1):1-9.
PubMedArticle
12.
Meehan  WP  III, Mannix  RC, Stracciolini  A,  et al.  Symptom severity predicts prolonged recovery after sport-related concussion, but age and amnesia do not. J Pediatr. 2013;163(3):721-725.
PubMedArticle
13.
Morgan  CD, Zuckerman  SL, Lee  YM,  et al.  Predictors of postconcussion syndrome after sports-related concussion in young athletes. J Neurosurg Pediatr. 2015;15(6):589-598.
PubMedArticle
14.
Barlow  KM, Crawford  S, Stevenson  A,  et al.  Epidemiology of postconcussion syndrome in pediatric mild traumatic brain injury. Pediatrics. 2010;126(2):e374-e381.
PubMedArticle
15.
Committee on Sports-Related Concussions in Youth; Board on Children, Youth, and Families; Institute of Medicine; National Research Council. In: Graham R, Rivara FP, Ford MA, Spicer CM, eds. Sports-Related Concussions in Youth: Improving the Science, Changing the Culture. Washington, DC: National Academies Press; 2013.
16.
Zemek  R, Osmond  MH, Barrowman  N,  et al.  Predicting and Preventing Postconcussive Problems in Paediatrics (5P) study [published correction appears in BMJ Open. 2013;3(8):e003550corr1]. BMJ Open. 2013;3(8):1-10.
PubMedArticle
17.
Bouwmeester  W, Zuithoff  NPA, Mallett  S,  et al.  Reporting and methods in clinical prediction research. PLoS Med. 2012;9(5):1-12.
PubMedArticle
18.
McCrory  P, Meeuwisse  WH, Aubry  M,  et al.  Consensus statement on concussion in sport. Br J Sports Med. 2013;47(5):250-258.
PubMedArticle
19.
Harris  PA, Taylor  R, Thielke  R,  et al.  Research electronic data capture (REDCap). J Biomed Inform. 2009;42(2):377-381.
PubMedArticle
20.
Gioia  GA, Collins  M, Isquith  PK.  Improving identification and diagnosis of mild traumatic brain injury with evidence. J Head Trauma Rehabil. 2008;23(4):230-242.
PubMedArticle
21.
Gioia  GA, Schneider  JC, Vaughan  CG, Isquith  PK.  Which symptom assessments and approaches are uniquely appropriate for paediatric concussion? Br J Sports Med. 2009;43(suppl 1):i13-i22.
PubMedArticle
22.
Sady  MD, Vaughan  CG, Gioia  GA.  Psychometric characteristics of the Postconcussion Symptom Inventory in children and adolescents. Arch Clin Neuropsychol. 2014;29(4):348-363.
PubMedArticle
23.
 Child SCAT3. Br J Sports Med. 2013;47(5):263.
PubMed
24.
Stiell  IG, Wells  GA.  Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999;33(4):437-447.
PubMedArticle
25.
Marcano Belisario  JS, Jamsek  J, Huckvale  K,  et al.  Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev. 2015;7:MR000042.
PubMed
26.
Steindel  SJ.  International Classification of Diseases, 10th edition, clinical modification and procedure coding system. J Am Med Inform Assoc. 2010;17(3):274-282.
PubMedArticle
27.
Osmond  MH, Klassen  TP, Wells  GA,  et al.  CATCH. CMAJ. 2010;182(4):341-348.
PubMedArticle
28.
Zemek  R, Clarkin  C, Farion  KJ,  et al.  Parental anxiety at initial acute presentation is not associated with prolonged symptoms following pediatric concussion. Acad Emerg Med. 2013;20(10):1041-1049.
PubMedArticle
29.
Zemek  R, Eady  K, Moreau  K,  et al.  Canadian pediatric emergency physician knowledge of concussion diagnosis and initial management. CJEM. 2015;17(2):115-122.
PubMed
30.
Harrell  F  Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York, NY: Springer; 2010.
31.
Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use. Stat Med. 2004;23(10):1631-1660.
PubMedArticle
32.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models. Epidemiology. 2010;21(1):128-138.
PubMedArticle
33.
DeLong  ER, DeLong  DM, Clarke-Pearson  DL.  Comparing the areas under two or more correlated receiver operating characteristic curves. Biometrics. 1988;44(3):837-845.
PubMedArticle
34.
Meehan  WP  III, Mannix  R, Monuteaux  MC,  et al.  Early symptom burden predicts recovery after sport-related concussion. Neurology. 2014;83(24):2204-2210.
PubMedArticle
35.
Scopaz  KA, Hatzenbuehler  JR.  Risk modifiers for concussion and prolonged recovery. Sports Health. 2013;5(6):537-541.
PubMedArticle
36.
Burke  MJ, Fralick  M, Nejatbakhsh  N,  et al.  In search of evidence-based treatment for concussion. Brain Inj. 2015;29(3):300-305.
PubMedArticle
37.
Zemek  R, Eady  K, Moreau  K,  et al.  Knowledge of paediatric concussion among front-line primary care providers. Paediatr Child Health. 2014;19(9):475-480.
PubMed
38.
Gravel  J, D’Angelo  A, Carrière  B,  et al.  Interventions provided in the acute phase for mild traumatic brain injury. Syst Rev. 2013;2:63.
PubMedArticle
39.
Rivara  FP, Graham  R.  Sports-related concussions in youth. JAMA. 2014;311(3):239-240.
PubMedArticle
40.
Zemek  R, Duval  S, Dematteo  C. Guidelines for Diagnosing and Managing Pediatric Concussion. Toronto: Ontario Neurotrauma Foundation; 2014.
41.
Lau  BC, Kontos  AP, Collins  MW,  et al.  Which on-field signs/symptoms predict protracted recovery from sport-related concussion among high school football players? Am J Sports Med. 2011;39(11):2311-2318.
PubMedArticle
42.
Carson  JD, Lawrence  DW, Kraft  SA,  et al.  Premature return to play and return to learn after a sport-related concussion. Can Fam Physician. 2014;60(6):e310, e312-e315.
PubMed
43.
Marshall  SW, Guskiewicz  KM, Shankar  V,  et al.  Epidemiology of sports-related concussion in seven US high school and collegiate sports. Inj Epidemiol. 2015;2(1):13.Article
×