Validation of a Machine Learning Brain Electrical Activity–Based Index to Aid in Diagnosing Concussion Among Athletes | Traumatic Brain Injury | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Flowchart of Participant Enrollment and Study Population
Flowchart of Participant Enrollment and Study Population

EEG indicates electroencephalogram; RTP, return to play.

Figure 2.  Longitudinal Change in Concussion Index Among Athletes With or Without Concussion
Longitudinal Change in Concussion Index Among Athletes With or Without Concussion

Concussion index value at day 0, return to play (RTP) and RTP+45 days, in the athletes with or without concussion. The dotted line indicates the threshold for the Concussion Index, where more than 70 is not concussed and 70 or less is concussed. Vertical lines indicate the 95% CI.

aSignificance of the difference on day 0 between the mean Concussion Index among athletes with concussion compared with those without concussion (P < .001), with the Concussion Index significantly lower among the athletes with concussion.

bAmong athletes without concussion, significant noninferiority (equivalence) in the mean Concussion Index between day 0 and RTP+45 days was found, with the Concussion Index of 78% of athletes with concussion exceeding the 90th percentile Concussion Index of athletes without concussion.

Table 1.  Characteristics of Study Participants
Characteristics of Study Participants
Table 2.  Number of Positive (Likely Concussed) and Negative (Likely Not Concussed) Concussion Index Test Results for Participants With or Without Concussion on Day-of-Injury Assessment
Number of Positive (Likely Concussed) and Negative (Likely Not Concussed) Concussion Index Test Results for Participants With or Without Concussion on Day-of-Injury Assessment
Table 3.  EEG and Neurocognitive Features With the Highest Contribution to the Concussion Index
EEG and Neurocognitive Features With the Highest Contribution to the Concussion Index
1.
Elbin  RJ, Sufrinko  A, Schatz  P,  et al.  Removal from play after concussion and recovery time.   Pediatrics. 2016;138(3):e20160910. doi:10.1542/peds.2016-0910PubMedGoogle Scholar
2.
Asken  BM, Bauer  RM, Guskiewicz  KM,  et al; CARE Consortium Investigators.  Immediate removal from activity after sport-related concussion is associated with shorter clinical recovery and less severe symptoms in collegiate student-athletes.   Am J Sports Med. 2018;46(6):1465-1474. doi:10.1177/0363546518757984 PubMedGoogle ScholarCrossref
3.
Asken  BM, McCrea  MA, Clugston  JR, Snyder  AR, Houck  ZM, Bauer  RM.  “Playing through it”: delayed reporting and removal from athletic activity after concussion predicts prolonged recovery.   J Athl Train. 2016;51(4):329-335. doi:10.4085/1062-6050-51.5.02 PubMedGoogle ScholarCrossref
4.
Anderson  V, Godfrey  C, Rosenfeld  JV, Catroppa  C.  10 Years outcome from childhood traumatic brain injury.   Int J Dev Neurosci. 2012;30(3):217-224. doi:10.1016/j.ijdevneu.2011.09.008 PubMedGoogle ScholarCrossref
5.
Scholten  AC, Haagsma  JA, Cnossen  MC, Olff  M, van Beeck  EF, Polinder  S.  Prevalence of and risk factors for anxiety and depressive disorders after traumatic brain injury: a systematic review.   J Neurotrauma. 2016;33(22):1969-1994. doi:10.1089/neu.2015.4252 PubMedGoogle ScholarCrossref
6.
McMahon  P, Hricik  A, Yue  JK,  et al; TRACK-TBI Investigators.  Symptomatology and functional outcome in mild traumatic brain injury: results from the prospective TRACK-TBI study.   J Neurotrauma. 2014;31(1):26-33. doi:10.1089/neu.2013.2984 PubMedGoogle ScholarCrossref
7.
Sariaslan  A, Sharp  DJ, D’Onofrio  BM, Larsson  H, Fazel  S.  Long-term outcomes associated with traumatic brain injury in childhood and adolescence: a nationwide Swedish cohort study of a wide range of medical and social outcomes.   PLoS Med. 2016;13(8):e1002103. doi:10.1371/journal.pmed.1002103 PubMedGoogle Scholar
8.
Thatcher  RW, North  DM, Curtin  RT,  et al.  An EEG severity index of traumatic brain injury.   J Neuropsychiatry Clin Neurosci. 2001;13(1):77-87. doi:10.1176/jnp.13.1.77 PubMedGoogle ScholarCrossref
9.
Virji-Babul  N, Hilderman  CG, Makan  N,  et al.  Changes in functional brain networks following sports-related concussion in adolescents.   J Neurotrauma. 2014;31(23):1914-1919. doi:10.1089/neu.2014.3450 PubMedGoogle ScholarCrossref
10.
Dunkley  BT, Da Costa  L, Bethune  A,  et al.  Low-frequency connectivity is associated with mild traumatic brain injury.   Neuroimage Clin. 2015;7:611-621. doi:10.1016/j.nicl.2015.02.020 PubMedGoogle ScholarCrossref
11.
Sponheim  SR, McGuire  KA, Kang  SS,  et al.  Evidence of disrupted functional connectivity in the brain after combat-related blast injury.   Neuroimage. 2011;54(suppl 1):S21-S29. doi:10.1016/j.neuroimage.2010.09.007 PubMedGoogle ScholarCrossref
12.
Slobounov  SM, Zhang  K, Pennell  D, Ray  W, Johnson  B, Sebastianelli  W.  Functional abnormalities in normally appearing athletes following mild traumatic brain injury: a functional MRI study.   Exp Brain Res. 2010;202(2):341-354. doi:10.1007/s00221-009-2141-6 PubMedGoogle ScholarCrossref
13.
Balkan  O, Virji-Babul  N, Miyakoshi  M, Makeig  S, Garudadri  H.  Source-domain spectral EEG analysis of sports-related concussion via measure projection analysis.   Annu Int Conf IEEE Eng Med Biol Soc. 2015;2015:4053-4056. doi:10.1109/EMBC.2015.7319284 PubMedGoogle Scholar
14.
Korn  A, Golan  H, Melamed  I, Pascual-Marqui  R, Friedman  A.  Focal cortical dysfunction and blood-brain barrier disruption in patients with postconcussion syndrome.   J Clin Neurophysiol. 2005;22(1):1-9. doi:10.1097/01.WNP.0000150973.24324.A7 PubMedGoogle ScholarCrossref
15.
Conley  AC, Cooper  PS, Karayanidis  F,  et al.  Resting state electroencephalography and sports-related concussion: a systematic review.   J Neurotrauma. 2018;36(1):1-13. doi:10.1089/neu.2018.5761 PubMedGoogle ScholarCrossref
16.
Duda  RO, Hart  PE, Stork  DG.  Pattern Classification. 2nd ed. Wiley-Interscience; 2001.
17.
Hanley  D, Prichep  LS, Bazarian  J,  et al.  Emergency department triage of traumatic head injury using a brain electrical activity biomarker: a multisite prospective observational validation trial.   Acad Emerg Med. 2017;24(5):617-627. doi:10.1111/acem.13175 PubMedGoogle ScholarCrossref
18.
Hanley  D, Prichep  LS, Badjatia  N,  et al.  A brain electrical activity electroencephalographic-based biomarker of functional impairment in traumatic brain injury: a multi-site validation trial.   J Neurotrauma. 2018;35(1):41-47. doi:10.1089/neu.2017.5004 PubMedGoogle ScholarCrossref
19.
Jacquin  A, Kanakia  S, Oberly  D, Prichep  LS.  A multimodal biomarker for concussion identification, prognosis and management.   Comput Biol Med. 2018;102:95-103. doi:10.1016/j.compbiomed.2018.09.011 PubMedGoogle ScholarCrossref
20.
Mollayeva  T, El-Khechen-Richandi  G, Colantonio  A.  Sex & gender considerations in concussion research.   Concussion. 2018;3(1):CNC51. doi:10.2217/cnc-2017-0015 PubMedGoogle ScholarCrossref
21.
National Collegiate Athletic Association. Concussion safety best practices for campuses. Accessed October 8, 2020. http://www.ncaa.org/sport-science-institute/concussion-safety-best-practices-campuses
22.
McCrory  P, Meeuwisse  WH, Aubry  M,  et al.  Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012.   Br J Sports Med. 2013;47(5):250-258. doi:10.1136/bjsports-2013-092313 PubMedGoogle ScholarCrossref
23.
McCrory  P, Meeuwisse  W, Dvořák  J,  et al.  Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016.   Br J Sports Med. 2017;51(11):838-847.PubMedGoogle Scholar
24.
Lau  BC, Collins  MW, Lovell  MR.  Sensitivity and specificity of subacute computerized neurocognitive testing and symptom evaluation in predicting outcomes after sports-related concussion.   Am J Sports Med. 2011;39(6):1209-1216. doi:10.1177/0363546510392016 PubMedGoogle ScholarCrossref
25.
McCrory  P, Meeuwisse  WH, Aubry  M,  et al.  Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport, Zurich, November 2012.   J Athl Train. 2013;48(4):554-575. doi:10.4085/1062-6050-48.4.05PubMedGoogle ScholarCrossref
26.
Sport concussion assessment tool–5th edition.  Br J Sports Med. 2017;51(11):851-858. doi:10.1136/bjsports-2017-097506SCAT5PubMed
27.
Randolph  C, Millis  S, Barr  WB,  et al.  Concussion symptom inventory: an empirically derived scale for monitoring resolution of symptoms following sport-related concussion.   Arch Clin Neuropsychol. 2009;24(3):219-229. doi:10.1093/arclin/acp025 PubMedGoogle ScholarCrossref
28.
McCrea  M, Kelly  JP, Randolph  C,  et al.  Standardized assessment of concussion (SAC): on-site mental status evaluation of the athlete.   J Head Trauma Rehabil. 1998;13(2):27-35. doi:10.1097/00001199-199804000-00005 PubMedGoogle ScholarCrossref
29.
Cernich  A, Reeves  D, Sun  W, Bleiberg  J.  Automated Neuropsychological Assessment Metrics sports medicine battery.   Arch Clin Neuropsychol. 2007;22(1)(suppl 1):S101-S114. doi:10.1016/j.acn.2006.10.008 PubMedGoogle ScholarCrossref
30.
Nelson  LD, LaRoche  AA, Pfaller  AY,  et al.  Prospective, head-to-head study of three computerized neurocognitive assessment tools (CNTs): reliability and validity for the assessment of sport-related concussion.   J Int Neuropsychol Soc. 2016;22(1):24-37. doi:10.1017/S1355617715001101 PubMedGoogle ScholarCrossref
31.
Prichep  LS, Jacquin  A, Filipenko  J,  et al.  Classification of traumatic brain injury severity using informed data reduction in a series of binary classifier algorithms.   IEEE Trans Neural Syst Rehabil Eng. 2012;20(6):806-822. doi:10.1109/TNSRE.2012.2206609 PubMedGoogle ScholarCrossref
32.
Raymer  ML, Punch  ED, Goodman  ED, Kuhn  LA, Jain  AK.  Dimensionality reduction using genetic algorithms.   IEEE Trans Evol Comput. 2000;4(2):164–171. doi:10.1109/4235.850656 Google ScholarCrossref
33.
Oh  IS, Lee  JS, Moon  BR.  Hybrid genetic algorithms for feature selection.   IEEE Trans Pattern Anal Mach Intell. 2004;26(11):1424-1437. doi:10.1109/TPAMI.2004.105 PubMedGoogle ScholarCrossref
34.
McCrea  M, Prichep  L, Powell  MR, Chabot  R, Barr  WB.  Acute effects and recovery after sport-related concussion: a neurocognitive and quantitative brain electrical activity study.   J Head Trauma Rehabil. 2010;25(4):283-292. doi:10.1097/HTR.0b013e3181e67923 PubMedGoogle ScholarCrossref
35.
Prichep  LS, McCrea  M, Barr  W, Powell  M, Chabot  RJ.  Time course of clinical and electrophysiological recovery after sport-related concussion.   J Head Trauma Rehabil. 2013;28(4):266-273. doi:10.1097/HTR.0b013e318247b54e PubMedGoogle ScholarCrossref
36.
Teel  EF, Ray  WJ, Geronimo  AM, Slobounov  SM.  Residual alterations of brain electrical activity in clinically asymptomatic concussed individuals: an EEG study.   Clin Neurophysiol. 2014;125(4):703-707. doi:10.1016/j.clinph.2013.08.027 PubMedGoogle ScholarCrossref
37.
Johnson  EW, Kegel  NE, Collins  MW.  Neuropsychological assessment of sport-related concussion.   Clin Sports Med. 2011;30(1):73-88, viii-ix. doi:10.1016/j.csm.2010.08.007 PubMedGoogle ScholarCrossref
38.
Shenton  ME, Hamoda  HM, Schneiderman  JS,  et al.  A review of magnetic resonance imaging and diffusion tensor imaging findings in mild traumatic brain injury.   Brain Imaging Behav. 2012;6(2):137-192. doi:10.1007/s11682-012-9156-5 PubMedGoogle ScholarCrossref
39.
Chamard  E, Lichtenstein  JD.  A systematic review of neuroimaging findings in children and adolescents with sports-related concussion.   Brain Inj. 2018;32(7):816-831. doi:10.1080/02699052.2018.1463106 PubMedGoogle ScholarCrossref
40.
Davenport  EM, Whitlow  CT, Urban  JE,  et al.  Abnormal white matter integrity related to head impact exposure in a season of high school varsity football.   J Neurotrauma. 2014;31(19):1617-1624. doi:10.1089/neu.2013.3233 PubMedGoogle ScholarCrossref
41.
McAllister  TW, Ford  JC, Flashman  LA,  et al.  Effect of head impacts on diffusivity measures in a cohort of collegiate contact sport athletes.   Neurology. 2014;82(1):63-69. doi:10.1212/01.wnl.0000438220.16190.42 PubMedGoogle ScholarCrossref
42.
Wilde  EA, Goodrich-Hunsaker  NJ, Ware  AL,  et al.  Diffusion tensor imaging indicators of white matter injury are correlated with a multimodal electroencephalography-based biomarker in slow recovering, concussed collegiate athletes.   J Neurotrauma. 2020;37(19):2093-2101. doi:10.1089/neu.2018.6365 PubMedGoogle ScholarCrossref
43.
Iverson  GL, Schatz  P.  Advanced topics in neuropsychological assessment following sport-related concussion.   Brain Inj. 2015;29(2):263-275. doi:10.3109/02699052.2014.965214 PubMedGoogle ScholarCrossref
44.
Broglio  SP, Puetz  TW.  The effect of sport concussion on neurocognitive function, self-report symptoms and postural control: a meta-analysis.   Sports Med. 2008;38(1):53-67. doi:10.2165/00007256-200838010-00005 PubMedGoogle ScholarCrossref
45.
Resch  JE, Schneider  MW, Munro Cullum  C.  The test-retest reliability of three computerized neurocognitive tests used in the assessment of sport concussion.   Int J Psychophysiol. 2018;132(pt A):31-38. doi:10.1016/j.ijpsycho.2017.09.011 PubMedGoogle ScholarCrossref
46.
Echemendia  RJ, Bruce  JM, Bailey  CM, Sanders  JF, Arnett  P, Vargas  G.  The utility of post-concussion neuropsychological data in identifying cognitive change following sports-related MTBI in the absence of baseline data.   Clin Neuropsychol. 2012;26(7):1077-1091. doi:10.1080/13854046.2012.721006 PubMedGoogle ScholarCrossref
47.
Valovich McLeod  TC, Hale  TD.  Vestibular and balance issues following sport-related concussion.   Brain Inj. 2015;29(2):175-184. doi:10.3109/02699052.2014.965206 PubMedGoogle ScholarCrossref
48.
Corwin  DJ, Wiebe  DJ, Zonfrillo  MR,  et al.  Vestibular deficits following youth concussion.   J Pediatr. 2015;166(5):1221-1225. doi:10.1016/j.jpeds.2015.01.039 PubMedGoogle ScholarCrossref
49.
Murray  NG, Ambati  VN, Contreras  MM, Salvatore  AP, Reed-Jones  RJ.  Assessment of oculomotor control and balance post-concussion: a preliminary study for a novel approach to concussion management.   Brain Inj. 2014;28(4):496-503. doi:10.3109/02699052.2014.887144 PubMedGoogle ScholarCrossref
50.
Zhou  G, Brodsky  JR.  Objective vestibular testing of children with dizziness and balance complaints following sports-related concussions.   Otolaryngol Head Neck Surg. 2015;152(6):1133-1139. doi:10.1177/0194599815576720 PubMedGoogle ScholarCrossref
51.
Rawlins  MLW, Suggs  DW, Bierema  L, Miller  LS, Reifsteck  F, Schmidt  JD.  Examination of collegiate student-athlete concussion reporting intentions and behavior.   J Clin Transl Res. 2020;5(4):186-196.PubMedGoogle Scholar
52.
Arrieux  JP, Cole  WR, Ahrens  AP.  A review of the validity of computerized neurocognitive assessment tools in mild traumatic brain injury assessment.   Concussion. 2017;2(1):CNC31. doi:10.2217/cnc-2016-0021 PubMedGoogle ScholarCrossref
53.
Llewellyn  T, Burdette  GT, Joyner  AB, Buckley  TA.  Concussion reporting rates at the conclusion of an intercollegiate athletic career.   Clin J Sport Med. 2014;24(1):76-79. doi:10.1097/01.jsm.0000432853.77520.3d PubMedGoogle ScholarCrossref
54.
O’Brien  AM, Casey  JE, Salmon  RM.  Short-term test-retest reliability of the ImPACT in healthy young athletes.   Appl Neuropsychol Child. 2018;7(3):208-216. doi:10.1080/21622965.2017.1290529 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Physical Medicine and Rehabilitation
    February 15, 2021

    Validation of a Machine Learning Brain Electrical Activity–Based Index to Aid in Diagnosing Concussion Among Athletes

    Author Affiliations
    • 1Department of Emergency Medicine, University of Rochester School of Medicine, Rochester, New York
    • 2Office for Sports Concussion Research, University of Arkansas, Fayetteville
    • 3Korey Stringer Institute, University of Connecticut, Storrs
    • 4UHealth Concussion Program, University of Miami, Miami, Florida
    • 5Department of Physical Therapy Education, SUNY Upstate Medical University, Syracuse, New York
    • 6Morsani College of Medicine, Orthopedics and Sports Medicine, University of South Florida, Tampa
    • 7Department of Psychology, University of Texas at Austin, Austin
    • 8Arnold School of Public Health, University of South Carolina, Columbia
    • 9Department of Kinesiology, Michigan State University, East Lansing
    JAMA Netw Open. 2021;4(2):e2037349. doi:10.1001/jamanetworkopen.2020.37349
    Key Points

    Question  Can the previously derived, machine learning, multimodal, brain electrical activity–based Concussion Index be prospectively validated in an independent population?

    Findings  In this diagnostic study of a cohort of 207 athletes with concussion and 373 matched athletes without concussion, the Concussion Index showed high accuracy in assessing the likelihood of concussion at the time of injury and was shown to return to within the limits of the control athletes in concussed athletes cleared to return to play.

    Meaning  This study suggests that, as an objective, reliable indicator of the presence of concussive brain injury and readiness for return to activity, the Concussion Index has potential to aid in clinical diagnosis and reduce long-term concussion-related disability.

    Abstract

    Importance  An objective, reliable indicator of the presence and severity of concussive brain injury and of the readiness for the return to activity has the potential to reduce concussion-related disability.

    Objective  To validate the classification accuracy of a previously derived, machine learning, multimodal, brain electrical activity–based Concussion Index in an independent cohort of athletes with concussion.

    Design, Setting, and Participants  This prospective diagnostic cohort study was conducted at 10 clinical sites (ie, US universities and high schools) between February 4, 2017, and March 20, 2019. A cohort comprising a consecutive sample of 207 athletes aged 13 to 25 years with concussion and 373 matched athlete controls without concussion were assessed with electroencephalography, cognitive testing, and symptom inventories within 72 hours of injury, at return to play, and 45 days after return to play. Variables from the multimodal assessment were used to generate a Concussion Index at each time point. Athletes with concussion had experienced a witnessed head impact, were removed from play for 5 days or more, and had an initial Glasgow Coma Scale score of 13 to 15. Participants were excluded for known neurologic disease or history within the last year of traumatic brain injury. Athlete controls were matched to athletes with concussion for age, sex, and type of sport played.

    Main Outcomes and Measures  Classification accuracy of the Concussion Index at time of injury using a prespecified cutoff of 70 or less (total range, 0-100, where ≤70 indicates it is likely the individual has a concussion and >70 indicates it is likely the individual does not have a concussion).

    Results  Of 580 eligible participants with analyzable data, 207 had concussion (124 male participants [59.9%]; mean [SD] age, 19.4 [2.5] years), and 373 were athlete controls (187 male participants [50.1%]; mean [SD] age, 19.6 [2.2] years). The Concussion Index had a sensitivity of 86.0% (95% CI, 80.5%-90.4%), specificity of 70.8% (95% CI, 65.9%-75.4%), negative predictive value of 90.1% (95% CI, 86.1%-93.3%), positive predictive value of 62.0% (95% CI, 56.1%-67.7%), and area under receiver operator characteristic curve of 0.89. At day 0, the mean (SD) Concussion Index among athletes with concussion was significantly lower than among athletes without concussion (75.0 [14.0] vs 32.7 [27.2]; P < .001). Among athletes with concussion, there was a significant increase in the Concussion Index between day 0 and return to play, with a mean (SD) paired difference between these time points of −41.2 (27.0) (P < .001).

    Conclusions and Relevance  These results suggest that the multimodal brain activity–based Concussion Index has high classification accuracy for identification of the likelihood of concussion at time of injury and may be associated with the return to control values at the time of recovery. The Concussion Index has the potential to aid in the clinical diagnosis of concussion and in the assessment of athletes’ readiness to return to play.

    Introduction

    There is no objective standard for the diagnosis of mild traumatic brain injury (concussion), which remains a diagnosis based largely on the patient’s subjective report of signs and symptoms. Accurate objective identification of the presence and severity of concussion and the assessment of the readiness to return to activity present significant clinical challenges to health care professionals. Children, adolescents, and young adults are particularly at risk because significant brain development continues throughout these years. The lack of, or delay in, concussion diagnosis has been shown to be associated with much slower recovery,1-3 may be associated with academic or cognitive and emotional functioning,4-6 and has been associated with impaired adult functioning in those sustaining concussive injury before the age of 25 years.7

    An extensive literature demonstrates that changes in brain electrical activity seen on an electroencephalogram (EEG) occur in individuals with concussion, reflected in measures of connectivity (disruption in associations between brain regions),8-10 changes in complexity of the signal (disorganization of neural networks),11,12 and changes in the frequency spectra (associated with changes in oxygen use, glucose metabolism, and neurochemistry).13-15 Quantitative features of EEG (qEEG) can be used to derive a classifier index using supervised machine learning methods.16 Using such methods, researchers have reported high accuracy in the objective identification of traumatic structural brain injury (hemorrhage of ≥1 mL)17 and brain function impairment (concussion)18,19 at the time of injury. The need for a concussion assessment that can be used at any point during the health care continuum was addressed with the derivation of a multimodal, objective Concussion Index.

    The Concussion Index is multimodal and includes neurocognitive performance and vestibular symptoms with qEEG data to enhance classification accuracy. The Concussion Index derivation study revealed that, at a threshold of 70 or less (total range, 0-100, where ≤70 indicates it is likely the individual has a concussion and >70 indicates it is likely the individual does not have a concussion), the Concussion Index accurately discriminated between athlete controls without concussion and athletes with concussion with a sensitivity of 84.9%, specificity of 76.0%, and area under the curve of 0.89.19

    The objectives of the present study were to (1) validate the performance of the previously derived Concussion Index in an independent prospective cohort for classification accuracy and prediction of concussion at the time of injury and (2) demonstrate that, over time, the Concussion Index is stable in athlete controls and improves (ie, in the direction of control values) among athletes with concussion.

    Methods
    Participants and Setting

    We performed a prospective diagnostic study from February 4, 2017, to March 20, 2019, at 10 clinical sites across the US. The study included a consecutive sample of high school and collegiate athletes with concussion and 2 control groups: (1) athletes without concussion matched for age and sex (which have been reported as factors associated with concussion recovery)20 and (2) preseason athletes aged 13 to 25 years without concussion. Both control groups were from the same “intended use” population (athletes at risk for head injury) to minimize differences between groups not associated with head injury. Athletes with concussion were assessed with a handheld EEG device within 72 hours of injury (day 0), at return to play (RTP), and 45 days after RTP (RTP+45). Readiness for RTP was clinically assessed by site standard practice. Matched control participants were assessed at the same time intervals. Inclusion of preseason control participants allowed for an estimate of “baseline” Concussion Index. Investigators were blinded to the EEG output. Only the independent biostatistician was unblinded to the results, enabling him to perform the statistical analysis. The study was approved by the institutional review boards of the primary sites (University of Rochester School of Medicine, State University of New York Upstate Medical University, Washington University in St. Louis, University of Connecticut, University of Arkansas, University of South Carolina, University of Texas at Austin, University of South Florida, University of Miami, and Michigan State University). All participants provided written informed consent, and for minors, parental written informed consent and adolescent assent were also obtained. The study was registered on ClinicalTrials.gov (NCT02957461 and NCT03671083) and followed the Standards for Reporting of Diagnostic Accuracy (STARD) reporting guideline.

    Inclusion Criteria

    Athletes with concussion consisted of male and female individuals between the ages of 13 and 25 years who met the study definition of concussion and had a Glasgow Coma Scale score of 13 or more (total range, 3-15, where 3 indicates severe injury and 15 indicates no or minor injury) at the time of injury and no hospital admission owing to either head injury or collateral injuries for more than 24 hours. Control participants had a Glasgow Coma Scale score of 15 at time of assessment.

    Exclusion Criteria

    Exclusion criteria included forehead, scalp, or skull abnormalities or a clinical condition that would not allow electrode placement; current psychoactive prescription medications taken daily (with the exception of medications being taken for attention-deficit/hyperactivity disorder); history of brain surgery or neurologic disease; pregnancy; acute intoxication; active fever, defined as greater than 37.8 °C; and inability to speak or read English. Athletes with a concussion were excluded if they had a loss of consciousness of 20 minutes or more related to the concussion injury or showed evidence of abnormality visible on a computed tomography scan of the head related to the traumatic event. Control participants were excluded if they showed focal neurologic signs, including aphasia, apraxia, diplopia, facial droop, and dysarthria or slurred speech, and had a history of traumatic brain injury or concussion or were in a motor vehicle collision requiring an emergency department visit in the past year.

    Study Definition of Concussion

    Athletes with concussion were defined as those who had experienced a witnessed head impact and who, by site guidelines, were removed from play for 5 days or more. The use of site guidelines ensured broader applicability of the results to the general population of interest. Assessment of RTP (reported as the number of days from injury date to “cleared to play” date) was made in accordance with a gradual or graded RTP protocol across multiple days, at the end of which an athlete was cleared to play. For college-based and high school–based sites, this protocol conformed to National Collegiate Athletic Association and policy guidelines.21-23 Once the participant was free of symptoms, these guidelines included (1) light aerobic exercise, (2) sport-specific activity with no head impact, (3) noncontact sport drills and resumption of progressive resistance training, (4) unrestricted training, and (5) return to competition. If at any point participants became symptomatic, they were returned to the previous level.

    Participant Subgroups

    For the purpose of further assessing the performance of the Concussion Index relative to severity and outcome, athletes with concussion were subdivided into groups with (1) RTP between 5 and 13 days (normal or rapid recovery) and (2) RTP of 14 days or more (protracted or prolonged recovery). These time points were consistent with those reported from prior research24 and were the median RTPs for the algorithm development population.19

    Clinical Assessments

    Study participants were evaluated at each assessment time point with 3 sections of the Sports Concussion Assessment Tool–3rd edition25 or 5th edition26: (1) Glasgow Coma Scale score, (2) 22-item Concussion Symptom Inventory (CSI) self-rated on a Likert scale (0-6 per item; total score range, 0-132, where 0 indicates the absence of postconcussive symptoms, and 132 indicates the full range and highest severity of postconcussive symptoms),27 and (3) Standard Assessment of Concussion: a brief neurocognitive screening tool (total score range, 0-30, where 0 indicates normal mental state and 30 indicates deficits in orientation, memory, and/or attention).28 The total score on the CSI (total score range, 0-13) was used as an estimate of symptom burden in this study. History of head injury and concussion(s) was also acquired.

    Neurocognitive Performance Tests

    Two neurocognitive tests in previous concussion research29,30 were performed by all participants on the handheld device under the supervision of trained research assistants. These tests included Simple Reaction Time and Procedural Reaction Time tests. Results from these tests served as candidates for inclusion in the multimodal Concussion Index and for additional characterization of the population.

    EEG Data Acquisition

    Ten minutes of EEG data were collected while the participant was resting with eyes closed. A trained research assistant observed the participants throughout data acquisition for vigilance. The EEG data were recorded using a disposable head set that included the Fp1, Fp2, F7, F8, AFz, A1, and A2 locations of the expanded International 10 to 20 Electrode Placement System, rereferenced to linked ears, and all electrode impedances were below 10 kΩ. Data were acquired at a sampling rate of 1 kHz. Amplifiers had a bandpass filter from 0.3 to 250 Hz (3-dB points) and downsampled to 100 Hz for feature extraction.

    EEG Data Processing and Quantitative EEG Feature Extraction

    The EEG signals were processed using a real-time suite of algorithms for artifact detection,31 which identified for removal physiological and nonphysiological contamination (eg, lateral and horizontal eye movements and muscle activity), ensuring the quality of the EEG data. Only artifact-free data (1-2 minutes) were submitted to all further analyses. The previously specified set of EEG features were then computed and z-transformed relative to age-expected normal values and used as inputs to the Concussion Index algorithm.31

    The Concussion Index was previously derived using a machine learning method known as the genetic algorithm. The genetic algorithm method performs a stochastic search involving a series of candidate solutions in which each is informed by its predecessors, similar to an evolutionary algorithm.32,33 In the previous derivation study, a discriminant algorithm consisting of a weighted combination of selected linear and nonlinear EEG features and selected clinical features was identified using this genetic algorithm method to optimally distinguish between participants with and participants without concussion. Brain electrical activity features were the highest contributors to the classifier, especially “connectivity” measures (eg, phase synchrony) that reflect the transmission of information between brain regions. The cutoff for assessment of the likelihood of concussion was obtained from the Concussion Index derivation data, with the threshold derived from the receiver operating characteristic curve for the final algorithm, and used for prospective validation (a participant with a Concussion Index of ≤70 was considered concussed). Further details of the Concussion Index derivation study have been published.19 The algorithm was then applied to this validation study patient population, and the prospective independent performance was reported in this publication.

    Statistical Analysis

    Data were pooled from all 10 clinical sites. The rationale for pooling was based on 3 critical features: all sites used the same protocol, used the same data gathering mechanism, and were monitored to ensure protocol compliance. All analyses were performed by an independent biostatistician.

    The Concussion Index at day 0 (within 72 hours of injury) was used to assess its classification accuracy, including sensitivity, specificity, negative predictive value, positive predictive value, and area under the receiver operator curve, for distinguishing athletes with concussion from control athletes using the previously derived Concussion Index threshold of 70 or less.19 The significance of the difference in the Concussion Index between athletes with concussion and athlete controls was evaluated using an unpaired 2-way t test for mean values and the Wilcoxon rank sum test for median values. To assess the association with symptom burden, the day 0 Concussion Index among athletes with concussion was correlated with total CSI scores using a regression analysis. The significance of the difference between the Concussion Index at day 0 for the athletes with concussion with RTP less than 14 days and those with RTP of 14 days or more was tested using a 1-sided 2-sample t test. In addition, the mean Concussion Index at day 0 was compared with the mean Concussion Index at RTP to assess the extent to which the Concussion Index improved with clinical recovery, using a paired t test. P values were deemed statistically significant at P < .05.

    To estimate the statistical significance of the change in the Concussion Index over time among athletes with concussion required demonstrating the stability of the Concussion Index over time among controls. The mean Concussion Index among controls was compared at day 0 and RTP+45 days, using a paired t test of noninferiority (equivalence) with a preestablished margin of 4.5 discriminant points (using a predetermined margin based on the 0.3 SD of the distribution of discriminant scores for controls). To evaluate the percentage of athletes with concussion who returned to within the normal or control range, a target “normal” value was assessed based on the 10th percentile of the control group of ranked measurements at RTP, which was computed to be a Concussion Index of 65.1 as the cutoff for return to within normal limits. A sample size estimation based on achieving 80% power at a 1-sided α of .03 (for sensitivity and specificity) required 343 participants.

    Results

    Figure 1 shows the 729 participants eligible for enrollment and the final study population of 608 participants (with assessments at day 0 or preseason). Matched athlete controls and athletes with concussion underwent follow-up EEG and clinical assessments at RTP and RTP+45 time points. An additional 28 participants were excluded owing to poor EEG quality or other missing information required as input to the Concussion Index algorithm, leaving 580 participants available for analysis. Of 1357 total EEG evaluations performed across all participants and all time points, only 39 (2.9%) were not usable. Because no pairwise analyses were performed between athletes with and without concussion, all participants with complete data sets were used in the analyses, resulting in a different number participants in each group. No severe adverse events or adverse events were reported.

    The baseline characteristics of the analysis population at day 0 are presented in Table 1. There were more male participants in the group with concussion than in the control group (124 of 207 [59.9%] vs 187 of 373 [50.1%]; P = .02). All athletes with concussion had a Glasgow Coma Scale score of 15 at the time of injury and were most commonly (41 of 207 [19.8%]) injured playing football. As expected, mean (SD) day 0 CSI scores were higher in the group with concussion than in the control group (27.20 [20.42] vs 2.25 [4.00]), and 57.0% of athletes with concussion (118 of 207) required 14 days or more to RTP.

    Classification Accuracy of Concussion Index

    The performance of the Concussion Index for the discrimination between participants with and participants without concussion (at the predetermined threshold of 70) in this independent population is shown in Table 2. Sensitivity was 86.0% (95% CI, 80.5%-90.4%), specificity was 70.8% (95% CI, 65.9%-75.4%), the negative predictive value was 90.1% (95% CI, 86.1%-93.3%), the positive predictive value was 62.0% (95% CI, 56.1%-67.7%), and the prevalence was 35.7% (95% CI, 31.8%-39.7%). The area under the receiver operating characteristic curve for the Concussion Index in this population was 0.89.

    The z-score values for the EEG features with the highest contribution to the classification (highest weights in the previously established algorithm) and the neurocognitive performance measure are summarized in Table 3 for the validation population. The EEG features were heavily represented by measures of connectivity between regions (92.3% [12 of 13]), which are known to be important in the physiology of concussion. Phase synchrony and coherence are EEG features associated with disruption in neuronal transmission between brain regions.

    At day 0, the Concussion Index among athletes with concussion was significantly lower than among the athletes without concussion (mean [SD], 75.0 [14.0] vs 32.7 [27.2]; P < .001; median [interquartile range], 77.7 [68.3-84.8] vs 26.6 [7.9-55.4]; P < .001). Among all participants, there was a strong correlation between the Concussion Index at day 0 and CSI scores (R2 = 0.64; r = 0.80).

    Change in Concussion Index Between Injury and RTP

    In the group with concussion, there was a significant increase in the Concussion Index between day 0 and RTP, with a mean (SD) paired difference in the Concussion Index of −41.2 (27.0) and a median paired difference in the Concussion Index of –45.2 (interquartile range, –63.5 to –17.6) (P < .001) (Figure 2). The negative difference indicates that the Concussion Index was significantly lower at the time of injury compared with RTP.

    Stability of Concussion Index Over Time Among Athlete Controls and Return to “Normal” Among Athletes With Concussion

    Differences in the Concussion Index between day 0 and RTP+45 for athletes without concussion were close to 0, with a mean (SD) paired difference in the Concussion Index of −3.2 (15.4) and a median paired difference in the Concussion Index of −1.45 (interquartile range, –8.7 to 6.0) (P < .001) (Figure 2). This indicates that the Concussion Index obtained at these different time points is significantly equivalent, demonstrating the stability of the Concussion Index over time among athletes without concussion. At RTP, the Concussion Index of 78.2% (95% CI, 71.8%-83.7%) of the athletes with concussion was at or above the predetermined threshold for control athletes. This implies that a high percentage of athletes with concussion cleared to RTP by standard clinical practice (using the graded RTP protocol) had Concussion Indexes in the range of 90% of control athletes without concussion, consistent with recovery. This is graphically demonstrated in Figure 2, which shows clear overlap in the Concussion Index 95% CI error bars for the athletes with or without concussion.

    Concussion Index Differences at Time of Injury Between Those With Rapid Recovery and Those With Protracted Recovery

    Participants with concussion with prolonged RTP had a significantly lower Concussion Index at day 0 compared with those with quick recovery (mean [SD], 38.5 [28.1] vs 28.4 [25.9]; P = .004; median, 38.5 vs 18.2).

    Discussion

    This validation study confirmed that a multimodal, EEG-based Concussion Index can be used with high accuracy to distinguish between athletes with a concussion and those without on the day of injury, supporting the use of the Concussion Index as an objective indicator of brain function impairment at the time of injury for participants with concussion. Significant differences between the Concussion Index at the time of injury and the Concussion Index at RTP may be associated with changes over time in the population with concussion when they are clinically cleared to RTP. Likewise, the demonstration of stability of the Concussion Index across time among the controls allows for confidence in the interpretation of changes when seen in the population with concussion, suggesting the potential utility for monitoring change throughout the recovery period and as a component of the clinical assessment of readiness to RTP. The significance of the difference between the Concussion Index at the time of injury among athletes with concussion with rapid recovery and the Concussion Index at the time of injury among athletes with concussion with protracted recovery suggests future investigations of algorithms to predict outcome. The results from this study were used in support of the submission to the US Food and Drug Administration (FDA) for the commercialization of this algorithm, which was granted on September 11, 2019.

    In addition, although for most athletes with concussion (78.2%) the Concussion Index at RTP improved to within the Concussion Index range of uninjured controls, these results suggest that some athletes with concussion may have been cleared for RTP before brain function impairment had resolved. Similar findings were reported in the literature attesting to the persistence of brain function abnormalities in individuals with sports-related concussion beyond the point when clinical symptoms have resolved.34-37 Because current RTP graded protocols have a significant dependence on participant-reported symptoms, which may resolve when brain function impairment persists, the potential importance of an additional objective measure as part of the final assessment of RTP is highlighted.

    This study used a previously derived machine learning classification algorithm for the assessment of the likelihood of concussion, the components of which have important implications about the pathophysiology of concussion. The features with the highest contribution to the classification algorithm were the EEG features that characterized deviations from normal connectivity between brain regions, both between and within hemispheres. Furthermore, connectivity as reflected in advanced neuroimaging studies of concussion has led to a consensus that the underlying physiology of concussion is associated, in part, with the disruption of neuronal transmission.38-41 A publication using data from the Concussion Index derivation study reported significant correlations between disruption of white matter tracts evidenced on diffusion tensor imaging and the Concussion Index.42 These results indicate that the Concussion Index is associated with the underlying disruption of neuronal transmission.

    Although centered around the EEG features, the Concussion Index included neurocognitive performance measures and vestibular symptoms. These measures were predicted from the prior research to have multimodal associations because both neurocognitive performance deficits30,43-46 and vestibular symptoms47-50 have been reported to be common sequelae of mild traumatic brain injury. The combination of multiple dimensions characterizing concussion was associated with the high accuracy of the algorithm.

    The association between the Concussion Index and symptom burden at the time of injury was demonstrated in the highly significant regression obtained between the CSI and Concussion Index across the population, with decreasing Concussion Index (more abnormal) significantly correlated with increasing CSI (higher symptom burden). The high correlation suggests that the severity of concussion as assessed by CSI was associated with the Concussion Index. Although the CSI is based on self-report and subject to underreporting and/or overreporting, the Concussion Index is an objective measure less prone to subjective reporting51-53 and poor reliability.54

    Limitations

    This study has some limitations, including the lack of intermediate time points for the assessment of change throughout recovery. Future studies will need to aim to perform more evaluations through the recovery phase to better estimate the recovery rates of individuals and the association with changes in the Concussion Index and to further investigate the predictive accuracy of the Concussion Index at the time of injury. Another limitation is the restricted age range for which this study was performed (13-25 years); although these ages are critical for high school and college student athletes, studies are currently under way to address this age limitation.

    Conclusions

    In this diagnostic study, the objective multimodal Concussion Index with the EEG at its core classified participants with concussion at the time of injury with high accuracy and showed significant improvement in the level of uninjured controls at time of recovery (RTP). The FDA-cleared Concussion Index is easy to use (embedded in the BrainScope handheld device) and lends itself to being incorporated into existing standard assessments of concussion to aid in clinical diagnosis and assessment of readiness to RTP.

    Back to top
    Article Information

    Accepted for Publication: December 27, 2020.

    Published: February 15, 2021. doi:10.1001/jamanetworkopen.2020.37349

    Open Access: This is an open access article distributed under the terms of the CC-BY-NC-ND License. © 2021 Bazarian JJ et al. JAMA Network Open.

    Corresponding Author: Jeffrey J. Bazarian MD, Department of Emergency Medicine, University of Rochester, 601 Elmwood Ave, PO Box 655c, Rochester, NY 14642 (jeff_bazarian@urmc.rochester.edu).

    Author Contributions: Dr Bazarian had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Bazarian, Covassin.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Bazarian.

    Critical revision of the manuscript for important intellectual content: All authors.

    Obtained funding: Lopez.

    Administrative, technical, or material support: Casa, Hotz, Neville, Schnyer, Yeargin.

    Supervision: Elbin, Casa, Neville, Lopez, Schnyer, Yeargin, Covassin.

    Conflict of Interest Disclosures: All authors were principal investigators at one of the clinical data acquisition sites participating in the research, whose institutions received contracts to support participant recruitment, consenting, and data acquisition and have no competing financial interests to disclaim. Drs Bazarian, Casa, Lopez, Schnyer, Yeargin, and Covassin reported receiving grants from BrainScope Co Inc during the conduct of the study. Dr Bazarian reported receiving personal fees from Abbott and Q30 Innovations outside the submitted work. Dr Elbin reported receiving grants and consulting fees from BrainScope Co Inc during the conduct of the study. Dr Neville reported being an equity owner of Quadrant Biosciences Inc outside the submitted work. No other disclosures were reported.

    Funding/Support: The clinical study was funded in part by contract W911QY-14-C-0098 to BrainScope Co Inc from the US Navy (Naval Health Research Center).

    Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The views, opinions and/or findings contained in this report are those of the author(s) and should not be construed as an official Department of the Navy position, policy, or decision unless so designated by other documentation.

    Meeting Presentation: This study was presented at the 15th Annual Conference on Brain Injury of the North American Brain Injury Association; February 26, 2020; New Orleans, Louisiana.

    Additional Contributions: The authors acknowledge the contributions of those who made this research possible, including research staff at all the clinical sites, and the participants who volunteered for the study. Richard P. Chiacchierini, PhD, R.P. Chiacchierini Consulting LLC, served as the independent biostatistician for the US Food and Drug Administration (FDA) validation study, the results of which are reported in this article, and while the analyses were conducted as part of the Statistical Analysis Report to the FDA, he was not an author on this article; he was compensated for his contribution.

    References
    1.
    Elbin  RJ, Sufrinko  A, Schatz  P,  et al.  Removal from play after concussion and recovery time.   Pediatrics. 2016;138(3):e20160910. doi:10.1542/peds.2016-0910PubMedGoogle Scholar
    2.
    Asken  BM, Bauer  RM, Guskiewicz  KM,  et al; CARE Consortium Investigators.  Immediate removal from activity after sport-related concussion is associated with shorter clinical recovery and less severe symptoms in collegiate student-athletes.   Am J Sports Med. 2018;46(6):1465-1474. doi:10.1177/0363546518757984 PubMedGoogle ScholarCrossref
    3.
    Asken  BM, McCrea  MA, Clugston  JR, Snyder  AR, Houck  ZM, Bauer  RM.  “Playing through it”: delayed reporting and removal from athletic activity after concussion predicts prolonged recovery.   J Athl Train. 2016;51(4):329-335. doi:10.4085/1062-6050-51.5.02 PubMedGoogle ScholarCrossref
    4.
    Anderson  V, Godfrey  C, Rosenfeld  JV, Catroppa  C.  10 Years outcome from childhood traumatic brain injury.   Int J Dev Neurosci. 2012;30(3):217-224. doi:10.1016/j.ijdevneu.2011.09.008 PubMedGoogle ScholarCrossref
    5.
    Scholten  AC, Haagsma  JA, Cnossen  MC, Olff  M, van Beeck  EF, Polinder  S.  Prevalence of and risk factors for anxiety and depressive disorders after traumatic brain injury: a systematic review.   J Neurotrauma. 2016;33(22):1969-1994. doi:10.1089/neu.2015.4252 PubMedGoogle ScholarCrossref
    6.
    McMahon  P, Hricik  A, Yue  JK,  et al; TRACK-TBI Investigators.  Symptomatology and functional outcome in mild traumatic brain injury: results from the prospective TRACK-TBI study.   J Neurotrauma. 2014;31(1):26-33. doi:10.1089/neu.2013.2984 PubMedGoogle ScholarCrossref
    7.
    Sariaslan  A, Sharp  DJ, D’Onofrio  BM, Larsson  H, Fazel  S.  Long-term outcomes associated with traumatic brain injury in childhood and adolescence: a nationwide Swedish cohort study of a wide range of medical and social outcomes.   PLoS Med. 2016;13(8):e1002103. doi:10.1371/journal.pmed.1002103 PubMedGoogle Scholar
    8.
    Thatcher  RW, North  DM, Curtin  RT,  et al.  An EEG severity index of traumatic brain injury.   J Neuropsychiatry Clin Neurosci. 2001;13(1):77-87. doi:10.1176/jnp.13.1.77 PubMedGoogle ScholarCrossref
    9.
    Virji-Babul  N, Hilderman  CG, Makan  N,  et al.  Changes in functional brain networks following sports-related concussion in adolescents.   J Neurotrauma. 2014;31(23):1914-1919. doi:10.1089/neu.2014.3450 PubMedGoogle ScholarCrossref
    10.
    Dunkley  BT, Da Costa  L, Bethune  A,  et al.  Low-frequency connectivity is associated with mild traumatic brain injury.   Neuroimage Clin. 2015;7:611-621. doi:10.1016/j.nicl.2015.02.020 PubMedGoogle ScholarCrossref
    11.
    Sponheim  SR, McGuire  KA, Kang  SS,  et al.  Evidence of disrupted functional connectivity in the brain after combat-related blast injury.   Neuroimage. 2011;54(suppl 1):S21-S29. doi:10.1016/j.neuroimage.2010.09.007 PubMedGoogle ScholarCrossref
    12.
    Slobounov  SM, Zhang  K, Pennell  D, Ray  W, Johnson  B, Sebastianelli  W.  Functional abnormalities in normally appearing athletes following mild traumatic brain injury: a functional MRI study.   Exp Brain Res. 2010;202(2):341-354. doi:10.1007/s00221-009-2141-6 PubMedGoogle ScholarCrossref
    13.
    Balkan  O, Virji-Babul  N, Miyakoshi  M, Makeig  S, Garudadri  H.  Source-domain spectral EEG analysis of sports-related concussion via measure projection analysis.   Annu Int Conf IEEE Eng Med Biol Soc. 2015;2015:4053-4056. doi:10.1109/EMBC.2015.7319284 PubMedGoogle Scholar
    14.
    Korn  A, Golan  H, Melamed  I, Pascual-Marqui  R, Friedman  A.  Focal cortical dysfunction and blood-brain barrier disruption in patients with postconcussion syndrome.   J Clin Neurophysiol. 2005;22(1):1-9. doi:10.1097/01.WNP.0000150973.24324.A7 PubMedGoogle ScholarCrossref
    15.
    Conley  AC, Cooper  PS, Karayanidis  F,  et al.  Resting state electroencephalography and sports-related concussion: a systematic review.   J Neurotrauma. 2018;36(1):1-13. doi:10.1089/neu.2018.5761 PubMedGoogle ScholarCrossref
    16.
    Duda  RO, Hart  PE, Stork  DG.  Pattern Classification. 2nd ed. Wiley-Interscience; 2001.
    17.
    Hanley  D, Prichep  LS, Bazarian  J,  et al.  Emergency department triage of traumatic head injury using a brain electrical activity biomarker: a multisite prospective observational validation trial.   Acad Emerg Med. 2017;24(5):617-627. doi:10.1111/acem.13175 PubMedGoogle ScholarCrossref
    18.
    Hanley  D, Prichep  LS, Badjatia  N,  et al.  A brain electrical activity electroencephalographic-based biomarker of functional impairment in traumatic brain injury: a multi-site validation trial.   J Neurotrauma. 2018;35(1):41-47. doi:10.1089/neu.2017.5004 PubMedGoogle ScholarCrossref
    19.
    Jacquin  A, Kanakia  S, Oberly  D, Prichep  LS.  A multimodal biomarker for concussion identification, prognosis and management.   Comput Biol Med. 2018;102:95-103. doi:10.1016/j.compbiomed.2018.09.011 PubMedGoogle ScholarCrossref
    20.
    Mollayeva  T, El-Khechen-Richandi  G, Colantonio  A.  Sex & gender considerations in concussion research.   Concussion. 2018;3(1):CNC51. doi:10.2217/cnc-2017-0015 PubMedGoogle ScholarCrossref
    21.
    National Collegiate Athletic Association. Concussion safety best practices for campuses. Accessed October 8, 2020. http://www.ncaa.org/sport-science-institute/concussion-safety-best-practices-campuses
    22.
    McCrory  P, Meeuwisse  WH, Aubry  M,  et al.  Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012.   Br J Sports Med. 2013;47(5):250-258. doi:10.1136/bjsports-2013-092313 PubMedGoogle ScholarCrossref
    23.
    McCrory  P, Meeuwisse  W, Dvořák  J,  et al.  Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016.   Br J Sports Med. 2017;51(11):838-847.PubMedGoogle Scholar
    24.
    Lau  BC, Collins  MW, Lovell  MR.  Sensitivity and specificity of subacute computerized neurocognitive testing and symptom evaluation in predicting outcomes after sports-related concussion.   Am J Sports Med. 2011;39(6):1209-1216. doi:10.1177/0363546510392016 PubMedGoogle ScholarCrossref
    25.
    McCrory  P, Meeuwisse  WH, Aubry  M,  et al.  Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport, Zurich, November 2012.   J Athl Train. 2013;48(4):554-575. doi:10.4085/1062-6050-48.4.05PubMedGoogle ScholarCrossref
    26.
    Sport concussion assessment tool–5th edition.  Br J Sports Med. 2017;51(11):851-858. doi:10.1136/bjsports-2017-097506SCAT5PubMed
    27.
    Randolph  C, Millis  S, Barr  WB,  et al.  Concussion symptom inventory: an empirically derived scale for monitoring resolution of symptoms following sport-related concussion.   Arch Clin Neuropsychol. 2009;24(3):219-229. doi:10.1093/arclin/acp025 PubMedGoogle ScholarCrossref
    28.
    McCrea  M, Kelly  JP, Randolph  C,  et al.  Standardized assessment of concussion (SAC): on-site mental status evaluation of the athlete.   J Head Trauma Rehabil. 1998;13(2):27-35. doi:10.1097/00001199-199804000-00005 PubMedGoogle ScholarCrossref
    29.
    Cernich  A, Reeves  D, Sun  W, Bleiberg  J.  Automated Neuropsychological Assessment Metrics sports medicine battery.   Arch Clin Neuropsychol. 2007;22(1)(suppl 1):S101-S114. doi:10.1016/j.acn.2006.10.008 PubMedGoogle ScholarCrossref
    30.
    Nelson  LD, LaRoche  AA, Pfaller  AY,  et al.  Prospective, head-to-head study of three computerized neurocognitive assessment tools (CNTs): reliability and validity for the assessment of sport-related concussion.   J Int Neuropsychol Soc. 2016;22(1):24-37. doi:10.1017/S1355617715001101 PubMedGoogle ScholarCrossref
    31.
    Prichep  LS, Jacquin  A, Filipenko  J,  et al.  Classification of traumatic brain injury severity using informed data reduction in a series of binary classifier algorithms.   IEEE Trans Neural Syst Rehabil Eng. 2012;20(6):806-822. doi:10.1109/TNSRE.2012.2206609 PubMedGoogle ScholarCrossref
    32.
    Raymer  ML, Punch  ED, Goodman  ED, Kuhn  LA, Jain  AK.  Dimensionality reduction using genetic algorithms.   IEEE Trans Evol Comput. 2000;4(2):164–171. doi:10.1109/4235.850656 Google ScholarCrossref
    33.
    Oh  IS, Lee  JS, Moon  BR.  Hybrid genetic algorithms for feature selection.   IEEE Trans Pattern Anal Mach Intell. 2004;26(11):1424-1437. doi:10.1109/TPAMI.2004.105 PubMedGoogle ScholarCrossref
    34.
    McCrea  M, Prichep  L, Powell  MR, Chabot  R, Barr  WB.  Acute effects and recovery after sport-related concussion: a neurocognitive and quantitative brain electrical activity study.   J Head Trauma Rehabil. 2010;25(4):283-292. doi:10.1097/HTR.0b013e3181e67923 PubMedGoogle ScholarCrossref
    35.
    Prichep  LS, McCrea  M, Barr  W, Powell  M, Chabot  RJ.  Time course of clinical and electrophysiological recovery after sport-related concussion.   J Head Trauma Rehabil. 2013;28(4):266-273. doi:10.1097/HTR.0b013e318247b54e PubMedGoogle ScholarCrossref
    36.
    Teel  EF, Ray  WJ, Geronimo  AM, Slobounov  SM.  Residual alterations of brain electrical activity in clinically asymptomatic concussed individuals: an EEG study.   Clin Neurophysiol. 2014;125(4):703-707. doi:10.1016/j.clinph.2013.08.027 PubMedGoogle ScholarCrossref
    37.
    Johnson  EW, Kegel  NE, Collins  MW.  Neuropsychological assessment of sport-related concussion.   Clin Sports Med. 2011;30(1):73-88, viii-ix. doi:10.1016/j.csm.2010.08.007 PubMedGoogle ScholarCrossref
    38.
    Shenton  ME, Hamoda  HM, Schneiderman  JS,  et al.  A review of magnetic resonance imaging and diffusion tensor imaging findings in mild traumatic brain injury.   Brain Imaging Behav. 2012;6(2):137-192. doi:10.1007/s11682-012-9156-5 PubMedGoogle ScholarCrossref
    39.
    Chamard  E, Lichtenstein  JD.  A systematic review of neuroimaging findings in children and adolescents with sports-related concussion.   Brain Inj. 2018;32(7):816-831. doi:10.1080/02699052.2018.1463106 PubMedGoogle ScholarCrossref
    40.
    Davenport  EM, Whitlow  CT, Urban  JE,  et al.  Abnormal white matter integrity related to head impact exposure in a season of high school varsity football.   J Neurotrauma. 2014;31(19):1617-1624. doi:10.1089/neu.2013.3233 PubMedGoogle ScholarCrossref
    41.
    McAllister  TW, Ford  JC, Flashman  LA,  et al.  Effect of head impacts on diffusivity measures in a cohort of collegiate contact sport athletes.   Neurology. 2014;82(1):63-69. doi:10.1212/01.wnl.0000438220.16190.42 PubMedGoogle ScholarCrossref
    42.
    Wilde  EA, Goodrich-Hunsaker  NJ, Ware  AL,  et al.  Diffusion tensor imaging indicators of white matter injury are correlated with a multimodal electroencephalography-based biomarker in slow recovering, concussed collegiate athletes.   J Neurotrauma. 2020;37(19):2093-2101. doi:10.1089/neu.2018.6365 PubMedGoogle ScholarCrossref
    43.
    Iverson  GL, Schatz  P.  Advanced topics in neuropsychological assessment following sport-related concussion.   Brain Inj. 2015;29(2):263-275. doi:10.3109/02699052.2014.965214 PubMedGoogle ScholarCrossref
    44.
    Broglio  SP, Puetz  TW.  The effect of sport concussion on neurocognitive function, self-report symptoms and postural control: a meta-analysis.   Sports Med. 2008;38(1):53-67. doi:10.2165/00007256-200838010-00005 PubMedGoogle ScholarCrossref
    45.
    Resch  JE, Schneider  MW, Munro Cullum  C.  The test-retest reliability of three computerized neurocognitive tests used in the assessment of sport concussion.   Int J Psychophysiol. 2018;132(pt A):31-38. doi:10.1016/j.ijpsycho.2017.09.011 PubMedGoogle ScholarCrossref
    46.
    Echemendia  RJ, Bruce  JM, Bailey  CM, Sanders  JF, Arnett  P, Vargas  G.  The utility of post-concussion neuropsychological data in identifying cognitive change following sports-related MTBI in the absence of baseline data.   Clin Neuropsychol. 2012;26(7):1077-1091. doi:10.1080/13854046.2012.721006 PubMedGoogle ScholarCrossref
    47.
    Valovich McLeod  TC, Hale  TD.  Vestibular and balance issues following sport-related concussion.   Brain Inj. 2015;29(2):175-184. doi:10.3109/02699052.2014.965206 PubMedGoogle ScholarCrossref
    48.
    Corwin  DJ, Wiebe  DJ, Zonfrillo  MR,  et al.  Vestibular deficits following youth concussion.   J Pediatr. 2015;166(5):1221-1225. doi:10.1016/j.jpeds.2015.01.039 PubMedGoogle ScholarCrossref
    49.
    Murray  NG, Ambati  VN, Contreras  MM, Salvatore  AP, Reed-Jones  RJ.  Assessment of oculomotor control and balance post-concussion: a preliminary study for a novel approach to concussion management.   Brain Inj. 2014;28(4):496-503. doi:10.3109/02699052.2014.887144 PubMedGoogle ScholarCrossref
    50.
    Zhou  G, Brodsky  JR.  Objective vestibular testing of children with dizziness and balance complaints following sports-related concussions.   Otolaryngol Head Neck Surg. 2015;152(6):1133-1139. doi:10.1177/0194599815576720 PubMedGoogle ScholarCrossref
    51.
    Rawlins  MLW, Suggs  DW, Bierema  L, Miller  LS, Reifsteck  F, Schmidt  JD.  Examination of collegiate student-athlete concussion reporting intentions and behavior.   J Clin Transl Res. 2020;5(4):186-196.PubMedGoogle Scholar
    52.
    Arrieux  JP, Cole  WR, Ahrens  AP.  A review of the validity of computerized neurocognitive assessment tools in mild traumatic brain injury assessment.   Concussion. 2017;2(1):CNC31. doi:10.2217/cnc-2016-0021 PubMedGoogle ScholarCrossref
    53.
    Llewellyn  T, Burdette  GT, Joyner  AB, Buckley  TA.  Concussion reporting rates at the conclusion of an intercollegiate athletic career.   Clin J Sport Med. 2014;24(1):76-79. doi:10.1097/01.jsm.0000432853.77520.3d PubMedGoogle ScholarCrossref
    54.
    O’Brien  AM, Casey  JE, Salmon  RM.  Short-term test-retest reliability of the ImPACT in healthy young athletes.   Appl Neuropsychol Child. 2018;7(3):208-216. doi:10.1080/21622965.2017.1290529 PubMedGoogle ScholarCrossref
    ×