Validation of a Machine Learning Brain Electrical Activity–Based Index to Aid in Diagnosing Concussion Among Athletes

Key Points

Question  Can the previously derived, machine learning, multimodal, brain electrical activity–based Concussion Index be prospectively validated in an independent population?

Findings  In this diagnostic study of a cohort of 207 athletes with concussion and 373 matched athletes without concussion, the Concussion Index showed high accuracy in assessing the likelihood of concussion at the time of injury and was shown to return to within the limits of the control athletes in concussed athletes cleared to return to play.

Meaning  This study suggests that, as an objective, reliable indicator of the presence of concussive brain injury and readiness for return to activity, the Concussion Index has potential to aid in clinical diagnosis and reduce long-term concussion-related disability.

Importance  An objective, reliable indicator of the presence and severity of concussive brain injury and of the readiness for the return to activity has the potential to reduce concussion-related disability.

Objective  To validate the classification accuracy of a previously derived, machine learning, multimodal, brain electrical activity–based Concussion Index in an independent cohort of athletes with concussion.

Design, Setting, and Participants  This prospective diagnostic cohort study was conducted at 10 clinical sites (ie, US universities and high schools) between February 4, 2017, and March 20, 2019. A cohort comprising a consecutive sample of 207 athletes aged 13 to 25 years with concussion and 373 matched athlete controls without concussion were assessed with electroencephalography, cognitive testing, and symptom inventories within 72 hours of injury, at return to play, and 45 days after return to play. Variables from the multimodal assessment were used to generate a Concussion Index at each time point. Athletes with concussion had experienced a witnessed head impact, were removed from play for 5 days or more, and had an initial Glasgow Coma Scale score of 13 to 15. Participants were excluded for known neurologic disease or history within the last year of traumatic brain injury. Athlete controls were matched to athletes with concussion for age, sex, and type of sport played.

Main Outcomes and Measures  Classification accuracy of the Concussion Index at time of injury using a prespecified cutoff of 70 or less (total range, 0-100, where ≤70 indicates it is likely the individual has a concussion and >70 indicates it is likely the individual does not have a concussion).

Results  Of 580 eligible participants with analyzable data, 207 had concussion (124 male participants [59.9%]; mean [SD] age, 19.4 [2.5] years), and 373 were athlete controls (187 male participants [50.1%]; mean [SD] age, 19.6 [2.2] years). The Concussion Index had a sensitivity of 86.0% (95% CI, 80.5%-90.4%), specificity of 70.8% (95% CI, 65.9%-75.4%), negative predictive value of 90.1% (95% CI, 86.1%-93.3%), positive predictive value of 62.0% (95% CI, 56.1%-67.7%), and area under receiver operator characteristic curve of 0.89. At day 0, the mean (SD) Concussion Index among athletes with concussion was significantly lower than among athletes without concussion (75.0 [14.0] vs 32.7 [27.2]; P < .001). Among athletes with concussion, there was a significant increase in the Concussion Index between day 0 and return to play, with a mean (SD) paired difference between these time points of −41.2 (27.0) (P < .001).

Conclusions and Relevance  These results suggest that the multimodal brain activity–based Concussion Index has high classification accuracy for identification of the likelihood of concussion at time of injury and may be associated with the return to control values at the time of recovery. The Concussion Index has the potential to aid in the clinical diagnosis of concussion and in the assessment of athletes’ readiness to return to play.

There is no objective standard for the diagnosis of mild traumatic brain injury (concussion), which remains a diagnosis based largely on the patient’s subjective report of signs and symptoms. Accurate objective identification of the presence and severity of concussion and the assessment of the readiness to return to activity present significant clinical challenges to health care professionals. Children, adolescents, and young adults are particularly at risk because significant brain development continues throughout these years. The lack of, or delay in, concussion diagnosis has been shown to be associated with much slower recovery,^1-3 may be associated with academic or cognitive and emotional functioning,^4-6 and has been associated with impaired adult functioning in those sustaining concussive injury before the age of 25 years.⁷

An extensive literature demonstrates that changes in brain electrical activity seen on an electroencephalogram (EEG) occur in individuals with concussion, reflected in measures of connectivity (disruption in associations between brain regions),^8-10 changes in complexity of the signal (disorganization of neural networks),^11,12 and changes in the frequency spectra (associated with changes in oxygen use, glucose metabolism, and neurochemistry).^13-15 Quantitative features of EEG (qEEG) can be used to derive a classifier index using supervised machine learning methods.¹⁶ Using such methods, researchers have reported high accuracy in the objective identification of traumatic structural brain injury (hemorrhage of ≥1 mL)¹⁷ and brain function impairment (concussion)^18,19 at the time of injury. The need for a concussion assessment that can be used at any point during the health care continuum was addressed with the derivation of a multimodal, objective Concussion Index.

The Concussion Index is multimodal and includes neurocognitive performance and vestibular symptoms with qEEG data to enhance classification accuracy. The Concussion Index derivation study revealed that, at a threshold of 70 or less (total range, 0-100, where ≤70 indicates it is likely the individual has a concussion and >70 indicates it is likely the individual does not have a concussion), the Concussion Index accurately discriminated between athlete controls without concussion and athletes with concussion with a sensitivity of 84.9%, specificity of 76.0%, and area under the curve of 0.89.¹⁹

The objectives of the present study were to (1) validate the performance of the previously derived Concussion Index in an independent prospective cohort for classification accuracy and prediction of concussion at the time of injury and (2) demonstrate that, over time, the Concussion Index is stable in athlete controls and improves (ie, in the direction of control values) among athletes with concussion.

We performed a prospective diagnostic study from February 4, 2017, to March 20, 2019, at 10 clinical sites across the US. The study included a consecutive sample of high school and collegiate athletes with concussion and 2 control groups: (1) athletes without concussion matched for age and sex (which have been reported as factors associated with concussion recovery)²⁰ and (2) preseason athletes aged 13 to 25 years without concussion. Both control groups were from the same “intended use” population (athletes at risk for head injury) to minimize differences between groups not associated with head injury. Athletes with concussion were assessed with a handheld EEG device within 72 hours of injury (day 0), at return to play (RTP), and 45 days after RTP (RTP+45). Readiness for RTP was clinically assessed by site standard practice. Matched control participants were assessed at the same time intervals. Inclusion of preseason control participants allowed for an estimate of “baseline” Concussion Index. Investigators were blinded to the EEG output. Only the independent biostatistician was unblinded to the results, enabling him to perform the statistical analysis. The study was approved by the institutional review boards of the primary sites (University of Rochester School of Medicine, State University of New York Upstate Medical University, Washington University in St. Louis, University of Connecticut, University of Arkansas, University of South Carolina, University of Texas at Austin, University of South Florida, University of Miami, and Michigan State University). All participants provided written informed consent, and for minors, parental written informed consent and adolescent assent were also obtained. The study was registered on ClinicalTrials.gov (NCT02957461 and NCT03671083) and followed the Standards for Reporting of Diagnostic Accuracy (STARD) reporting guideline.

Athletes with concussion consisted of male and female individuals between the ages of 13 and 25 years who met the study definition of concussion and had a Glasgow Coma Scale score of 13 or more (total range, 3-15, where 3 indicates severe injury and 15 indicates no or minor injury) at the time of injury and no hospital admission owing to either head injury or collateral injuries for more than 24 hours. Control participants had a Glasgow Coma Scale score of 15 at time of assessment.

Exclusion criteria included forehead, scalp, or skull abnormalities or a clinical condition that would not allow electrode placement; current psychoactive prescription medications taken daily (with the exception of medications being taken for attention-deficit/hyperactivity disorder); history of brain surgery or neurologic disease; pregnancy; acute intoxication; active fever, defined as greater than 37.8 °C; and inability to speak or read English. Athletes with a concussion were excluded if they had a loss of consciousness of 20 minutes or more related to the concussion injury or showed evidence of abnormality visible on a computed tomography scan of the head related to the traumatic event. Control participants were excluded if they showed focal neurologic signs, including aphasia, apraxia, diplopia, facial droop, and dysarthria or slurred speech, and had a history of traumatic brain injury or concussion or were in a motor vehicle collision requiring an emergency department visit in the past year.

Athletes with concussion were defined as those who had experienced a witnessed head impact and who, by site guidelines, were removed from play for 5 days or more. The use of site guidelines ensured broader applicability of the results to the general population of interest. Assessment of RTP (reported as the number of days from injury date to “cleared to play” date) was made in accordance with a gradual or graded RTP protocol across multiple days, at the end of which an athlete was cleared to play. For college-based and high school–based sites, this protocol conformed to National Collegiate Athletic Association and policy guidelines.^21-23 Once the participant was free of symptoms, these guidelines included (1) light aerobic exercise, (2) sport-specific activity with no head impact, (3) noncontact sport drills and resumption of progressive resistance training, (4) unrestricted training, and (5) return to competition. If at any point participants became symptomatic, they were returned to the previous level.

For the purpose of further assessing the performance of the Concussion Index relative to severity and outcome, athletes with concussion were subdivided into groups with (1) RTP between 5 and 13 days (normal or rapid recovery) and (2) RTP of 14 days or more (protracted or prolonged recovery). These time points were consistent with those reported from prior research²⁴ and were the median RTPs for the algorithm development population.¹⁹

Study participants were evaluated at each assessment time point with 3 sections of the Sports Concussion Assessment Tool–3rd edition²⁵ or 5th edition²⁶: (1) Glasgow Coma Scale score, (2) 22-item Concussion Symptom Inventory (CSI) self-rated on a Likert scale (0-6 per item; total score range, 0-132, where 0 indicates the absence of postconcussive symptoms, and 132 indicates the full range and highest severity of postconcussive symptoms),²⁷ and (3) Standard Assessment of Concussion: a brief neurocognitive screening tool (total score range, 0-30, where 0 indicates normal mental state and 30 indicates deficits in orientation, memory, and/or attention).²⁸ The total score on the CSI (total score range, 0-13) was used as an estimate of symptom burden in this study. History of head injury and concussion(s) was also acquired.

Two neurocognitive tests in previous concussion research^29,30 were performed by all participants on the handheld device under the supervision of trained research assistants. These tests included Simple Reaction Time and Procedural Reaction Time tests. Results from these tests served as candidates for inclusion in the multimodal Concussion Index and for additional characterization of the population.

Ten minutes of EEG data were collected while the participant was resting with eyes closed. A trained research assistant observed the participants throughout data acquisition for vigilance. The EEG data were recorded using a disposable head set that included the Fp1, Fp2, F7, F8, AFz, A1, and A2 locations of the expanded International 10 to 20 Electrode Placement System, rereferenced to linked ears, and all electrode impedances were below 10 kΩ. Data were acquired at a sampling rate of 1 kHz. Amplifiers had a bandpass filter from 0.3 to 250 Hz (3-dB points) and downsampled to 100 Hz for feature extraction.

The EEG signals were processed using a real-time suite of algorithms for artifact detection,³¹ which identified for removal physiological and nonphysiological contamination (eg, lateral and horizontal eye movements and muscle activity), ensuring the quality of the EEG data. Only artifact-free data (1-2 minutes) were submitted to all further analyses. The previously specified set of EEG features were then computed and z-transformed relative to age-expected normal values and used as inputs to the Concussion Index algorithm.³¹

The Concussion Index was previously derived using a machine learning method known as the genetic algorithm. The genetic algorithm method performs a stochastic search involving a series of candidate solutions in which each is informed by its predecessors, similar to an evolutionary algorithm.^32,33 In the previous derivation study, a discriminant algorithm consisting of a weighted combination of selected linear and nonlinear EEG features and selected clinical features was identified using this genetic algorithm method to optimally distinguish between participants with and participants without concussion. Brain electrical activity features were the highest contributors to the classifier, especially “connectivity” measures (eg, phase synchrony) that reflect the transmission of information between brain regions. The cutoff for assessment of the likelihood of concussion was obtained from the Concussion Index derivation data, with the threshold derived from the receiver operating characteristic curve for the final algorithm, and used for prospective validation (a participant with a Concussion Index of ≤70 was considered concussed). Further details of the Concussion Index derivation study have been published.¹⁹ The algorithm was then applied to this validation study patient population, and the prospective independent performance was reported in this publication.

Data were pooled from all 10 clinical sites. The rationale for pooling was based on 3 critical features: all sites used the same protocol, used the same data gathering mechanism, and were monitored to ensure protocol compliance. All analyses were performed by an independent biostatistician.

The Concussion Index at day 0 (within 72 hours of injury) was used to assess its classification accuracy, including sensitivity, specificity, negative predictive value, positive predictive value, and area under the receiver operator curve, for distinguishing athletes with concussion from control athletes using the previously derived Concussion Index threshold of 70 or less.¹⁹ The significance of the difference in the Concussion Index between athletes with concussion and athlete controls was evaluated using an unpaired 2-way t test for mean values and the Wilcoxon rank sum test for median values. To assess the association with symptom burden, the day 0 Concussion Index among athletes with concussion was correlated with total CSI scores using a regression analysis. The significance of the difference between the Concussion Index at day 0 for the athletes with concussion with RTP less than 14 days and those with RTP of 14 days or more was tested using a 1-sided 2-sample t test. In addition, the mean Concussion Index at day 0 was compared with the mean Concussion Index at RTP to assess the extent to which the Concussion Index improved with clinical recovery, using a paired t test. P values were deemed statistically significant at P < .05.

To estimate the statistical significance of the change in the Concussion Index over time among athletes with concussion required demonstrating the stability of the Concussion Index over time among controls. The mean Concussion Index among controls was compared at day 0 and RTP+45 days, using a paired t test of noninferiority (equivalence) with a preestablished margin of 4.5 discriminant points (using a predetermined margin based on the 0.3 SD of the distribution of discriminant scores for controls). To evaluate the percentage of athletes with concussion who returned to within the normal or control range, a target “normal” value was assessed based on the 10th percentile of the control group of ranked measurements at RTP, which was computed to be a Concussion Index of 65.1 as the cutoff for return to within normal limits. A sample size estimation based on achieving 80% power at a 1-sided α of .03 (for sensitivity and specificity) required 343 participants.

Figure 1 shows the 729 participants eligible for enrollment and the final study population of 608 participants (with assessments at day 0 or preseason). Matched athlete controls and athletes with concussion underwent follow-up EEG and clinical assessments at RTP and RTP+45 time points. An additional 28 participants were excluded owing to poor EEG quality or other missing information required as input to the Concussion Index algorithm, leaving 580 participants available for analysis. Of 1357 total EEG evaluations performed across all participants and all time points, only 39 (2.9%) were not usable. Because no pairwise analyses were performed between athletes with and without concussion, all participants with complete data sets were used in the analyses, resulting in a different number participants in each group. No severe adverse events or adverse events were reported.

The baseline characteristics of the analysis population at day 0 are presented in Table 1. There were more male participants in the group with concussion than in the control group (124 of 207 [59.9%] vs 187 of 373 [50.1%]; P = .02). All athletes with concussion had a Glasgow Coma Scale score of 15 at the time of injury and were most commonly (41 of 207 [19.8%]) injured playing football. As expected, mean (SD) day 0 CSI scores were higher in the group with concussion than in the control group (27.20 [20.42] vs 2.25 [4.00]), and 57.0% of athletes with concussion (118 of 207) required 14 days or more to RTP.

The performance of the Concussion Index for the discrimination between participants with and participants without concussion (at the predetermined threshold of 70) in this independent population is shown in Table 2. Sensitivity was 86.0% (95% CI, 80.5%-90.4%), specificity was 70.8% (95% CI, 65.9%-75.4%), the negative predictive value was 90.1% (95% CI, 86.1%-93.3%), the positive predictive value was 62.0% (95% CI, 56.1%-67.7%), and the prevalence was 35.7% (95% CI, 31.8%-39.7%). The area under the receiver operating characteristic curve for the Concussion Index in this population was 0.89.

The z-score values for the EEG features with the highest contribution to the classification (highest weights in the previously established algorithm) and the neurocognitive performance measure are summarized in Table 3 for the validation population. The EEG features were heavily represented by measures of connectivity between regions (92.3% [12 of 13]), which are known to be important in the physiology of concussion. Phase synchrony and coherence are EEG features associated with disruption in neuronal transmission between brain regions.

At day 0, the Concussion Index among athletes with concussion was significantly lower than among the athletes without concussion (mean [SD], 75.0 [14.0] vs 32.7 [27.2]; P < .001; median [interquartile range], 77.7 [68.3-84.8] vs 26.6 [7.9-55.4]; P < .001). Among all participants, there was a strong correlation between the Concussion Index at day 0 and CSI scores (R² = 0.64; r = 0.80).

In the group with concussion, there was a significant increase in the Concussion Index between day 0 and RTP, with a mean (SD) paired difference in the Concussion Index of −41.2 (27.0) and a median paired difference in the Concussion Index of –45.2 (interquartile range, –63.5 to –17.6) (P < .001) (Figure 2). The negative difference indicates that the Concussion Index was significantly lower at the time of injury compared with RTP.

Differences in the Concussion Index between day 0 and RTP+45 for athletes without concussion were close to 0, with a mean (SD) paired difference in the Concussion Index of −3.2 (15.4) and a median paired difference in the Concussion Index of −1.45 (interquartile range, –8.7 to 6.0) (P < .001) (Figure 2). This indicates that the Concussion Index obtained at these different time points is significantly equivalent, demonstrating the stability of the Concussion Index over time among athletes without concussion. At RTP, the Concussion Index of 78.2% (95% CI, 71.8%-83.7%) of the athletes with concussion was at or above the predetermined threshold for control athletes. This implies that a high percentage of athletes with concussion cleared to RTP by standard clinical practice (using the graded RTP protocol) had Concussion Indexes in the range of 90% of control athletes without concussion, consistent with recovery. This is graphically demonstrated in Figure 2, which shows clear overlap in the Concussion Index 95% CI error bars for the athletes with or without concussion.

Participants with concussion with prolonged RTP had a significantly lower Concussion Index at day 0 compared with those with quick recovery (mean [SD], 38.5 [28.1] vs 28.4 [25.9]; P = .004; median, 38.5 vs 18.2).

This validation study confirmed that a multimodal, EEG-based Concussion Index can be used with high accuracy to distinguish between athletes with a concussion and those without on the day of injury, supporting the use of the Concussion Index as an objective indicator of brain function impairment at the time of injury for participants with concussion. Significant differences between the Concussion Index at the time of injury and the Concussion Index at RTP may be associated with changes over time in the population with concussion when they are clinically cleared to RTP. Likewise, the demonstration of stability of the Concussion Index across time among the controls allows for confidence in the interpretation of changes when seen in the population with concussion, suggesting the potential utility for monitoring change throughout the recovery period and as a component of the clinical assessment of readiness to RTP. The significance of the difference between the Concussion Index at the time of injury among athletes with concussion with rapid recovery and the Concussion Index at the time of injury among athletes with concussion with protracted recovery suggests future investigations of algorithms to predict outcome. The results from this study were used in support of the submission to the US Food and Drug Administration (FDA) for the commercialization of this algorithm, which was granted on September 11, 2019.

In addition, although for most athletes with concussion (78.2%) the Concussion Index at RTP improved to within the Concussion Index range of uninjured controls, these results suggest that some athletes with concussion may have been cleared for RTP before brain function impairment had resolved. Similar findings were reported in the literature attesting to the persistence of brain function abnormalities in individuals with sports-related concussion beyond the point when clinical symptoms have resolved.^34-37 Because current RTP graded protocols have a significant dependence on participant-reported symptoms, which may resolve when brain function impairment persists, the potential importance of an additional objective measure as part of the final assessment of RTP is highlighted.

This study used a previously derived machine learning classification algorithm for the assessment of the likelihood of concussion, the components of which have important implications about the pathophysiology of concussion. The features with the highest contribution to the classification algorithm were the EEG features that characterized deviations from normal connectivity between brain regions, both between and within hemispheres. Furthermore, connectivity as reflected in advanced neuroimaging studies of concussion has led to a consensus that the underlying physiology of concussion is associated, in part, with the disruption of neuronal transmission.^38-41 A publication using data from the Concussion Index derivation study reported significant correlations between disruption of white matter tracts evidenced on diffusion tensor imaging and the Concussion Index.⁴² These results indicate that the Concussion Index is associated with the underlying disruption of neuronal transmission.

Although centered around the EEG features, the Concussion Index included neurocognitive performance measures and vestibular symptoms. These measures were predicted from the prior research to have multimodal associations because both neurocognitive performance deficits^30,43-46 and vestibular symptoms^47-50 have been reported to be common sequelae of mild traumatic brain injury. The combination of multiple dimensions characterizing concussion was associated with the high accuracy of the algorithm.

The association between the Concussion Index and symptom burden at the time of injury was demonstrated in the highly significant regression obtained between the CSI and Concussion Index across the population, with decreasing Concussion Index (more abnormal) significantly correlated with increasing CSI (higher symptom burden). The high correlation suggests that the severity of concussion as assessed by CSI was associated with the Concussion Index. Although the CSI is based on self-report and subject to underreporting and/or overreporting, the Concussion Index is an objective measure less prone to subjective reporting^51-53 and poor reliability.⁵⁴

This study has some limitations, including the lack of intermediate time points for the assessment of change throughout recovery. Future studies will need to aim to perform more evaluations through the recovery phase to better estimate the recovery rates of individuals and the association with changes in the Concussion Index and to further investigate the predictive accuracy of the Concussion Index at the time of injury. Another limitation is the restricted age range for which this study was performed (13-25 years); although these ages are critical for high school and college student athletes, studies are currently under way to address this age limitation.

In this diagnostic study, the objective multimodal Concussion Index with the EEG at its core classified participants with concussion at the time of injury with high accuracy and showed significant improvement in the level of uninjured controls at time of recovery (RTP). The FDA-cleared Concussion Index is easy to use (embedded in the BrainScope handheld device) and lends itself to being incorporated into existing standard assessments of concussion to aid in clinical diagnosis and assessment of readiness to RTP.

Accepted for Publication: December 27, 2020.

Published: February 15, 2021. doi:10.1001/jamanetworkopen.2020.37349

Open Access: This is an open access article distributed under the terms of the CC-BY-NC-ND License. © 2021 Bazarian JJ et al. JAMA Network Open.

Corresponding Author: Jeffrey J. Bazarian MD, Department of Emergency Medicine, University of Rochester, 601 Elmwood Ave, PO Box 655c, Rochester, NY 14642 (jeff_bazarian@urmc.rochester.edu).

Author Contributions: Dr Bazarian had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Bazarian, Covassin.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Bazarian.

Critical revision of the manuscript for important intellectual content: All authors.

Obtained funding: Lopez.

Administrative, technical, or material support: Casa, Hotz, Neville, Schnyer, Yeargin.

Supervision: Elbin, Casa, Neville, Lopez, Schnyer, Yeargin, Covassin.

Conflict of Interest Disclosures: All authors were principal investigators at one of the clinical data acquisition sites participating in the research, whose institutions received contracts to support participant recruitment, consenting, and data acquisition and have no competing financial interests to disclaim. Drs Bazarian, Casa, Lopez, Schnyer, Yeargin, and Covassin reported receiving grants from BrainScope Co Inc during the conduct of the study. Dr Bazarian reported receiving personal fees from Abbott and Q30 Innovations outside the submitted work. Dr Elbin reported receiving grants and consulting fees from BrainScope Co Inc during the conduct of the study. Dr Neville reported being an equity owner of Quadrant Biosciences Inc outside the submitted work. No other disclosures were reported.

Funding/Support: The clinical study was funded in part by contract W911QY-14-C-0098 to BrainScope Co Inc from the US Navy (Naval Health Research Center).

Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The views, opinions and/or findings contained in this report are those of the author(s) and should not be construed as an official Department of the Navy position, policy, or decision unless so designated by other documentation.

Meeting Presentation: This study was presented at the 15th Annual Conference on Brain Injury of the North American Brain Injury Association; February 26, 2020; New Orleans, Louisiana.

Additional Contributions: The authors acknowledge the contributions of those who made this research possible, including research staff at all the clinical sites, and the participants who volunteered for the study. Richard P. Chiacchierini, PhD, R.P. Chiacchierini Consulting LLC, served as the independent biostatistician for the US Food and Drug Administration (FDA) validation study, the results of which are reported in this article, and while the analyses were conducted as part of the Statistical Analysis Report to the FDA, he was not an author on this article; he was compensated for his contribution.