Figure 1. Classification of command-following data in healthy control subjects. Statistical comparisons between whole-brain and region-of-interest (ROI) analyses were performed using 2-tailed t test (* P < .05 was considered significant.) Horizontal line indicates chance level.
Figure 2. Univariate analysis of command-following data. A, In a representative control subject. B-E, In patients 1, 2, and 3 (D and E show responses of patient 3 at test 1 and test 2, respectively). Numerals on the color bar indicate t scores. Univariate analysis methods are given in the eAppendix .
Figure 3. Classification of command-following data in patients with severe brain injury. Statistical comparisons between whole-brain and region-of-interest (ROI) analyses were performed using 2-tailed t tests (asterisks) * P < .05 was considered significant. Patient No. 3 (1) indicates 3 (test 1); 3(2), 3 (test 2). Mean POS indicates the mean calculation for subjects who had a positive result in the univariate analysis. Horizontal line indicates chance level.
Figure 4. Univariate analysis for the multiple-choice card-guessing paradigm. A, Classification of communication face card data in patient 1. Top: Shown is the performance of a classifier trained on the patient's command-following data and tested on the face card data. Statistical significance was determined first using 1-way analysis of variance, followed by application of Scheffé test (P < .05, corrected for multiple comparisons). Bottom: The results of a univariate general linear model (GLM) analysis are shown for each face card. The symbol for the correct card is outlined in white. B, Same as in A for the suit card data. C, Same as in B for a representative control subject. *Statistically significant differences using the statistical test described earlier.
Bardin JC, Schiff ND, Voss HU. Pattern Classification of Volitional Functional Magnetic Resonance Imaging Responses in Patients With Severe Brain Injury. Arch Neurol. 2012;69(2):176-181. doi:10.1001/archneurol.2011.892
Author Affiliations: Department of Neuroscience, Weill Cornell Graduate School of Medical Sciences (Mr Bardin), and Departments of Neurology and Neuroscience (Dr Schiff) and Radiology and Citigroup Biomedical Imaging Center (Dr Voss), Weill Cornell Medical College, Cornell University, New York, New York.
Background Recent neuroimaging investigations have explored the use of mental imagery tasks as proxies for an overt motor response, in which patients are asked to imagine performing a task, such as “Imagine yourself swimming.”
Objectives To detect covert volitional brain activity in patients with severe brain injury using pattern classification of the blood oxygenation level–dependent (BOLD) response during mental imagery and to compare these results with those of a univariate functional magnetic resonance imaging analysis.
Design Case-control study.
Setting Academic research.
Participants Experiments were performed in 8 healthy control subjects and in 5 patients with severe brain injury. The patients with severe brain injury constituted a convenience sample.
Main Outcome Measures Functional magnetic resonance imaging data were acquired as the patients were asked to follow commands or to answer questions using motor imagery as a proxy response.
Results In the controls, the responses were accurately classified. In the patient group, the responses of 3 of 5 patients were correctly classified. The remaining 2 patients showed no significant BOLD response in a standard univariate analysis, suggesting that they did not perform the task. In addition, we showed that a classifier trained on command-following data can be used to evaluate a later communication run. This technique was used to successfully disambiguate 2 potential BOLD responses to a single question.
Conclusions Pattern classification in functional magnetic resonance imaging is a promising technique for advancing the understanding of volitional brain responses in patients with severe brain injury and may serve as a powerful complement to traditional general linear model–based univariate analysis methods.
Many patients with severe brain injury lack a reliable motor output channel, making accurate assessments of cognition difficult or impossible.1- 3 Recent neuroimaging investigations have explored the use of mental imagery tasks as proxies for an overt motor response, in which patients are asked to imagine performing a task, such as “Imagine yourself swimming.” These paradigms have been useful in identifying covert command following and, in a single case,4 the ability to answer simple yes or no questions.5,6
These investigations have used traditional univariate data analysis techniques, focusing primarily on the presence or absence of clusters of activation in or near the supplementary motor area (SMA) for readout of motor imagery performance. In particular, they have focused on region-of-interest (ROI) approaches, limiting the analysis to the SMA. While univariate techniques allow for the discovery of clusters of individual voxels that meet a statistical threshold, multivariate techniques can allow for detection of spatially distributed patterns of activation that are unlikely to be revealed using univariate techniques, particularly those focused on a limited ROI.7 A previous study5 among patients with severe brain injury revealed significant activation outside of the SMA, suggesting that a spatially agnostic approach may be useful in detecting responses in this group. In this study, we apply multivariate pattern analysis (MVPA) classification to functional magnetic resonance (fMR) imaging data of patients following a command to “Imagine yourself swimming” in 8 healthy control subjects and in 5 patients with severe brain injury.
Experiments were performed in 8 healthy control subjects and in 5 patients with severe brain injury. The Institutional Review Board of Weill Cornell Medical College approved all experiments, and informed consent was obtained from the healthy volunteers and from the legally authorized representatives of the patients with severe brain injury. The patients with severe brain injury constituted a convenience sample. For the patient group, the Table summarizes demographic information, cause of injury, and behavioral assessments using the Coma Recovery Scale–Revised.8 More details about individual patients are given in the eAppendix.
In the command-following paradigm, the controls heard instructions to imagine themselves swimming, starting with a command “Imagine yourself swimming” and stopping with a “Stop” command. In the interim, the controls were required to think of nothing in particular. Eight blocks of 16 seconds of rest alternated with 16 seconds of motor imagery. Instructions were part of the task blocks (4 seconds). For the patients, the same timing was used, but the instructions during imaging were “Imagine yourself swimming. . . . Stop imagining swimming.” One run of 16 trials each was collected for each participant.
In the multiple-choice paradigm, the participants were taught the suit and face of a playing card. Controls chose a card at random from a stack of face cards. The patients with severe brain injury who performed this task were shown and told the identity of a face card by an investigator who was not involved in the data analysis. The task consisted of 12 seconds of response, in which the participant imagined swimming, followed by 4 seconds of rest, for each suit or face, repeated 4 times. The wording was “If your card is a [club/diamond/heart/spade or ace/jack/king/queen], imagine swimming now. . . . Stop.” Two runs of 16 trials each were collected for each participant.
Before the MR imaging, the participants were instructed to lie still with their eyes closed. Soft padding was placed around the head and was anchored by the head coil caging to limit motion. Tasks were verbally explained to the participants before the experiments, and instructions were repeated immediately before each corresponding imaging. The participants used foam earplugs for noise protection and headphones with noise protection capability. During data acquisition, prerecorded auditory instructions were played out on a PC with an MR imaging–compatible audio system (Resonance Technology, Inc). The volume of the headphones was adjusted to the comfort level of the controls. For the patients, the volume was set at the comfort level of one of us (H.U.V.). Data were acquired on a 3.0-T MR imaging system (Signa Excite HDx; General Electric) with an 8-channel head receive–only coil. For fMR imaging, a GE-EPI sequence was used (2-second repetition time, 30-millisecond echo time, 70° flip angle, 64 × 64–pixel acquisition and reconstruction matrix, 28 sections of 5-mm thickness, and 24-cm field of view). The resulting voxel size was 3.75 × 3.75 × 5 mm. The paradigm was the same for controls and for patients (128 total repetitions during 4 minutes and 16 seconds). To ensure saturation of the signal, at least 4 acquisitions at the beginning of each imaging session were discarded before starting the tasks.
The MVPA classification was performed using the open-source Princeton MVPA Toolbox (http://code.google.com/p /princeton-mvpa-toolbox)9 running in a commercially available software program (MATLAB; MathWorks). Data from 1 functional run per participant were broken up into sixteen 12-second trials (8 trials of rest and 8 trials of imagining swimming). The periods when instructions were being given were excluded from the analysis. All data were convolved with a Cox proportional hazards regression model special hemodynamic response function using the waver function in the open-source Analysis of Functional NeuroImages toolbox (http://afni.nimh.nih.gov/afni). To eliminate uninformative voxels, we implemented a feature selection step in which voxels that did not significantly correlate with the regressors of interest were eliminated. We determined significance for each voxel individually using an analysis of variance (ANOVA) as implemented by the Princeton MVPA Toolbox code statmap_anova (P < .01 was considered significant). Crucially, the ANOVA was only conducted on training data for each run, ensuring that feature selection was not “peeking.” The data were also z scored to control for baseline shifts between different runs.9
For the command-following paradigm, classification was performed using a scaled conjugate gradient10 neural network approach. It was implemented as a leave-one-out classifier (as performed by the Princeton MVPA Toolbox code train_bp_netlab), in which the data from 15 trials were used to predict the data in the 16th trial. Briefly, the scaled conjugate gradient algorithm iteratively adjusts an initially randomized set of internal weights based on the spatial pattern of blood oxygenation level–dependent (BOLD) activity present in the training data, and the final set of weights is used to classify the test data. The algorithm finds this optimal set of weights by iteratively moving stepwise through weight space until the error function is minimized.10 Although this approach is perfectly optimized only for a truly convex weight space, it is a powerful training algorithm and has been validated in fMR imaging studies.11- 13
This process was performed 16 times, once for each trial, and the results were averaged to calculate a mean classification rate across the run. Because each trial had a duration of 12 seconds with a volume acquisition time of 2 seconds, the classification of each trial had a maximum correct performance of 6. This analysis was performed twice per participant, once with a whole-brain mask, where the classifier was trained and tested on whole-brain data, and once with a mask encompassing the premotor and motor areas, where the classifier was both trained and tested on data from this ROI. The objective of this dual analysis was to determine whether an analysis restricted to areas classically associated with motor imagery would perform differently than one using data from the entire brain. Such an approach is motivated by the common use of ROI methods in this subfield, which may or may not be optimal for the detection of volition in the severely injured brain.
The analysis of data during the communication task was performed using the same classification techniques as already described. To determine whether responses observed in a multiple-choice communication paradigm were similar to those in a command-following paradigm, the classifier was trained on command-following data and then applied to the communication data. This approach was motivated by the results of applying a standard univariate analysis (methods are described in the Appendix), which revealed activations at or near the location of the command-following response for more than 1 multiple-choice response in one of the patients with severe brain injury studied herein.5 These results raised the question of whether multiple responses were truly signaled or if one response was more similar to the command-following response than the other, suggesting only one signaled response.
In controls and in patients, we found that using MVPA classification with a whole-brain data set provided more accurate results than a classification restricted to the SMA. This suggests that a distributed pattern of activity best distinguishes motor imagery from rest. Most important, we found that a classifier trained on command-following data discriminated between potential responses in a communication paradigm in which a patient with severe brain injury was asked to respond to a multiple-choice question using motor imagery.
All the control subjects showed MVPA classification rates significantly above the chance level of α = .50 in the whole-brain and ROI-masked analyses (Figure 1). The mean classification rate α across 8 controls using data from the whole brain was α = .80, while the mean classification rate for the ROI analysis was α = .66. This difference was statistically significant (P < .001, 2-tailed t test). Individual classification performances by all controls were better for the whole-brain approach than for the ROI approach, although not all were statistically significantly different. Univariate analysis of these data found that all controls demonstrated a statistically significant BOLD response; these responses in a representative control subject are shown in Figure 2A.
Three of 5 patients with severe brain injury showed MVPA classification rates significantly above chance level (Figure 3). Patient 1 and patient 2 demonstrated MVPA classification rates above chance level in the whole-brain analyses (α = .76 and α = .78, respectively) and in the ROI analyses (α = .68 and α = .65, respectively), consistent with the results of the univariate analysis, which demonstrated significant task-related BOLD activity for both patients in and around the SMA (Figure 2B and C). Patient 3 was evaluated on 2 separate visits and demonstrated MVPA classification rates above chance level on both visits for the whole-brain analyses (α = .67 and α = .80, respectively) and on the first visit only for the ROI analyses (α = .66 and α = .45, respectively) (Figure 3). Whole-brain univariate analysis of data from patient 3 at her first visit revealed activity in and near the ROI (Figure 2D).5 Univariate analysis of data from her second visit revealed a BOLD response outside of the ROI near the dorsolateral prefrontal cortex, without significant activity in the ROI (Figure 2E).4 Notably, results from the multivariate analysis strongly dissociate the whole-brain and ROI analyses (Figure 3). Patient 4 and patient 5 demonstrated MVPA classification rates at chance level. These results are consistent with the results of the whole-brain univariate analyses.5 Although the difference between the whole-brain and ROI analyses was not significant across all patients with severe brain injury, analysis across those patients who demonstrated a response in the univariate analysis showed a significant difference between the analyses (P < .001, 2-tailed t test) (Figure 3).
We also performed univariate analysis for the multiple-choice card-guessing paradigm. For patient 1, that analysis demonstrated a statistically significant response to more than 1 potential stimulus (Figure 4B).5 To attempt to resolve this intermediate result, we examined whether a classifier trained on command-following data would discriminate between these 2 BOLD responses. We initially tested this approach on patient data when univariate methods suggested only one response, specifically when patient 1 was asked to indicate the face of the card learned (Figure 4A). In this data set, the classifier trained on command-following data performed above chance level in detecting a response to a jack (α = .69), while it performed at near-chance levels when attempting to detect a response to an ace (α = .50) and all other face cards. An ANOVA, followed by application of the Scheffé test, determined that this difference was statistically significant (P < .05, corrected for multiple comparisons).
We then used the same approach with the suit data, where the univariate results were more ambiguous and a signal seemed to indicate both club and spade (Figure 4B). Notably, the classifier trained on command-following data disambiguated these responses, classifying a response to a club well above chance level (α = .69), while classifying a response to spade and all other face cards at chance levels. An ANOVA, followed by application of Scheffé test, determined that this difference was statistically significant (P < .05, corrected for multiple comparisons). Interpretations of this disambiguation are discussed in the “Comment” section.
For comparison, an identical analysis was performed for a control subject who performed the command-following and communication tasks during the same experimental session. Similarly, an ANOVA, followed by application of Scheffé test, determined that the control subject's response to the correct card was statistically significant (P < .05, corrected for multiple comparisons) (Figure 4C).
In this study, we found that an MVPA machine-learning algorithm successfully detected covert volitional neural activity across 8 healthy control subjects and 5 patients with severe brain injury. A related goal was to determine whether a classifier trained on command-following data from an ROI encompassing the premotor and motor areas would perform better than one trained on data from the whole brain, given that motor imagery is traditionally associated with differential BOLD activity in such areas and previous investigations have used analysis methods restricted to the ROI. We found the opposite: in 6 of 8 control subjects and in 3 of 5 patients with severe brain injury, classifiers performed significantly better when data from the whole brain were used. Indeed, there is evidence from univariate analyses that significant activity exists well outside of the ROI, even for individuals who show ROI activation.5 As such, there is growing evidence that the SMA ROI may be an overly limited region for capturing these BOLD activity changes. This seems to be especially true in the case of certain patients with severe brain injury who appear to have only task-related activity outside of the SMA, such as patient 3 herein.
Our data also suggest that pattern classification methods can be used to disambiguate multiple potential BOLD responses to a single question. We found that a single patient herein produced a strong BOLD response to 2 adjacent responses to a multiple-choice question (Figure 4B), one of which was correct and one of which was 1 response block too late. Most important, that same patient, when asked a slightly different question, had only one significant BOLD response, in the response block directly after the correct answer (Figure 4A). As such, we hypothesized that a cognitive delay could explain both responses; the patient may have attempted to respond only during the appropriate response period, but because of her injury, the response was delayed by a single response block.
In this context, we asked whether a classifier could disambiguate these multiple BOLD responses. We hypothesized that the classifier would classify the 1-block delayed responses above chance level, while classifying the correct answers at chance levels. Indeed, this is what our data show (Figure 4). This finding supports the hypothesis of cognitive delay and suggests that patient 1 with severe brain injury in fact performed a successful communication event. More generally, it demonstrates that pattern classification methods can be useful in disambiguating intermediate responses by patients who are unable to clarify their answers themselves. In principle, demonstrating that an individual seems to have communicated is extremely important because the identification of communication events can guide medical care and surrogate decision making and may be linked to recovery.
The MVPA classification is increasingly being recognized as a powerful fMR imaging data analysis method.7 While most MVPA studies to date have focused on detecting nuanced differences between different classes of stimuli via classification of covert neuronal activation patterns, such as movies14 or visually presented words,15 the objective of the present study was to determine whether such a machine-learning algorithm could successfully determine the presence or absence of task-related volitional brain activity. The success of the method suggests that MVPA analysis is a promising approach for detection of volitional activity in patients who have impaired motor function.
While all command-following and communication fMR imaging studies in patients with severe brain injury to date have been performed using univariate methods, there are several reasons why pattern classification approaches may be more appropriate. We argue that the fundamental goal of this research should not be to discover exact locations of task-related BOLD activity but rather to determine whether a rigorous statistical analysis finds a difference between the conditions. As a result, analyses may be able to be spatially agnostic without losing information about task performance. As a corollary, hypotheses of where activity should be found have been necessarily based solely on control data,16 and as such this pattern of activity may not capture the variance that can be expected from a severely injured brain (as shown in Figure 2). There is significant evidence in particular that reorganization of functional networks after brain injury can occur, which may render activation maps of controls obsolete.17- 19 If these paradigms are used in the future to ask patients important clinical or personal questions, many of which are more complex than a simple yes or no response, it is essential that the interpretation of the response should be as accurate as possible. Within this context, our results represent a proof of concept that pattern classification methods can be useful in increasing the accuracy and specificity of the analysis of these volitional paradigms.
In this study, we provide evidence that MVPA analysis is sufficient to identify command-following performance in healthy control subjects and in patients with severe brain injury. In addition, we show that in this paradigm whole-brain MVPA analyses perform better than those restricted to the SMA ROI. Last, we found in a single patient that command-following data could be used as a localizer to train a classifier for subsequent detection of communication responses. In addition, the significant improvements in whole-brain analyses over ROI analyses provide further evidence that a distributed pattern of activity underlies the signal being measured.
Given the small sample of this study, it is important for these techniques to be further benchmarked in a study of far greater size. In addition, increasing the imaging time or the number of trials may significantly improve the quality of the MVPA classification. However, these initial results provide compelling evidence that MVPA classification may provide a powerful complementary approach to measuring volitional responses in patients with severe brain injury.
Correspondence: Nicholas D. Schiff, MD, Department of Neurology and Neuroscience, Weill Cornell Medical College, 1300 York Ave, New York, NY 10021 (firstname.lastname@example.org).
Accepted for Publication: May 19, 2011.
Author Contributions:Study concept and design: Bardin, Schiff, and Voss. Acquisition of data: Bardin, Schiff, and Voss. Analysis and interpretation of data: Bardin, Schiff, and Voss. Drafting of the manuscript: Bardin, Schiff, and Voss. Critical revision of the manuscript for important intellectual content: Bardin, Schiff, and Voss. Statistical analysis: Bardin, Schiff, and Voss. Obtained funding: Schiff. Administrative, technical, and material support: Schiff and Voss. Study supervision: Schiff.
Financial Disclosure: None reported.
Funding/Support: This study was supported by grants R01 HD51912 from the Eunice Kennedy Shriver National Institute of Child Health and Human Development from the National Institutes of Health and UL1 RR024996 from the Weill Cornell Center for Translational Science Activity, and funds from the James S. McDonnell Foundation, Dana Foundation (Dr Schiff), Lounsbery Foundation (Dr Schiff), and Jerold B. Katz Foundation (Dr Schiff).