A, Each dot represents a point value from 0 to 6 points. Point values are only shown up to 6 because no patients in the database had 7 points. The x-axis is the probable seizure risk based on the SLIM 6-variable RiskSLIM model. The y-axis is the actual observed risk, estimated as the fraction of patients with a given score who had seizures. The black line has a slope of 1 and intercept at the origin. Proximity to this line indicates goodness of fit and is used as a marker to look for bias. The number associated with each dot is the number of patients in the Critical Care EEG Research Consortium database with the associated number of points. B, Receiver operating characteristics curve for the RiskSLIM model with 95% CIs developed from bootstrapping from the full training set is represented by the dashed lines. The solid black line represents the null classifier. Area under the curve = 0.819.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Struck AF, Ustun B, Ruiz AR, et al. Association of an Electroencephalography-Based Risk Score With Seizure Probability in Hospitalized Patients. JAMA Neurol. 2017;74(12):1419–1424. doi:10.1001/jamaneurol.2017.2459
Can the risk of seizures in critically ill patients be accurately determined with a simple clinical tool?
In this study, a point system using 6 variables (brief [ictal] rhythmic discharges [2 points]; presence of lateralized periodic discharges, lateralized rhythmic delta activity, or bilateral independent periodic discharges [1 point]; prior seizure [1 point]; sporadic epileptiform discharges [1 point]; frequency greater than 2.0 Hz of periodic/rhythmic pattern [1 point]; and presence of “plus” features [1 point]) was associated with seizure risk of 5% with a score of 0, 12% with a score of 1, 27% with the score of 2, 50% with the score of 3, 73% with a score of 4, 88% with a score of 5, and greater than 95% with a score of 6 or 7.
The 2HELPS2B score may provide accurate seizure risk stratification from patient history and initial electroencephalography.
Continuous electroencephalography (EEG) use in critically ill patients is expanding. There is no validated method to combine risk factors and guide clinicians in assessing seizure risk.
To use seizure risk factors from EEG and clinical history to create a simple scoring system associated with the probability of seizures in patients with acute illness.
Design, Setting, and Participants
We used a prospective multicenter (Emory University Hospital, Brigham and Women’s Hospital, and Yale University Hospital) database containing clinical and electrographic variables on 5427 continuous EEG sessions from eligible patients if they had continuous EEG for clinical indications, excluding epilepsy monitoring unit admissions. We created a scoring system model to estimate seizure risk in acutely ill patients undergoing continuous EEG. The model was built using a new machine learning method (RiskSLIM) that is designed to produce accurate, risk-calibrated scoring systems with a limited number of variables and small integer weights. We validated the accuracy and risk calibration of our model using cross-validation and compared its performance with models built with state-of-the-art logistic regression methods. The database was developed by the Critical Care EEG Research Consortium and used data collected over 3 years. The EEG variables were interpreted using standardized terminology by certified reviewers.
All patients had more than 6 hours of uninterrupted EEG recordings.
Main Outcomes and Measures
The main outcome was the average risk calibration error.
There were 5427 continuous EEGs performed on 4772 participants (2868 men, 49.9%; median age, 61 years) performed at 3 institutions, without further demographic stratification. Our final model, 2HELPS2B, had an area under the curve of 0.819 and average calibration error of 2.7% (95% CI, 2.0%-3.6%). It included 6 variables with the following point assignments: (1) brief (ictal) rhythmic discharges (B[I]RDs) (2 points); (2) presence of lateralized periodic discharges, lateralized rhythmic delta activity, or bilateral independent periodic discharges (1 point); (3) prior seizure (1 point); (4) sporadic epileptiform discharges (1 point); (5) frequency greater than 2.0 Hz for any periodic or rhythmic pattern (1 point); and (6) presence of “plus” features (superimposed, rhythmic, sharp, or fast activity) (1 point). The probable seizure risk of each score was 5% for a score of 0, 12% for a score of 1, 27% for a score of 2, 50% for a score of 3, 73% for a score of 4, 88% for a score of 5, and greater than 95% for a score of 6 or 7.
Conclusions and Relevance
The 2HELPS2B model is a quick accurate tool to aid clinical judgment of the risk of seizures in critically ill patients.
Continuous electroencephalography (cEEG) provides real-time monitoring of brain function in hospitalized patients. The use of cEEG is expanding, motivated by reports showing a high incidence of subclinical seizures in encephalopathic patients with conditions ranging from sepsis to traumatic brain injury.1-3
Quiz Ref IDFeatures of EEG reported as factors associated with of seizures include epileptiform and periodic discharges.4 However, to our knowledge, no study has examined how these factors affect seizure risk jointly, that is, it is unknown how seizure risk changes when several patterns occur simultaneously.
We propose a simple scoring system for seizure risk that we refer to as the 2HELPS2B score. Our tool provides a joint assessment of seizure risk from cEEG observations and history of seizures, and it allows physicians to make accurate, risk-calibrated probabilities by hand. We expect our tool to help physicians identify patients in need of continued cEEG monitoring and who are likely to benefit from interventions.
Following institutional review board approval at Emory University, Brigham and Women’s Hospital, and Yale University, institutions prospectively entered participant data into an anonymized database.5 Waiver of consent was granted because of minimal risk to patients. The database includes reports of clinical information and findings on cEEG greater than or equal to 6 hours. The cEEG findings were coded using American Clinical Neurophysiology Society standardized terminology.6 Clinical variables were collected as described in Lee et al.5 Patients admitted for elective epilepsy monitoring were excluded. Data from 5427 cEEG sessions on 4772 different patients were collected. All investigators entering patient data had to undergo a module explaining the patterns and an examination demonstrating mastery of the material. This method has been shown to have high interrater reliability.7 Seizures are not defined in the American Clinical Neurophysiology Society terminology, but most clinicans used the modified Young et al8 criteria to define seizures. Both electrographic and electroclinical seizures were included.
Quiz Ref IDWe considered 24 candidate variables for inclusion in risk models (Table 1). Posterior dominant rhythm; brief (ictal) rhythmic discharges (B[I]RDs); reactivity; sporadic (nonperiodic and nonrythmic) epileptiform discharges; history of seizure, generalized rhythmic delta activity (GRDA), lateralized rhythmic delta activity (LRDA), generalized periodic discharges (GPDs), lateralized periodic discharges (LPDs), and bilateral independent periodic discharges (BIPDs); primary neurological diagnosis (altered mental status, infection, inflammatory disease, cerebral neoplasm, hypoxic/ischemic encephalopathy, intracerebral hemorrhage, metabolic encephalopathy, stroke, subarachnoid hemorrhage, subdural hemorrhage, traumatic brain injury, and hydrocephalus); frequency of rhythmic or periodic patterns; presence of a stimulus-induced pattern; and presence of a “plus factor” (ie, superimposed rhythmic, fast, or sharp activity). Candidate variables were selected based on prevalence within the database and previous associations with seizures.
Variables were combined into single factors to simplify the prediction model and increase the effect size for each factor. This was performed for variables that are associated with a similar risk of seizures and rarely co-occur. To create a frequency binary variable, frequency was divided into binary variables at each 0.5-Hz interval from 0.5 to 3 Hz. Each potential dividing point was analyzed to find the cut point with maximal predictive value.
Descriptive statistics are reported with 95% CIs. Odds ratios and Fisher exact test results are reported for candidate variables with α set to .05.
Our goal was to create a risk score similar to CHADS2 (congestive heart failure, hypertension, age greater than 75, diabetic, and history of stroke [doubled]),9 that is, a simple additive model with a limited number of factors and small integer weights for quick calculations. There is no standard method to create such models. Existing tools were built manually (eg, CHADS2 a point system for stroke risk with atrial fibrillation)9 or by combining logistic regression with ad hoc feature selection and rounding (eg, simplified acute physiology score [SAPS II], a point system for mortality in the intensive care unit).10
Existing approaches may fail to produce risk-calibrated models. Therefore, we built our risk score using a new method known as Risk-Calibrated Supersparse Linear Integer Model (RiskSLIM).11 This RiskSLIM method uses optimization techniques to find the best logistic regression model with bounded integer coefficients (integers between –10 and 10), and a limited number of risk factors (at most 6). In such settings, RiskSLIM can output an optimized risk score with superior risk-calibration and/or area under the curve (AUC). Because RiskSLIM is a new method, we compared RiskSLIM models with baseline models built using state-of-the-art methods: penalized logistic regression (PLR) with a combined L1/L2 penalty using the same constraints.
We evaluated all models for accuracy and risk calibration (ie, how well the predicted probability of a seizure matches the true prevalence). To assess accuracy, we computed the area under the receiver operating characteristic curve (ROC). Quiz Ref IDTo assess risk calibration, we constructed reliability diagrams plotting the observed prevalence of seizures vs the predicted probability (eg, Figure, A).12 In addition, we examined the average calibration (CAL) error, the mean squared error between the predicted probability and the observed prevalence. When a model has perfect risk calibration, the reliability curve should lie on the 45° line, and CAL should be 0% (Figure, A). The average CAL error is a measure of how close the probable risk of seizures and the actual risk of seizures are. It is minimized to find the best risk model.
We validated the performance of all models using standard 5-fold cross-validation (5-CV). That is, we randomly split the data into 5 parts, fit the model using 4 of 5 folds, and validated this model on last fold (that the model had not seen). This procedure was repeated 5 times, each time using a different fold for validation, to obtain 5 independent estimates of CAL and AUC. We report the mean of these estimates as 5-CV CAL and 5-CV AUC.
Because fitting models with PLR requires us to specify free parameters, we fit models for more than 1100 combinations of free parameters and picked the combination that maximized the 5-CV test AUC. This required us to validate the performance using a nested 5-CV procedure. All results for model performance are reported with respect to the left-out data (the fold used for testing) only; testing data were held out and were not used for either choosing the values of free parameters nor for training the model. This rigorous separation of training and testing data provides protection against overfitting and minimizes bias in the reported model performance.
Among 5427 cEEG sessions, 719 (12.52%) had a seizure during cEEG; 2315 (40.03%) had GRDA, LRDA, BIPDs, LPDs, or GPDs. A total of 340 (5.92%) had sporadic epileptiform discharges.
After fitting several models using RiskSLIM and PLR for model size constraints ranging between 4 and 27, we selected a RiskSLIM model with 6 variables shown in Table 2.
In contrast to the baseline PLR model, the RiskSLIM model was simpler, had superior risk calibration (mean 5-CV CAL of 2.7% [95% CI, 2.0%-3.6%] vs 8.9% [95% CI, 7.9%-9.8%] for PLR), and had comparable AUC (mean 5-CV AUC of 0.819 [95% CI, 0.799-0.849] vs 0.821 [95% CI, 0.801-0.855] for PLR). We also compared 2HELPS2B with a PLR model where we did not round the coefficients or constrain the number of variables. In this case, we did obtain a model with slightly better risk calibration (mean 5-CV CAL 2.0% [95% CI, 1.5%-3.0%]) and improved AUC (mean 5-CV AUC of 0.837 [95% CI, 0.815-0.868]), but this model is no longer simple enough to use for quick predictions as the points are not integers and it used 21 of the 29 variables.
As a mnemonic, we call this RiskSLIM model the 2HELPS2B score, which represents GRDA, LRDA, BIPDs, LPDs, or GPDs with a frequency greater than 2 Hz (1 point); epileptiform discharges (1 point); LPDs or LRDA or BIPDs (1 point); GRDA, LRDA, BIPDs, LPDs, or GPDs with plus features (superimposed rhythmic, fast, or sharp activity); any history of seizures; (acute or remote) (1 point); and B(I)RDs (2 points).
Quiz Ref IDThe risks of seizures for each possible 2HELPS2B score are 5% for a score of 0, 12% for a score of 1, 27% for a score of 2, 50% for a score of 3, 73% for a score of 4, 88% for a score of 5, and greater than 95% for a score of 6 or 7. Table 2 provides a reference with the probabilities for each score from 1 to 6. The area under the ROC for this model applied to all patients was 0.819 and for the 5 folds ranged between 0.776 and 0.849. Figure, A is a risk-calibration plot of probable vs actual incidence of seizures at each point level. Figure, B plots the ROC with 95% CIs.
The 2HELPS2B score is an accurate, simple, and clinically practical risk score for seizure occurrence in hospitalized patients undergoing cEEG. The large sample size of data collected at multiple institutions with a systematic application of standardized EEG nomenclature fostered development of a robust risk scoring system. The large sample size provides statistical power; the multiple institutions and uniform data collection ensure broad applicability.
Quiz Ref IDThe 2HELPS2B system combines 5 readily observable EEG features with a single factor from the patient history (any known history of seizure, remote or acute) to assign a score between 0 and 7. The score has good face validity, being based on established clinical and EEG risk factors. Moreover, it shows excellent CAL: the probabilities it assigns for each level of risk closely match those observed in our cohort. The association of higher frequency (>1.5-Hz) discharges and increased risk of seizures seen in the study by Rodriguez Ruiz et al13 was confirmed to have independent association value in the 2HELPS2B investigation.
The rigorous cross-validation method that we used and the large cohort size of 5427 ensures our results are widely applicable. Supporting the generalizability of our study, the incidence of seizures in our cohort is within the 8% to 34% range of published reports.1,14-22 Subgroups also have an incidence similar to prior studies, such as stroke at 10% (range, 6%-26%) and subarachnoid hemorrhage at 7% (range, 4%-19%).1-3,17
There are some limitations of the study. The duration of cEEG was not included in the database; thus, this study does not address the change in probability of seizures with increased observation duration. This issue has been partially addressed in prior studies. Risk of a seizure within 72 hours was found to be less than 5% if a seizure was not detected within 16 hours of monitoring.2,4 Future studies should explore the association between the time-dependent risk for seizures under continued observation in relation to the 2HELPS2B score. No cEEG sessions of less than 6 hours were included in this study; hence, these criteria should be applied with caution to studies of less than 6 hours. However, a reasonable approach for use of the 2HELPS2B score would be to calculate the score at the initial reading of the cEEG, typically within the first half hour of recording (>68% of EEG abnormalities are evident by this time).2 If new EEG findings emerge, the 2HELPS2B score should be modified at the second reporting, typically on the order of 6 to 8 hours. By this time, 95% of epileptiform abnormalities have been detected.2 Initially, 2HELPS2B can serve as a tool to augment clinical judgment regarding duration of monitoring and need for antiseizure medications. We anticipate future clinical studies using 2HELPS2B as a risk-stratifying metric to define rigorous cut points guiding clinical management, similar to the way the CHADS2 score guides anticoagulation in atrial fibrillation.9
The 2HELPS2B score is an easy-to-use tool to augment clinical judgment of the risk for seizures in individual patients. The simplicity of the system allows for easy integration into clinical workflow. With increasing familiarity, 2HELPS2B will improve communication between EEG interpreters and clinicians through the use of a quickly comprehensible single number to describe seizure risk for patients on cEEG.
Corresponding Author: Aaron F. Struck, MD, Department of Neurology, University of Wisconsin, 7131 MFCB, 1685 Highland Ave, Madison, WI 53705 (firstname.lastname@example.org).
Accepted for Publication: June 14, 2017.
Published Online: October 9, 2017. doi:10.1001/jamaneurol.2017.2459
Author Contributions: Drs Struck and Westover had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Struck and Ustun share first authorship.
Study concept and design: Struck, Ustun, LaRoche, Hirsch, Gilmore, Westover.
Acquisition, analysis, or interpretation of data: Struck, Ustun, Rodriguez Ruiz, Lee, LaRoche, Hirsch, Vlachy, Haider, Rudin, Westover.
Drafting of the manuscript: Struck, Ustun, Rodriguez Ruiz, Westover.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Struck, Ustun, Rodriguez Ruiz, Vlachy, Rudin, Westover.
Administrative, technical, or material support: Lee, LaRoche, Vlachy, Westover.
Study supervision: Struck, LaRoche, Hirsch, Westover.
Conflict of Interest Disclosures: Dr Rodriguez Ruiz has received funding from Neuropace to attend an educational conference on neurostimulation, Optima Neuroscience INC, with the project funded by grant 2R44NS064647-05A1 from the National Institute of Neurological Disorders (research funding). Dr Lee receives funding from the National Institute of Neurological Disorders and Stroke (NINDS), performs contract work for SleepMed/DigiTrace and Advance Medical, and is a consultant for Lundbeck. Dr LaRoche receives royalties from Demos Publishing. Dr Hirsch has received research support to Yale for investigator-initiated studies from Upsher-Smith, Lundbeck, Eisai, Sunovion, and Acorda; consultation fees for advising from Upsher-Smith, Neuropace, Marinus, Monteris, Sunovion, and Ceribell; royalties for authoring chapters for UpToDate–Neurology, chapters for Medlink-Neurology, and from Wiley for coauthoring the book Atlas of EEG in Critical Care. Dr Hirsch spends about 25% of his clinical billable time implementing and interpreting critical care EEG studies. Dr Gilmore is supported by Yale Center for Clinical Investigaton CTSA grant ULTR000142, Yale’s Claude D. Pepper Older Americans Independence Center (P30AG021342 grant from the National Institutes of Health (NIH)/National Institute of Aging), American Brain Foundation, and the NIH (Loan Repayment Program). She also receives royalties from Jaypee Brothers Medical Publishers. Dr Rudin receives funding from the National Science Foundation, Defense Advanced Research Project Agency. Dr Westover receives funding from NIH-NINDS grant K23 NS090900, the Rappaport Foundation, and the Andrew David Heitman Neuroendovascular Research Fund. No other disclosures were reported.
Funding/Support: This study was supported by a Research Infrastructure Award from the American Epilepsy Society and the Epilepsy Foundation.
Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Create a personal account or sign in to: