eTable 1. Baseline Characteristics of the Study Population and Patients Lost to Follow-up
eTable 2. Cumulative Recurrence Rates According to the Clinical RRE Model
Customize your JAMA Network experience by selecting one or more topics from the list below.
Arsava EM, Kim G, Oliveira-Filho J, et al. Prediction of Early Recurrence After Acute Ischemic Stroke. JAMA Neurol. 2016;73(4):396–401. doi:10.1001/jamaneurol.2015.4949
Approximately half of recurrent strokes occur within days and weeks of an ischemic stroke. It is imperative to identify patients at imminent risk of recurrent stroke because recurrent events lead to prolonged hospitalization, worsened functional outcome, and increased mortality.
To test the validity of a prognostic score that was exclusively developed to predict early risk of stroke recurrence in a multicenter setting.
Design, Setting, and Participants
This hospital-based cohort study examined patients with and without magnetic resonance imaging–confirmed recurrent stroke within 90 days after an ischemic stroke. The study was performed at 3 teaching hospitals in the United States, Brazil, and South Korea and comprised adult patients admitted within 72 hours of symptom onset with a magnetic resonance imaging–confirmed diagnosis of acute ischemic stroke. Recruitment to the US cohort was performed from June 1, 2009, through April 30, 2011. Recruitment to the Korean and Brazilian cohorts was performed from January 1, 2007, through December 31, 2011. Data analysis was performed from June 1, 2013, to December 31, 2014.
Main Outcomes and Measures
The primary outcome was recurrent ischemic stroke as defined by a clinical incident that was clearly attributable to a new area of brain infarction occurring within the 90 days of index infarction. An investigator who was masked to the patient’s recurrence status calculated the Recurrence Risk Estimator (RRE) score for each patient based on information available after initial line of testing in the emergency department. We assessed the predictive performance of the RRE by computing the area under the receiver operating characteristic curve.
The study included 1468 consecutive patients with 59 recurrent ischemic stroke events. The median age of the patients was 69 (interquartile range, 58-79) years, and 633 (43.1%) were female. The cumulative 90-day recurrence rate was 4.2% (95% CI, 3.2%-5.2%). The mean RRE score was 2.2 (95% CI, 1.9-2.5) in patients with recurrence and 1.0 (95% CI, 1.0-1.1) in patients without. The risk of recurrence increased with a higher RRE score (log-rank test, P < .001). The area under the receiver operating characteristic curve for discrimination was 0.76 (95% CI, 0.70-0.82). The RRE identified 710 patients (48.4%) in the study population as high risk (>10%) or low risk (<1%). The sensitivity and specificity were 38% and 93% for identifying low-risk subsets and 41% and 90% for identifying high-risk subsets, respectively.
Conclusions and Relevance
This study confirms the validity of the RRE score in a multicenter cohort of patients with diverse characteristics. Our findings suggest that the RRE could be useful in identifying high- and low-risk patients for targeted stroke prevention.
It is imperative to identify patients at high imminent risk of developing a subsequent stroke after an ischemic stroke who may benefit from streamlined evaluation and rapid institution of preventive treatments. Quiz Ref IDThe Recurrence Risk Estimator (RRE) is a web-based prognostic instrument that has been developed to predict the 90-day risk of recurrent stroke based on information typically available at the time of hospital admission (http://www.nmr.mgh.harvard.edu/RRE/).1Quiz Ref ID The RRE incorporates 4 brain imaging features with 2 clinical features of stroke and provides risk estimates ranging from approximately 1% to approximately 40%. In the original derivation cohort of 1257 patients with 54 recurrent events from a single tertiary care center, the discriminative value as measured by the area under the receiver operating characteristic curve (AUC) was 0.80 for a 90-day risk of recurrence.1 In the present study, we sought to assess the ability of the RRE to predict 90-day stroke recurrence in an independent population of patients with ischemic stroke.
Question: Is it possible to identify patients at imminent risk of recurrent stroke after an ischemic stroke?
Findings: This study tested the ability of the Recurrence Risk Estimator (RRE) score to predict the 90-day risk of recurrent stroke based on clinical history, presumed stroke cause, and brain imaging findings in a multicenter setting. The risk of recurrence increased with a higher RRE score. The RRE identified 710 patients (48.4%) in the study population as high risk (>10%) or low risk (<1%).
Meanings: These findings support the generalizability and validity of the RRE.
This study included 1 US cohort (Massachusetts General Hospital) and 2 cohorts from teaching hospitals in South Korea (Sungkyunkwan University School of Medicine) and Brazil (Federal University of Bahia). Partners Institutional Review Board, the institutional review board of the Federal University of Bahia, and the institutional review board of the Samsung Medical Center approved the study. Because this was a retrospective data collection study, no consent from patients was required. All data were deidentified before being delivered for analysis as per the institutional review board requirements.
The US cohort comprised consecutive adult patients (>18 years old) admitted to Massachusetts General Hospital within 72 hours of symptom onset with a magnetic resonance imaging (MRI)–confirmed diagnosis of acute ischemic stroke. The US cohort was different from the original derivation cohort used to develop the RRE score.1 Recruitment to the US cohort was performed within the context of a prospective National Institutes of Health–funded study (Heart-Brain Interactions Study) from June 1, 2009, through April 30, 2011. The Korean and Brazilian cohorts were retrospectively recruited from institutional registries and consisted of consecutive adult patients with MRI-confirmed ischemic stroke admitted within 72 hours of symptom onset from January 1, 2007, through December 31, 2011. Data analysis was performed from June 1, 2013, to December 31, 2014.
Each study site received a Microsoft Excel–based data collection form and a study guide that provided a detailed protocol for outcome definition, outcome assessment, and RRE score determination. Clinical and imaging predictors were collected by participating stroke neurologists based on the review of notes in the medical record and visual inspection of brain images. The following data were requested from each site: (1) baseline patient characteristics: age, sex, and vascular risk factors (hypertension, diabetes mellitus, hyperlipidemia, and atrial fibrillation); (2) index stroke characteristics: National Institutes of Health Stroke Scale score, time between symptom onset and brain imaging, antecedent stroke or transient ischemic attack within the prior month, etiologic stroke subtype determined by the Causative Classification System based on information available after baseline assessment in the emergency department,2 and preventive treatment (antiplatelet, anticoagulant or statin treatment, or vascular intervention); (3) imaging characteristics of index stroke: infarct location (cortical, subcortical, or both), multiple acute infarcts, simultaneous multiple acute infarcts in different cerebral circulations (right and left anterior or anterior and posterior circulations), and multiple infarcts of different ages (combination of acute, subacute, or chronic infarcts); and (4) outcome data: date of recurrent stroke and date of death.
The RRE score was calculated for each patient by an investigator at each study site (H.A., G.-M.K., and J.O.-F.) who was masked to the patient’s recurrence status using the clinical and imaging data available after baseline investigations on the day of admission. The RRE is a 7-point score composed of the following 6 predictors: prior transient ischemic attack or stroke within the preceding month (1 point), Causative Classification System subtype (1 point if the cause of stroke is large artery atherosclerosis or uncommon causes, such as vasculitis, arterial dissection, and prothrombotic disorders, and 0 points when stroke develops secondary to cardiac embolism, small artery occlusion, or undetermined causes), the presence of multiple acute infarcts (1 point), simultaneous acute infarcts in both hemispheres or in both anterior and posterior circulations (1 point), multiple infarcts of different ages (1 point for combination of acute and subacute infarcts), and isolated cortical location (1 point).1 We calculated the RRE score by summing the scores for each independent predictor for a given patient. We also calculated another score for each patient without including the imaging predictors as described previously (the clinical model).1
Outcome assessment was performed prospectively via in-person evaluation or telephone interviews at a mean (SD) of 90 days (n = 587) and retrospectively through inspection of registry data, inpatient medical record notes, and routine 3-month outpatient assessment notes (n = 881) by investigators (H.A., H.J.N., and M.d.J.L.) who were masked to the RRE scores. Physician notes included a detailed description and timing of the follow-up event. Survival status was confirmed using the Social Security Death Index in the US cohort. The primary outcome was recurrent stroke within 90 days of the index stroke. Recurrent stroke was defined as a clinical incident that was clearly attributable to a new area of brain infarct visualized by imaging as a spatially distinct lesion from the index infarct.1
Numerical variables were expressed as median (interquartile range [IQR]) or mean (95% CI). The Fisher exact test or χ2 test and the Mann-Whitney test were used, respectively, to compare categorical and continuous variables. Kaplan-Meier analysis was used to determine cumulative recurrence rates. Data were censored at the time of death or recurrent stroke. The log-rank test was used to examine the association between the RRE scores and cumulate recurrence rates. The discriminative ability of the RRE to predict 90-day recurrence was evaluated by computing the AUCs. Accuracy (sum of correct prediction divided by total predictions), sensitivity, specificity, and positive and negative predictive values for RRE strata of 0 (low risk) and 3 or greater (high risk) were calculated. Definition of high- and low-risk strata was based on risk distribution data in the original derivation study where scores 1 and 2 corresponded to risk estimates that were in the range of the population mean, whereas scores 0 and 3 to 6 indicated risks that clearly deviated from the population mean.1 The Hosmer-Lemeshow test was used to assess calibration.3 Sensitivity analyses were performed to calculate the predictive ability of the RRE in subsets of patients who presented within 24 hours of symptom onset and who did not develop a recurrent event secondary to a cardiac or vascular intervention. P < .05 was considered statistically significant. All statistical analyses were performed with SPSS statistical software, version 16.0 (SPSS, Inc).
Quiz Ref IDThe study population consisted of 888 patients from the United States and 726 patients from the Korean and Brazilian sites. A total of 146 patients were unavailable for follow-up; the final study population comprised the remaining 1468 patients. Table 1 presents baseline patient characteristics and clinical stroke features of the study population. Baseline stroke features did not differ between the study population and patients unavailable for follow-up (eTable 1 in the Supplement).
A total of 59 recurrent strokes occurred during the study period. The cumulative 90-day recurrence rate was 5.3% (95% CI, 3.7%-6.9%) in the US cohort, 3.0% (95% CI, 1.6%-4.4%) in Korean and Brazilian cohorts, and 4.2% (95% CI, 3.2%-5.2%) in the overall study cohort. A total of 137 patients died during the 90-day follow-up period. The median (IQR) time to death was 9 (IQR, 4-28) days. None of the deaths were associated with a recurrent stroke. Among patients with 90-day follow-up assessment, baseline stroke severity, vascular risk factors, and the use of antithrombotic treatments did not differ between patients with and without a recurrent stroke (Table 2). Patients who developed a recurrent stroke were more likely to have large artery atherosclerosis compared with those who did not develop a recurrent stroke (Table 2). The risk of recurrence increased with a higher RRE score (log-rank test, P < .001) (Table 3). The mean RRE score was 2.2 (95% CI, 1.9-2.5) in patients with recurrence and 1.0 (95% CI, 1.0-1.1) in patients without. The AUC for 90-day recurrence was 0.76 (95% CI, 0.68-0.84) in the US cohort, 0.75 (95% CI, 0.66-0.85) in Korean and Brazilian cohorts, and 0.76 (95% CI, 0.70-0.82) in the overall study cohort (Table 4). The discriminative ability of the RRE was similar in subsets of patients who underwent imaging within 24 hours of symptom onset (n = 944; AUC, 0.76; 95% CI, 0.69-0.84) and in whom the cause of recurrent stroke was not secondary to a vascular intervention (n = 1329; AUC, 0.76; 95% CI, 0.70-0.82). A time-epoch analysis revealed that the performance of the RRE did not change with respect to time: the AUC was 0.74 (95% CI, 0.64-0.84) in the cohort recruited during the first half and 0.78 (95% CI, 0.71-0.85) during the second half of the study (P = .46). The AUC for the clinical model was 0.65 (95% CI, 0.59-0.71) (eTable 2 in the Supplement).
The P value of the Hosmer-Lemeshow test was .008, suggesting imperfect calibration. The calibration slope was 0.61. The sensitivity, specificity, positive predictive value, negative predictive value, and accuracy for identifying the population at low risk (<1% risk) were 38%, 93%, 99%, 7%, and 41%, respectively. The corresponding performance characteristics for detection of a high-risk subset (>10% risk) were 41%, 90%, 16%, 97%, and 88%, respectively.
Prior studies1,4 have found that the RRE provides good discrimination for predicting a 14-day and 90-day risk of recurrence after an ischemic stroke and a 7-day risk of subsequent stroke after transient symptoms with infarction. Quiz Ref IDThe present study expands on prior studies1,4 by revealing that the discriminatory power of the RRE for predicting 90-day risk of recurrent stroke after an ischemic stroke is maintained when applied to a separate cohort of patients recruited from 3 academic centers with different practice patterns, supporting the generalizability and external validity of the RRE. Approximately half of the patients with stroke (710 [48.4%]) were assigned to either high-risk (>10%) or low-risk (<1%) categories with an accuracy that ranged from 41% to 88%. The RRE falsely classified only 4 patients (6.7%) with a recurrent stroke as being low risk. Good discrimination suggests that the RRE may be useful in tailoring stroke management based on baseline stroke risk. For instance, elective management of low-risk patients could be considered to ensure efficient use of health care resources. In contrast, high-risk patients could benefit from prompt evaluation and targeted preventive treatment in dedicated stroke centers.
The risk of recurrent stroke is highest immediately after an ischemic stroke, reaching a cumulative rate of 1.3% to 4.3% at 7 days,5,6 3.4% at 14 days,7 and 4.9% to 12.9% at 90 days8,9; thereafter, the risk gradually decreases and attains a steady state by 6 to 12 months after the incident stroke. Prior studies10-14 have revealed that there are different predictors for short- and long-term recurrence; cardiovascular risk factors, such as hypertension, diabetes, and hyperlipidemia, confer a risk in the long term. In contrast, underlying stroke cause poses a risk in the short and long term.15-17 Hence, prognostic scores that are solely based on the presence or absence of cardiovascular risk factors, such as the Framingham Risk Score, Stroke Prognosis Instrument II, and Essen Risk Score, provide imperfect estimates of the short-term risk.1,13,14,18 A distinctive feature of the RRE is that it has been exclusively developed to predict short-term risk. The RRE harmonizes etiologic stroke mechanism with other predictors of early stroke recurrence and identifies individuals in whom the underlying cause has the potential to cause another stroke in the short term (unstable cause).1,4 Although the RRE exhibited good discrimination, the AUC was 0.76, and the positive predictive value for the high-risk subset was 16%. Hence, there is much room for improvement. Future prognostic tools that incorporate more specific markers to assess the potential of underlying causes to initiate a stroke could provide risk predictions with higher accuracy.
The discriminative ability of the RRE was slightly lower in the present study (AUC, 0.76) than in the original derivation cohort (AUC, 0.80).1 Some deterioration in predictive ability is expected when prognostic scores are tested in independent or external data sets caused by overfitting in the original model, different selection of patients in the validation setting, changes in preventive treatments, and differences in definition of predictors and outcome.19 To our knowledge, there is only 1 published external validation study20 of the RRE, performed in a hospital-based retrospective population in Germany. In that study, the RRE predicted a 7-day of risk of stroke recurrence with a moderate AUC (0.65; 95% CI, 0.58-0.73). The use of Trial of ORG 10172 in Acute Stroke Treatment subtypes rather than Causative Classification System subtypes when calculating the RRE scores, absence of MRI data in more than one-third of the patients, and diagnosis of recurrent stroke without the need for imaging evidence might explain the lower AUC in that study. In addition, because of the inherent difficulty in differentiating early recurrence from worsening of the index stroke based on clinical criteria, the uncertainty in risk predictions tends to increase as the time from stroke onset to outcome assessment decreases.1,20 Hence, a lower AUC could be expected when predicting the 7-day risk than when predicting the 90-day risk.1
Our results are subject to certain limitations. The number of outcome events (n = 59) was less than desirable for a validation study.21 Variance introduced by the small validation sample might have played a role in both reduced discrimination and poor calibration in the present data set. Poor calibration may have also resulted from differences in case mix between the development and validation data sets and in interpretation of the definitions for the predictor and the outcome variables. Calibration of the RRE can be improved by recalibrating the score in larger external data sets in the future. The risk of stroke recurrence was different between the US and the other 2 cohorts. This difference could be partly attributable to differences in physicians’ judgment about obtaining neuroimaging to confirm a recurrent stroke in the event of new symptoms. Higher availability and easier access to MRI in the US cohort might have resulted in more frequent detection of recurrent stroke. Recruitment of patients at a later time point in external sites may have also contributed to the lower recurrence rate as a result of missing very early recurrent events. We also note that assessment of outcome based on retrospective review of notes from medical records might have led to underestimation of recurrent stroke events in some study sites. Follow-up information was not available for 146 patients. Nevertheless, individual predictors of recurrence and RRE scores were similar for those with and without follow-up information. Hence, it is unlikely that missing follow-up information has altered the predictive performance of the RRE. Mislabeling a fatal recurrent stroke as death could alter the predictive performance of the RRE. It is, however, unlikely that the present results were subject to a significant survival bias because none of the patients died as a result of a recurrent stroke.
For a score to be clinically useful in predicting early stroke risk, it is critical that the predictor variables are easily attainable immediately after stroke. Quiz Ref IDThe RRE is well suited to accomplish this goal because it provides risk estimates based on clinical history, presumed stroke cause, and brain imaging findings that are readily available to the physician immediately after hospital admission in most clinical settings. The present study reveals that the RRE retains its discriminative ability when applied to an independent population with discrete characteristics. Further studies are needed to assess the ability of the RRE to guide stroke evaluation and preventive treatment and improve the use of sparse health care resources in settings with more diverse practice patterns.
Accepted for Publication: December 16, 2015.
Corresponding Author: Hakan Ay, MD, A. A. Martinos Center for Biomedical Imaging and Stroke Service, Departments of Radiology and Neurology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Room 2301, Charlestown, MA 02129 (firstname.lastname@example.org).
Published Online: February 29, 2016. doi:10.1001/jamaneurol.2015.4949.
Author Contributions: Dr Ay had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Arsava, Kim, Ay.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Arsava, Kim, Ay.
Critical revision of the manuscript for important intellectual content: Kim, Oliveira-Filho, Gungor, Noh, de Jesus Lordelo, Avery, Maier, Ay.
Statistical analysis: Arsava, Kim, Ay.
Administrative, technical, or material support: Oliveira-Filho, Gungor, Noh, Avery, Maier, Ay.
Study supervision: Ay.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was supported by grant R01-NS059710 from the National Institutes of Health (Dr Ay).
Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.