Simplified Global Registry of Acute Coronary Events mortality model. P = .77, Hosmer-Lemeshow goodness-of-fit test (c statistic, 0.83). Killip class is described in the "End Point and Clinical Definitions" subsection of the "Methods" section. CI indicates confidence interval; SBP, systolic blood pressure. To convert serum creatinine level to micromoles per liter, multiply by 88.4. Horizontal lines through black bars signify 95% confidence interval.
Calibration of simplified Global Registry of Acute Coronary Events mortality model, in derived data set. The diagonal line indicates perfect calibration.
Calibration of Global Registry of Acute Coronary Events simplified mortality model, in the Global Use of Strategies to Open Occluded Coronary Arteries IIb data set. The diagonal line indicates perfect calibration.
Global Registry of Acute Coronary Events risk model nomogram. Killip class is described in the "End Point and Clinical Definitions" subsection of the "Methods" section. SBP indicates systolic blood pressure. To convert serum creatinine level to micromoles per liter, multiply by 88.4.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Granger CB, Goldberg RJ, Dabbous O, et al. Predictors of Hospital Mortality in the Global Registry of Acute Coronary Events. Arch Intern Med. 2003;163(19):2345–2353. doi:10.1001/archinte.163.19.2345
Copyright 2003 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2003
Management of acute coronary syndromes (ACS) should be guided by an estimate of patient risk.
To develop a simple model to assess the risk for in-hospital mortality for the entire spectrum of ACS treated in general clinical practice.
A multivariable logistic regression model was developed using 11 389 patients (including 509 in-hospital deaths) with ACS with and without ST-segment elevation enrolled in the Global Registry of Acute Coronary Events (GRACE) from April 1, 1999, through March 31, 2001. Validation data sets included a subsequent cohort of 3972 patients enrolled in GRACE and 12 142 in the Global Use of Strategies to Open Occluded Coronary Arteries IIb (GUSTO-IIb) trial.
The following 8 independent risk factors accounted for 89.9% of the prognostic information: age (odds ratio [OR], 1.7 per 10 years), Killip class (OR, 2.0 per class), systolic blood pressure (OR, 1.4 per 20-mm Hg decrease), ST-segment deviation (OR, 2.4), cardiac arrest during presentation (OR, 4.3), serum creatinine level (OR, 1.2 per 1-mg/dL [88.4-µmol/L] increase), positive initial cardiac enzyme findings (OR, 1.6), and heart rate (OR, 1.3 per 30-beat/min increase). The discrimination ability of the simplified model was excellent with c statistics of 0.83 in the derived database, 0.84 in the confirmation GRACE data set, and 0.79 in the GUSTO-IIb database.
Across the entire spectrum of ACS and in general clinical practice, this model provides excellent ability to assess the risk for death and can be used as a simple nomogram to estimate risk in individual patients.
ACUTE CORONARY syndromes (ACS) represent the most common cause of death in the western world.1,2 Considerable variability exists in the risk for adverse events across the spectrum of ACS. Different presenting characteristics, in large part related to identification of varying levels of risk, have become important factors in deciding on the level of care and choice of interventional and medical therapies.3-5 Current guidelines from the American College of Cardiology/American Heart Association3,4 and the European Society of Cardiology6 recommend that certain pharmacological and interventional strategies are most appropriate for higher-risk groups. Despite these recommendations, some contemporary registry data suggest that more aggressive therapy is not necessarily targeted at higher-risk patients, even in more cost-constrained health care systems.7 Although individual demographic and clinical characteristics may be associated with an increased risk for adverse outcomes, one must take multiple factors into account simultaneously to optimize the ability to assess risk accurately.
A number of multivariable prognostic models have been developed in populations of patients with ST-segment elevation acute myocardial infarction8-11 and with ACS without ST-segment elevation.5,12-14 Most of these models have been derived from databases from clinical trials, which tend to exclude high-risk patients and are not fully representative of the broad spectrum of patients with ACS encountered in general clinical practice. Other predictive models have been developed using large claims databases, which may be limited by including only the elderly Medicare population, by including descriptors beyond the time of hospital presentation, and by depending on the coding vagaries of the International Classification of Diseases, Ninth Revision.11,15 Some of the most robust predictors of mortality have been developed in the selected population of patients with ST-segment elevation myocardial infarction treated with fibrinolytic therapy,8,9 and these models may not be relevant to most patients seen in practice. To determine factors that are predictive of death across the entire spectrum of an unselected population of ACS patients, we developed a multivariable prognostic model for hospital mortality in the multinational, observational Global Registry of Acute Coronary Events (GRACE). The goals in developing this model were 2-fold. First, a full model was developed to help refine knowledge about which variables should be collected to predict risk in future studies and to adjust for differences in risk factors for examination of nonrandomized comparisons. Second, a simplified model was developed to enable practical and accurate prediction of in-hospital mortality in individual patients.
The GRACE design and methods have been previously published.16 Briefly, GRACE is designed to reflect an unbiased and generalizable sample of ACS patients within 18 geographic locations. At present, 94 hospitals located in 14 countries (Argentina, Australia, Austria, Belgium, Brazil, Canada, France, Germany, Italy, New Zealand, Poland, Spain, the United Kingdom, and the United States) are participating in this observational study.
Following a similar approach to that adopted in the MONICA Project,17,18 all acute care hospitals in well-defined geographic areas were recruited to participate in the study. Because patients who experience emergency symptoms associated with ACS are likely to seek care at nearby hospitals, careful sampling of qualified cases of ACS from these hospitals should result in a generalizable picture of the types of hospital care and outcomes experienced by ACS patients from the local community. A population-based approach to the study of ACS has been adopted at a number of participating sites16 in which ACS patients from a geographically defined catchment area are included. Our method has led to the selection of community and tertiary hospitals of varying size and capability, which were representative of the capabilities of acute care hospitals in the study region.
To facilitate the review of medical records in a systematic manner and to accommodate the varying ways in which the data were collected, prospective and retrospective surveillance approaches for identifying cases of ACS, similar to those used in the MONICA Project,17,18 have been used. Patients enrolled had to be at least 18 years of age, be admitted to participating hospitals with symptoms consistent with acute ischemia, and have at least 1 of the following: electrocardiographic changes consistent with ACS, serial increases in serum markers of cardiac necrosis, and/or documentation of coronary artery disease.16 The qualifying ACS must not have been precipitated or accompanied by a significant comorbidity, trauma, or surgery. Patients transferred from a registry to a nonregistry hospital were enrolled if they were at the registry hospital for at least 48 hours.
Where required, study investigators received approval from their local hospital ethics or institutional review board, and a signed consent form for follow-up contact was obtained. For those sites using active surveillance for case identification, verbal or written consent was obtained from patients to review information contained in their medical charts. Patients who died within the first 24 hours of their index hospitalization tended to be excluded from study consideration at the sites where prospective case ascertainment was carried out.
Data were collected at each site by a trained coordinator using a standardized 6-page case report form. Demographic characteristics, medical history, presenting signs and symptoms, biochemical and electrocardiographic findings, medication, cardiac procedures, and a variety of hospital outcome data were collected. Standardized definitions of all patient-related variables and clinical diagnoses were used.16
Death was defined as all-cause mortality during hospitalization. Vital signs and Killip class findings were collected at the time of hospital presentation. Killip class I was defined as the absence of congestive heart failure, class II as the presence of rales and/or jugular venous distention, class III as the presence of pulmonary edema, and class IV as cardiogenic shock. Electrocardiograms were read locally and noted to record ST-segment elevation or ST depression in anterior, inferior, or lateral lead groups of at least 1 mm, Q waves one third the height of the R wave or greater than 0.04 second, or left bundle branch block. Prior medical conditions were categorized based on assessment by the patient's physician. Major categories of long-term medications were also collected.
The distributions of continuous variables were described using medians and 25th and 75th percentiles, and discrete variables were presented as frequencies and percentages. Odds ratios (ORs) and 95% confidence intervals (CIs) were also used to illustrate the association between potential prognostic factors and in-hospital death. Candidate variables (Table 1) for our predictive models were selected from clinical variables based on published model results from other studies and on clinical expert opinion. A logistic regression model was used to examine the individual relationship between each variable and in-hospital death. A multivariable stepwise logistic regression (backward elimination) approach was used to estimate the probability of in-hospital death. Variables that achieved a significance level of α≤.25 were eligible to enter the stepwise multiple logistic analysis. Only those variables associated with α≤.05 were retained in the final model.
Separate logistic regression models were developed for each continuous variable to test for a linear relationship with the outcome. If the relationship was not linear, then a transformation of the variable was performed using a fractional polynomial approach. Selected testing was performed for interactions using the significant prognostic variables from the final model based on interactions that have been reported from other published models.
The goodness of fit of the final regression model was evaluated using the Hosmer-Lemeshow goodness-of-fit test. The discriminative power of the final model was assessed by the mean of the area under the receiver operating characteristic curve (c statistic). Accuracy of calibration was evaluated by plotting the predicted vs observed mortality according to population deciles of predicted risk.
For most variables, less than 1% of the data were missing. However, data were missing in 6.6% of the study population for creatinine levels and in 27.9% for height. A multiple imputation technique based on Markov chain Monte Carlo approach19 was used to estimate the missing values for each of the missing data points in the study, including death. The primary model included imputed variables, with a second model that excluded patients with missing variables to determine consistency of general findings with each approach. A reduced model was created by taking only the variables with the most prognostic significance and refitting these to maximize discrimination. This model was used to develop a nomogram of patient risk.20 Internal validity was evaluated by means of bootstrap techniques.21 We assessed the external validity of our model by prospectively testing it in a subsequent, independent cohort of patients enrolled in GRACE and in the Global Use of Strategies to Open Occluded Coronary Arteries IIb (GUSTO-IIb) trial database, which included patients with the entire spectrum of ACS from unstable angina to ST-segment elevation myocardial infarction. The analysis was performed with SAS22 and S-Plus software.23
A total of 13 708 patients were enrolled in GRACE from April 1, 1999, through March 31, 2001. Patients with nonacute coronary syndromes and a noncardiac diagnosis at the time of hospital admission (898 patients) and those who were transferred into a GRACE enrollment hospital (1421 patients) were excluded. Our study population consisted of the 11 389 ACS patients. In-hospital mortality status was available in 98.1% of these individuals. There were 509 in-hospital deaths (4.6%) in the study sample. The median time of death was 4 days after hospital presentation, with an earlier time of death (3 days) among patients with ST-segment elevation acute myocardial infarction than among patients without ST-segment elevation (6 days). Nearly 22% (21.9%) of deaths occurred within 24 hours of hospital admission.
Demographic characteristics, medical history, and presenting clinical features are shown in Table 1 for patients who died, and for the overall population as the reference. Approximately one third of the population (35.3%) presented with ST-segment elevation; 15.2% received reperfusion therapy; and 31.6% had positive cardiac marker findings at presentation. Only 1.5% had resuscitated cardiac arrest, and 1.0% presented in cardiogenic shock (Killip class IV).
The associations with death are shown for each categorical baseline characteristic in Table 2 and for continuous variables in Table 3. The multivariable model results using the imputed data are shown in Table 4. The c statistic for this model was 0.84, indicating excellent discrimination. In the multivariable model, Killip class was the most powerful predictor, with a 2-fold increased risk for death with each worsening of class. Age had nearly the same prognostic significance, with a 1.7-fold increased risk for every 10 years. Systolic blood pressure was the next most important variable, followed by resuscitated cardiac arrest and initial serum creatinine level, in which a 1-mg/dL (88.4-µmol/L) increase was associated with a 1.2-fold increased risk for death. Univariable predictors that were not statistically significant multivariable predictors included sex and history of heart failure and renal insufficiency. When preadmission medication use was included in the model, all of the variables in the model without prior medication use remained significant. In addition, prior aspirin (OR, 0.73; 95% CI, 0.58-0.91) and statin (OR, 0.50; 95% CI, 0.34-0.97) use was each independently associated with lower risk for death. However, their inclusion resulted in only a small improvement in the c statistic to 0.85.
The model was rerun without imputation, excluding patients with missing variables. The main results were unchanged, with a c statistic that was similar to the main model at 0.84.
To develop a model that can be used in clinical practice, the overall model was reduced to include the most important variables that contain most predictive information (Figure 1). The c statistic of this simplified model is 0.83, or nearly the same as for the overall model.
Calibration of predictions from the simplified model was excellent as assessed by comparison of average predictions to the actual mortality across deciles of risk as shown in Figure 2. The simplified model performed well in all major subgroups. The c statistics were similar for patients with (0.83) and without (0.82) ST-segment elevation at hospital presentation, with (0.81) and without (0.83) elevated cardiac markers at presentation, and 65 years or younger (0.78) vs older than 65 years (0.82).
Results of internal validation revealed no "over-optimism" in the predictive discrimination of the simplified model (c index, 0.83), with the c index remaining unchanged at 0.83 with bootstrap techniques. External validation was performed on a subsequent sample of 3972 patients from the ongoing GRACE who were enrolled after March 21, 2001, with 215 deaths in this data set. The model performance was excellent, with a c index of 0.85. The model likewise performed well in the GUSTO-IIb data set of 12 142 patients with the full spectrum of ACS. Excellent discrimination of our model was reflected by a c index of 0.79, despite the fact that one of the predictors (cardiac arrest) had not been recorded in that study. The overall calibration was excellent (Figure 3). The model performed well in the ST-segment elevation (c index, 0.77) and the non–ST-segment elevation subgroups (c index, 0.81).
A nomogram based on the reduced model is presented in Figure 4 that can be used to calculate a prognostic score and to estimate the risk for death in individual patients.
This study, using data from a large multinational registry of ACS patients, confirms the prognostic importance of several baseline characteristics reported from previous models developed from other databases.8,9 The most important 8 factors—Killip class, age, blood pressure, resuscitated cardiac arrest, positive findings for cardiac markers, creatinine level, ST-segment shift, and heart rate—contained most of the prognostic information. Although we commonly categorize ACS according to the presence or the absence of ST-segment elevation at the time of presentation, this variable does not appear to be important for determining the risk for death after accounting for the presence of ST-segment deviation. Compared with models derived from clinical trial data sets, factors that appear to be more important in this registry of patients seen in general practice were renal dysfunction and resuscitated cardiac arrest. These 2 variables were among the 7 variables found to be most prognostically important in a Medicare database of patients 65 years and older with acute myocardial infarction.11 The GRACE predictive model has therefore incorporated the most important baseline variables from the large clinical trial data sets, in which clearly defined variables are identified at presentation, with variables previously identified as predictive in an unselected US elderly population.
Accurate determination of risk has become a major focus in the initial evaluation of ACS.3 Risk stratification is important to make appropriate decisions about the need for transfer to a tertiary care center, level of care, and length of stay and about which pharmacological and interventional treatments should be used. Patients at highest risk for adverse outcomes may derive greater absolute benefit from the hospital use of effective treatments, and the benefit may be more likely to outweigh risk inherent to certain treatments such as fibrinolytic therapy.24 Although a clinician can stratify risk in a general way based on patient age, hemodynamics, and electrocardiographic and cardiac marker findings, integration of the multiple sources of risk information is not possible without support from multivariable models. In practice, these must be in the form of a simple nomogram or in a format that can be incorporated into handheld computers. We have entered a period in which many clinicians have personal digital assistants, making the use of more sophisticated models practical for the first time. Whether practical, more accurate risk stratification tools can have an impact on patient care requires further study.
Robust and comprehensive mortality risk models have been developed from the GUSTO I8 and Intravenous nPA for Treatment of Infarcting Myocardium Early II (InTIME II) databases9 in patients with ST-segment elevation acute myocardial infarction. The GRACE model differs in the following 3 important ways from previously published work: it is the first mortality model to span the entire spectrum of ACS; it is based on a relatively unselected patient population representing those seen in general practice; and it incorporates new variables that add considerable predictive information. Moreover, the GRACE model has excellent ability to discriminate risk as reflected by the c statistic of 0.84.
The continuous nature of ACS has been stressed in recent years.3,25 The ACS patient is not only difficult to categorize at the time of presentation as having myocardial infarction or unstable angina but may have varying amounts of dynamic ST-segment shift, and may rapidly progress from one category to another. It is also clear that patients who receive fibrinolytic therapy make up a relatively small proportion of the population with acute myocardial infarction, and that the highest-risk patients may not be eligible for fibrinolytic therapy.26 When the model from the InTIME II fibrinolytic population was tested in a general population of patients with acute myocardial infarction, the c statistic was good for patients receiving reperfusion therapy at 0.79, but was only 0.65 among patients who did not receive reperfusion therapy.27 A risk prediction model with a c index of 0.6 to 0.7 has been suggested to be of limited clinical value.28 This highlights the need for more generalizable models that apply to a broad patient population. Therefore, as long as a single predictive instrument performed well, a major advantage to a single risk model for the entire spectrum of ACS would exist. GRACE has been developed to perform equally well with and without presenting ST-segment elevation and therefore can be used in any patient presenting with ACS.
Although the large clinical trial data sets have been the most complete and high-quality data sets with which to develop predictive regression models, even the large, simple trials are subject to substantial selection bias as to who is enrolled. Patients enrolled in clinical trials have lower risk features and better outcomes than contemporaneous patients who are not eligible for enrollment.26 Patients enrolled in fibrinolytic therapy trials have been selected based on explicit exclusion criteria and to avoid enrolling patients at high risk for complications. Therefore, factors associated with an unfavorable outcome and optimal risk prediction might differ substantially in an unselected population of hospitalized patients. The GRACE population is modestly older than clinical trial populations used to create prior mortality models. The GRACE model shares many variables with prior mortality models (Table 5). However, heart failure at presentation, expressed as Killip class, constituted a greater proportion of the predictive information in this model than in prior clinical trial models. This may be partially explained by the fact that these patients were not excluded from the GRACE database. Other variables that may be more relevant in the general population of ACS patients and that were independently predictive in GRACE were creatinine level and resuscitated cardiac arrest. Creatinine level is a variable that relates to more direct measures of renal function such as creatinine clearance according to age, sex, and body weight. We found better discrimination, however, in using the creatinine level itself rather than a calculated creatinine clearance variable. It is not surprising that renal function, which has been found to be an important predictor of mortality in claims data analysis of elderly patients with acute myocardial infarction,11 and in other cardiac conditions,31 is an important contributor to risk assessment in ACS. Initial creatinine concentrations, which have received relatively little attention in the published literature as a prognostic marker until recently,32 contained more prognostic information than elevation of cardiac markers at presentation. Because information about creatinine may not have been systematically collected in clinical trials,29,30 or because patients with renal dysfunction may be excluded,25 this variable has received less attention in these models. Likewise, we have confirmed that patients with resuscitated cardiac arrest, which occurred in 1.5% of our population, are at especially high risk.
Although models from large data sets have tended to perform reasonably well when applied to independent data sets,11,27 validation of regression models using internal and/or external methods is important. The simplified GRACE model appears to perform well in 2 independent data sets. These included a subsequent GRACE population and a clinical trial data set that was unique in including the entire spectrum of ACS and that was collecting information on baseline creatinine concentrations.
Important features of predictive models include accuracy, generalizability, and ease of use.28 Building on and extending prior models, our single model, which has evolved from an unselected population of ACS patients, includes new predictive variables and is easy to use in its simplified form.
Although GRACE has been designed to capture an unselected and representative patient population, some participating centers are required to obtain informed consent from patients before enrollment. Therefore, early deaths and patients with early clinical complications may be underrepresented. However, because 21.9% of deaths occurred during the first 24 hours, this does not appear to be a major factor. Differences in case ascertainment approach, hospital length of stay, and patient characteristics between centers could result in different performance of the model according to center, although the model validated well in 2 independent patient cohorts. This model may not be effective at stratifying the very low-risk population that was not included in this registry, but such patients with chest pain do not fulfill the criteria for ACS. There may be unmeasured variables that would have provided further prognostic and longer-term information,33 in particular troponin levels34 and markers of inflammation,14,35-37 which have been shown to be promising as specific independent markers. The aim of this model, however, was to provide insight into factors associated with increased risk for in-hospital death. The goal of achieving simplicity and ease of use must be balanced against completeness and accuracy. With little consensus on how to compare model performance, small decrements in discriminative significance and goodness of fit may reflect meaningful decrements in clinical value. Although risk stratification at the time of hospital presentation is of value, risk stratification ideally should be a dynamic, iterative process that is continuously updated depending on changes in the patient's clinical course.
A few variables have been shown to be consistent, powerful predictors of risk for death in ACS. This study shows that a single model can discriminate risk for the entire spectrum of ACS in a general practice population. The model requires identification of evolving infarction at clinical presentation. Two variables not previously identified from clinical trials databases, baseline creatinine level and cardiac arrest during presentation, are important factors in this broadly representative population. This information can help clinicians stratify risk for optimal triage and management.
Corresponding author and reprints: Christopher B. Granger, MD, Duke Clinical Research Institute, 2400 Pratt St, Room 0311, Terrace Level, Durham, NC 27705.
Accepted for publication December 13, 2002.
This research was funded by an unrestricted grant from Aventis Pharma, Bridgewater, NJ.
This research was presented in abstract form at the European Society of Cardiology Congress 2002, Berlin, Germany, September 2, 2002.
We thank the physicians and nurses participating in the GRACE research, whose continuing involvement is essential to the success of the study. Further information about the project, along with the complete list of participants, can be found at http://www.outcomes.org/grace.
Create a personal account or sign in to: