The COVID-GRAM Tool for Patients Hospitalized With COVID-19 in Europe | Clinical Decision Support | JAMA Internal Medicine | JAMA Network
[Skip to Navigation]
Table.  Observed vs Predicted Risk of Critical Illness Based on COVID-GRAM Score by Quintile of Predicted Risk Among 214 Patients Hospitalized With COVID-19 in Spain
Observed vs Predicted Risk of Critical Illness Based on COVID-GRAM Score by Quintile of Predicted Risk Among 214 Patients Hospitalized With COVID-19 in Spain
1.
Liang  W, Liang  H, Ou  L,  et al; China Medical Treatment Expert Group for COVID-19.  Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.   JAMA Intern Med. 2020;180(8):1081-1089. doi:10.1001/jamainternmed.2020.2033PubMedGoogle ScholarCrossref
2.
Steinberg  E, Balakrishna  A, Habboushe  J, Shawl  A, Lee  J.  Calculated decisions: COVID-19 calculators during extreme resource-limited situations.   Emerg Med Pract. 2020;22(4)(suppl):CD1-CD5.PubMedGoogle Scholar
3.
Metlay  JP, Waterer  GW, Long  AC,  et al.  Diagnosis and treatment of adults with community-acquired pneumonia: an official clinical practice guideline of the American Thoracic Society and Infectious Diseases Society of America.   Am J Respir Crit Care Med. 2019;200(7):e45-e67. doi:10.1164/rccm.201908-1581STPubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Research Letter
    April 5, 2021

    The COVID-GRAM Tool for Patients Hospitalized With COVID-19 in Europe

    Author Affiliations
    • 1Endocrinology and Nutrition Department, Alicante General University Hospital–Alicante Institute of Sanitary and Biomedical Research (ISABIAL), Alicante, Spain
    • 2Clinical Medicine Department, Miguel Hernández University, Alicante, Spain
    • 3Rheumatology Department, Alicante General University Hospital–Alicante Institute of Sanitary and Biomedical Research (ISABIAL), Alicante, Spain
    • 4Pneumology Department, Alicante General University Hospital–Alicante Institute of Sanitary and Biomedical Research (ISABIAL), Alicante, Spain
    • 5Preventive Department, Alicante General University Hospital–Alicante Institute of Sanitary and Biomedical Research (ISABIAL), Alicante, Spain
    • 6Unit of Infectious Diseases, Alicante General University Hospital–Alicante Institute of Health and Biomedical Research (ISABIAL), Alicante, Spain
    • 7Clinical Medicine Department, Miguel Hernández University, Elche, Spain
    JAMA Intern Med. 2021;181(7):1000-1001. doi:10.1001/jamainternmed.2021.0491

    Liang and colleagues1 recently validated a clinical risk tool (the COVID-GRAM) to predict the development of critical COVID-19 illness—defined as admission to the intensive care unit (ICU), requiring invasive mechanical ventilation, or death—after hospitalization admission in a nationwide cohort in China. Risk scores, applied to 10 variables that were independent predictors of critical illness, were used to classify patients as having a low (0.7% probability), medium (7.3%), or high risk (59.3%) of developing a critical illness.

    Accurate risk-predicting tools are imperative for managing the COVID-19 pandemic with limited health resources.2 High COVID-GRAM scores at presentation could warrant increased vigilance and treatment, while low scores could require only observation. The COVID-GRAM was developed among patients with a 1.5% incidence of severe pneumonia, as defined by the American Thoracic Society,3 and an 8.2% incidence of critical illness. Thus, the COVID-GRAM score should be replicated and validated for use in other clinical populations.

    Methods

    We retrospectively applied the COVID-GRAM tool to a cohort of patients with COVID-19 who were hospitalized from March 3 to May 2, 2020, in Alicante, Spain—a country with one of the most extensive outbreaks of SARS-CoV-2 in Europe. From the total cohort, we selected patients who (1) were eligible for intensive care and invasive mechanical ventilation, if needed (a major role of the COVID-GRAM would be to intensify treatment in patients at high risk of critical illness) and (2) had complete data for calculating their COVID-GRAM score. Similar to the outcome for the study that validated the COVID-GRAM tool in China,1 we defined critical illness as composed of admission to the ICU, invasive mechanical ventilation, or death.

    To validate the COVID-GRAM tool among the cohort in Spain, we classified patients by quintile of predicted risk for critical illness based on their COVID-GRAM score and compared the observed outcomes with the predicted outcomes using the χ2 test. Accuracy in the sample was tested by measuring the area under the receiver operating characteristic curve (AUC). Data analyses were performed from May to June 2020 using SPSS, version 26 (IBM Inc). P values were 2-tailed, and statistical significance was defined as P < .05.

    The research was conducted according to the principles of the Declaration of Helsinki. The ethics committee of the Alicante General University Hospital–Alicante Institute of Sanitary and Biomedical Research approved the study (expedient No. 200145) and because it was a retrospective study, informed consent was waived.

    Results

    Of the cohort of 306 patients hospitalized with COVID-19, intensive care was required for 236 (77.1%) patients of whom 214 (median [interquartile range] age, 60.5 [48.0-70.7] years; 86 [40.1%] women) had complete data. Patients with incomplete data were excluded but were similar to the study cohort in age, gender, degree of comorbidity (Charlson Comorbidity Index score), arterial hypertension, diabetes, obesity, extent of radiological involvement, use of tocilizumab, ICU admission, and need for invasive mechanical ventilation. Compared with the validation cohort in China, this study’s population was older (median age, 60.5 years vs 48.2 years) and had more comorbidities (54.6% vs 24.2% with ≥1 coexisting condition). With a median (interquartile range) follow-up of 43 (33-48) days, critical illness developed in 52 (21.8%) patients (40 admitted to the ICU, 35 required invasive mechanical ventilation, and 13 died) vs 12.3% in the Chinese validation cohort.

    The Table shows the distribution of observed vs predicted risk based on the COVID-GRAM score by quintile of predicted risk. The COVID-GRAM predictions were similar to the observed outcomes among patients in the first 4 quintiles of risk for critical illness, but it overestimated the predicted risk of events in the highest quintile by almost 2-fold. The accuracy of the COVID-GRAM in the cohort was moderate, with an AUC of 0.72 (95% CI, 0.64-0.80) compared with an AUC of 0.88 (95% CI, 0.84-0.93) in the Chinese validation cohort.1 A score of 89 or higher showed a sensitivity of 97.7% and a specificity of 32.7% for development of critical illness.

    Discussion

    We were unable to fully validate the COVID-GRAM tool for predicting critical illness among patients hospitalized with COVID-19 in Europe because, although the tool showed good predictive ability for critical illness in lower-risk patients, it overestimated risk in the highest-risk patients. The study may have been limited by differences in patient age and comorbidities, disease severity, and other variations between the cohorts in China and Spain. Still, these findings reflect that caution is needed when applying risk prediction tools in new populations.

    Back to top
    Article Information

    Accepted for Publication: January 30, 2021.

    Published Online: April 5, 2021. doi:10.1001/jamainternmed.2021.0491

    Corresponding Author: Óscar Moreno-Pérez, MD, PhD, Endocrinology and Nutrition Department, Alicante General University Hospital–Alicante Institute of Sanitary and Biomedical Research (ISABIAL), C/Pintor Baeza S/N, 03010 Alicante, Spain (omoreno@umh.es).

    Author Contributions: Dr Moreno-Pérez had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Moreno-Pérez, Andrés, León-Ramirez, Sánchez-Payá, Boix, Merino.

    Acquisition, analysis, or interpretation of data: Moreno-Pérez, Andrés, León-Ramirez, Sánchez-Payá, Gil, Merino.

    Drafting of the manuscript: Moreno-Pérez, Andrés, León-Ramirez, Sánchez-Payá, Gil, Merino.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Moreno-Pérez, Andrés, Sánchez-Payá.

    Supervision: León-Ramirez, Boix, Gil, Merino.

    Conflict of Interest Disclosures: Dr Moreno-Pérez reported consulting and speaking fees from AstraZeneca, Boehringer Ingelheim, Lilly, Merck Sharp & Dohme, and Novo Nordisk, all outside the submitted work. No other disclosures were reported.

    References
    1.
    Liang  W, Liang  H, Ou  L,  et al; China Medical Treatment Expert Group for COVID-19.  Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.   JAMA Intern Med. 2020;180(8):1081-1089. doi:10.1001/jamainternmed.2020.2033PubMedGoogle ScholarCrossref
    2.
    Steinberg  E, Balakrishna  A, Habboushe  J, Shawl  A, Lee  J.  Calculated decisions: COVID-19 calculators during extreme resource-limited situations.   Emerg Med Pract. 2020;22(4)(suppl):CD1-CD5.PubMedGoogle Scholar
    3.
    Metlay  JP, Waterer  GW, Long  AC,  et al.  Diagnosis and treatment of adults with community-acquired pneumonia: an official clinical practice guideline of the American Thoracic Society and Infectious Diseases Society of America.   Am J Respir Crit Care Med. 2019;200(7):e45-e67. doi:10.1164/rccm.201908-1581STPubMedGoogle ScholarCrossref
    ×