Assessment of the Performance Consistency of an Adverse Outcome Prediction Tool for Patients Hospitalized With COVID-19

This prognostic study reports on the performance of a previously validated COVID-19 severity prediction tool when applied to data during the second wave of the pandemic.


Introduction
The challenge of managing limited resources during the COVID-19 pandemic has sparked efforts to stratify risk among hospitalized patients. 1Few risk models have been validated or investigated for potential bias 2 even though inpatient populations, treatments, and outcomes for COVID-19 have changed over time.We previously 3 reported and validated a risk prediction tool based on COVID-19 hospitalizations during the initial wave of the pandemic.In this study, we report the performance of that same model on subsequent data from 6 hospitals collected during the second wave of patients with COVID-19.

Methods
In this prognostic study, we included individuals aged 18 years or older who were hospitalized at 1 of 2 academic medical centers and 4 community hospitals from June 7, 2020, through January 22, 2021, with a positive polymerase chain reaction test for SARS-CoV-2 within 5 days of admission, excluding those with an outcome on the day of hospitalization.The study protocol was approved by the Mass General Brigham Human Research Committee, which waived informed consent given that this is a minimal risk study using deidentified data.The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline for validation studies was applied.
Features of hospital course were extracted from the Mass General Brigham Data Registry 4 and the Enterprise Data Warehouse, including laboratory values and high and low flags.The Charlson Comorbidity Index was calculated using coded International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) diagnostic codes. 5Race and ethnicity were defined by patient self-report using US Census categories and were included to allow assessment of bias in model performance.
Patients were followed up from admission to hospital discharge or death, with follow-up censored at discharge.Primary outcomes were (1) a composite severe illness outcome, including admission to the intensive care unit (ICU), mechanical ventilation, or mortality and (2) mortality.
Coefficients from our previously reported least absolute shrinkage and selection operator risk models were applied to estimate the probability of each outcome without recalibration; these coefficients were drawn from sociodemographic features, the comorbidity index, and laboratory values. 3We applied median imputation of missing data.We characterized model performance with standard metrics of discrimination and calibration.All analyses were conducted with R version 4 (R Project for Statistical Computing).

Discussion
Applying a previously validated model to 2892 new COVID-19 admissions in the same 6 hospitals, we found that model performance decreased only modestly from the initial validation study. 3A key  The other race category included patients who self-reported multiracial or other race and patients whose race is unknown.
exception was PPV, likely reflecting substantial diminution in mortality and mechanical ventilation between the original and the subsequent study periods.Discrimination was generally consistent across subgroups, with the notable exception of younger age groups in whom performance was poorer.
Our results indicate that the population of individuals hospitalized for COVID-19 has shifted and the prevalence of the studied outcomes changed.However, they suggest that prediction models derived earlier in the pandemic may maintain discrimination after recalibration.A limitation is the reliance on 2 health systems in the same region.Our results also illustrate the importance of investigating risk stratification models across patient subgroups as a step toward ensuring that particular groups are not adversely affected by the application of such tools, particularly in settings of potential resource constraints.

Table 1
and compared with those of the previously reported cohort in which the predictive model was trained.For the 2892 individuals in the new Hispanic individuals, and 344 (11.9%)Black individuals.The mean (SD) length of hospital stay was 6.2(5.3)days;126 patients (4.4%) required an ICU stay and 68 (2.4%) mechanical ventilation, while 167 (5.8%) died prior to discharge.Overall model performance for mortality included an area under the receiver operating characteristic curve (AUC) of 0.83 (95% CI, 0.80-0.87),with a positive predictive value (PPV) of 0.22 and a negative predictive value (NPV) of 0.98 when using a cutoff corresponding to the highest 20% of predicted risk derived in the training set.By comparison, in the original model Open Access.This is an open access article distributed under the terms of the CC-BY License.JAMA Network Open.2021;4(7):e2118413.doi:10.1001/jamanetworkopen.2021.18413(Reprinted) July 27, 2021 1/5 Downloaded From: https://jamanetwork.com/ on 09/29/2023

Table 1 .
Sociodemographic and Illness Severity Comparison Between the Initial Model Training COVID-19 Admission Cohort and the Subsequent Admissions Used to Evaluate the Model a b The other race category included patients who selfreported multiracial or other race and patients whose race is unknown.cSevere COVID-19 outcome refers to the composite severe illness outcome, including admission to the ICU, mechanical ventilation, or mortality.

Table 2 .
Discrimination and Calibration Metrics of the COVID-19 Severity and Mortality Prediction Model by Subgroup a Specificity, sensitivity, PPV, and NPV are reported for the top 20% of risk score defined in the original training set.b