Comparison of 4 Acute Pulmonary Embolism Mortality Risk Scores in Patients Evaluated by Pulmonary Embolism Response Teams

Key Points

Question  How well do different risk assessment tools estimate 7- and 30-day mortality in patients with acute pulmonary embolism?

Findings  This cohort study of 416 patients with acute pulmonary embolism found that commonly used risk assessment tools have only moderate discriminative ability for 7- and 30-day mortality in patients with acute pulmonary embolism.

Meaning  These findings suggest that clinicians may need to integrate broad clinical information rather than relying on a single risk assessment tool to estimate mortality risk and determine management for patients with acute pulmonary embolism.

Importance  The risk of death from acute pulmonary embolism can range as high as 15%, depending on patient factors at initial presentation. Acute treatment decisions are largely based on an estimate of this mortality risk.

Objective  To assess the performance of risk assessment scores in a modern, US cohort of patients with acute pulmonary embolism.

Design, Setting, and Participants  This multicenter cohort study was conducted between October 2016 and October 2017 at 8 hospitals participating in the Pulmonary Embolism Response Team (PERT) Consortium registry. Included patients were adults who presented with acute pulmonary embolism and had sufficient information in the medical record to calculate risk scores. Data analysis was performed from March to May 2020.

Main Outcomes and Measures  All-cause mortality (7- and 30-day) and associated discrimination were assessed by the area under the receiver operator curve (AUC).

Results  Among 416 patients with acute pulmonary embolism (mean [SD] age, 61.3 [17.6] years; 207 men [49.8%]), 7-day mortality in the low-risk groups ranged from 1.3% (1 patient) to 3.1% (4 patients), whereas 30-day mortality ranged from 2.6% (1 patient) to 10.2% (13 patients). Among patients in the highest-risk groups, the 7-day mortality ranged from 7.0% (18 patients) to 16.3% (7 patients), whereas 30-day mortality ranged from 14.4% (37 patients) to 26.3% (26 patients). Each of the risk stratification tools had modest discrimination for 7-day mortality (AUC range, 0.616-0.666) with slightly lower discrimination for 30-day mortality (AUC range, 0.550-0.694).

Conclusions and Relevance  These findings suggest that commonly used risk tools for acute pulmonary embolism have modest estimating ability. Future studies to develop and validate better risk assessment tools are needed.

Most patients with acute pulmonary embolism (PE) experience few, if any, complications. However, an estimated 5% to 15% of patients with acute PE are at high risk of death or hemodynamic collapse.¹ Multiple risk assessment models have been developed to identify patients at risk for these complications. These commonly used risk scores are widely recommended in various treatment guidelines and expert recommendations.^2,3

Some risk scores are primarily used to identify low-risk patients for whom outpatient treatment may be appropriate.^4,5 Others aim to identify patients for whom the risk of death and hemodynamic deterioration is sufficiently high to consider the use of thrombolytic or other advanced therapies. Common examples include the Pulmonary Embolism Severity Index (PESI), the simplified PESI (sPESI), and Bova scores.^6,7 The PESI and sPESI have demonstrated the ability to discriminate between high and low risk for 30-day all-cause mortality.^2,8,9 However, they were derived from retrospective cohorts and may not be ideal for guiding clinical decision-making at the time of PE diagnosis. A fourth system, outlined in the 2014 and 2019 European Society of Cardiology (ESC) guidelines,^2,10 recommends using PESI or the sPESI but also adds biomarker and radiological markers to the risk stratification scheme.

It is unknown how similarly each of these risk tools will stratify an individual patient’s risk. In addition, inconsistent ability to estimate shorter-term outcomes (eg, 7- and 30-day mortality) limits the utility of many of these risk scores.¹¹ Our objective was to explore the short-term outcomes among patients with acute PE and to compare the ability of different risk scores to discriminate among low- and high-risk patients.

At 8 participating Pulmonary Embolism Response Team (PERT) Consortium Pilot Registry hospitals, patients with acute PE were referred for a PERT evaluation by clinical specialists. Institutional review board approval for a waiver of informed consent was obtained at each center because the study was deemed to pose minimal risk to patients.

Criteria for PERT consultation vary at each center but generally require a radiographically confirmed acute PE in an adult patient for whom the primary team has some question about PE-related management. PERT consultation typically involved discussions between the primary team and 1 or more PE experts to help assess PE-related risk and guide management decision-making.^12,13 Independent of clinical care, data were abstracted from the medical record into the PERT Consortium Pilot Registry using predefined data elements and a REDCap database. Included patients presented between October 2016 and October 2017. Abstraction at each center was performed by a physician or trained research staff member. Full details of the registry have been published elsewhere.¹⁴

We calculated the PESI, sPESI, and Bova scores for each patient with confirmed PE in the PERT Consortium pilot registry who had complete data required to calculate each risk score, using the methods from the derivation studies (eFigures 1, 2, 3, 4, and 5 in the Supplement).^6-8 We also assigned an ESC risk to each patient according to the presence of shock or hypotension (defined as systolic blood pressure [SBP] <90 mm Hg), signs of right ventricular dysfunction on echocardiogram or computed tomography, and elevated cardiac biomarkers.² We did not include the PESI or sPESI score in the ESC risk stratification to avoid collinearity when comparing ESC with PESI or sPESI.

In addition to calculating individual PESI, sPESI, and Bova scores, we categorized patients according to the risk categories from the derivation studies.^6-8 For PESI, we included 5 risk classes (I-V), whereas for sPESI we included a low risk (score = 0) and high risk (score >0). The Bova score categorizes patients into 3 risk groups (I-III) but excludes patients presenting with hypotension. To properly compare an individual’s risk between different scores, we modified the Bova score using 2 methods. First, we created a fourth risk group for patients with an SBP less than 90 mm Hg at presentation. Second, we included all patients with SBP less than 90 mm Hg into the high-risk group (class III).

Patient survival was assessed at standard intervals in the PERT Consortium pilot registry. We assessed for all-cause death at any time from hospital presentation to 7 days and up to 30 days. We addressed missing data using multiple imputation. Twenty-five data sets were imputed using the fully conditional specification method with logistic regression for class variable and estimated mean matching for continuous variables. These data sets were then used for the construction of the predictive logistic regression models, and the results from the multiple models were pooled for analysis using the inferential multiple imputation analysis strategy by Rubin.¹⁵

The discrimination of each risk score to estimate 7- and 30-day mortality was calculated by measuring the area under the receiver operator curve (AUC) and was compared using a contrast matrix approach.¹⁶ A 2-sided P < .05 was considered significant. All statistical analyses were performed using SAS statistical software version 9.4 (SAS Institute). Data analysis was performed from March to May 2020.

Our analysis includes 416 adult patients (mean [SD] age, 61.3 [17.6] years; 207 men [49.8%]; mean [SD] body mass index [calculated as weight in kilograms divided by height in meters squared], 31.9 [10.1]) with acute PE presenting to 8 hospitals (Table 1). Nearly one-half of the patients received anticoagulation therapy alone (188 patients [45.2%]), with fewer receiving thrombolysis (32 patients [7.7%]), thrombectomy (11 patients [2.6%]), or other advanced interventions (106 patients [25.5%]).

Among the 416 patients, all-cause death occurred within 7 days for 25 patients (6.0%) and within 30 days for 51 patients (12.3%). As seen in Table 2, 7-day mortality in the low-risk groups ranged from 1.3% (1 patient; sPESI) to 3.1% (4 patients; Bova), whereas 30-day mortality ranged from 2.6% (1 patient; PESI) to 10.2% (13 patients; Bova); the rate was 3.8% (3 patients) for sPESI. Among patients in the highest-risk groups, the 7-day mortality ranged from 7.0% (18 patients; sPESI) to 16.3% (7 patients; SBP <90 mm Hg in Bova), whereas 30-day mortality ranged from 14.4% (37 patients; sPESI) to 26.3% (26 patients; PESI). Distribution of patients according to risk scores is shown in eFigures 1, 2, 3, 4, and 5 in the Supplement.

Each of the risk stratification tools had modest discrimination for 7-day mortality (AUC range, 0.616-0.666) with slightly lower discrimination for 30-day mortality (AUC range, 0.550-0.694). Mean differences in AUC between the different scores were small for 7-day mortality (≤0.05) but somewhat larger for 30-day mortality, especially when comparing the Bova score with either the PESI (AUC difference, 0.13-0.14) or sPESI (AUC difference, 0.09-0.11) scores (eTable in the Supplement).

Our analysis demonstrates modest discrimination and ability to estimate 7- or 30-day mortality for each of the PE-specific risk scores, most of which are better for estimation of the shorter term outcomes. Additionally, there is little association among the 4 acute PE–specific risk scores.

Although not universally used, risk scores are increasingly recommended by guidelines for initial assessment of patients presenting with acute PE.^2,3,17 On the basis of these initial assessments, treatment options (eg, anticoagulation alone, thrombolysis, or thrombectomy) are considered. However, our data suggest that no single risk score is highly accurate or superior to another for estimating short-term (7-day) or slightly longer-term (30-day) mortality.

PE risk scores are often used in a variety of clinical situations. For instance, the sPESI score has been shown to identify a cohort of patients at very low risk for complications and health care resource utilization for whom outpatient treatment may be preferred.¹⁸ However, it has also been found to misclassify many patients into the high-risk category for which they may be hospitalized unnecessarily.¹⁹

A systematic review²⁰ from 2016 identified 17 different risk assessment models for PE-associated mortality. Overall 30-day mortality rates in that meta-analysis were similar to those for our study population among low-risk patients according to both the PESI (2.3% vs 2.6%) and the sPESI (1.5% vs 3.8%) scores. However, 30-day mortality rates differed between the meta-analysis and our study among those in the highest-risk groups for the PESI score (11.4% vs 26.3%) but less so for the sPESI score (10.7% vs 14.4%). This may be associated, in part, with the inclusion in our population of only patients for whom a PERT evaluation was requested. It is possible that patients with multiple comorbidities but hemodynamically stable PE did not have a PERT evaluation and, therefore, were not included in our registry.

From a research and quality improvement standpoint, use of any single PE risk stratification tool may not be adequate to appropriately risk-adjust patient outcomes. This is particularly relevant when comparing clinical outcomes across hospitals or organizations. In addition, the limited ability of any single risk score to estimate mortality may limit its use in identifying patients most likely to benefit from advanced therapies. It is possible that currently available therapies (eg, catheter-directed thrombolysis) may offer mortality benefit only if the patients most at risk for complications can easily be identified. To date, this risk assessment has limited success demonstrating patient-oriented clinical benefit in both the short and long term.^21,22

Our analysis has a number of important strengths, including the use of multicenter data from a prospective registry with predefined data elements. However, not all cases of acute PE at each center were captured. This is particularly true for lower-risk PE cases, for which PERT is not typically activated. It is possible that some of these scores are better estimators of outcomes and have an association with low-risk patients with acute PE. Second, only adverse events that were recorded in the medical record from the index hospital were recorded. In addition, mortality was assessed as all-cause rather than PE-specific because we are unable to assess death attributable to PE or to underlying comorbidities in the data set. Third, clinicians may have incorporated 1 or more of these scores into their management decisions, which potentially is associated with mortality outcomes. Fourth, larger validations studies are needed given the small overall number of deaths in our study population. However, we believe this to be one of the largest validation studies of these risk tools published to date. Fifth, we were unable to assess clinical gestalt as a factor associated with death, despite prior studies demonstrating good clinical estimation.²³ Similarly, we are unable to assess for patient preference for different treatments because of limitations in the available data. Sixth, because of variation in the number of patients contributed from each center, including at least 1 center without any recorded deaths, we were unable to adjust for clustering at the health center level in our statistical models. Furthermore, central adjudication of data collection and adverse events was not able to be performed.

These limitations notwithstanding, our analysis demonstrates only modest discrimination and ability to estimate short-term mortality in patients with acute PE. Furthermore, there appears to be minimal association among the different risk scores for individual patients. Future studies to develop and validate better risk assessment tools would improve both clinical care and PE-related research.

Accepted for Publication: May 6, 2020.

Published: August 26, 2020. doi:10.1001/jamanetworkopen.2020.10779

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Barnes GD et al. JAMA Network Open.

Corresponding Author: Geoffrey D. Barnes, MD, MSc, Frankel Cardiovascular Center, Department of Internal Medicine, University of Michigan, 2800 Plymouth Rd, B14 G214, Ann Arbor, MI 48109-2800 (gbarnes@umich.edu).

Author Contributions: Dr Barnes and Ms Muzikansky had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Barnes, Giri, Jaber, Wood, Courtney, Tapson, Kabrhel.

Acquisition, analysis, or interpretation of data: Muzikansky, Cameron, Heresi, Wood, Todoran, Courtney, Kabrhel.

Drafting of the manuscript: Barnes, Muzikansky, Cameron, Courtney, Kabrhel.

Critical revision of the manuscript for important intellectual content: Barnes, Muzikansky, Cameron, Giri, Heresi, Jaber, Wood, Todoran, Tapson, Kabrhel.

Statistical analysis: Barnes, Muzikansky, Cameron.

Obtained funding: Jaber, Kabrhel.

Administrative, technical, or material support: Barnes, Cameron, Heresi, Jaber, Todoran, Kabrhel.

Supervision: Giri, Tapson, Kabrhel.

Conflict of Interest Disclosures: Dr Barnes reported receiving grants and personal fees from Pfizer/Bristol-Myers Squib and personal fees from Janssen, Portola, and AMAG Pharmaceuticals during the conduct of the study. Dr Giri reported receiving nonfinancial support from the US PE Response Team (PERT) Consortium; personal fees from Inari Medical, Astra Zeneca, and New England Research Institute; and grants from Recor Medical and St Jude Medical outside the submitted work. Dr Jaber reported receiving personal fees from Inari Medical outside the submitted work. Dr Courtney reported receiving grants from Stago outside the submitted work. Dr Tapson reported receiving grants from BMS, Daiichi, Inari, Penumbra, and Bayer and personal fees from Janssen during the conduct of the study; he also reported being immediate past president of the PERT Consortium. Dr Kabrhel reported receiving grants from Diagnostica Stago, Siemens Healthcare Diagnostics, and Janssen and personal fees from Boston Scientific/EKOS Corp outside the submitted work. No other disclosures were reported.

Funding/Support: Dr Cameron was supported by grant K08HL128856 from the National Heart, Lung, and Blood Institute. No external funding specific to this project was used.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.