[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.161.130.145. Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Download PDF
Figure 1.
Comparison of Risk Scores for Predicting RRT Activation and Cardiac Arrest or ICU Transfer, RRT Activation, and a Combined Adverse Outcome
Comparison of Risk Scores for Predicting RRT Activation and Cardiac Arrest or ICU Transfer, RRT Activation, and a Combined Adverse Outcome

The accuracy of the Patient Acuity Rating (PAR), Modified Early Warning Score (MEWS), and the combined PAR-MEWS risk scores for predicting the (A) combined outcome of intensive care unit (ICU) transfer, cardiac arrest, or rapid response team (RRT) activation within 24 hours of an observation. For comparison, the individual outcomes of (B) ICU transfer or cardiac arrest, and (C) RRT activation within 24 hours are shown. Accuracy of each score is represented by the area under the receiver operator characteristic curve (AUC) with 95% confidence interval (error bars).

Figure 2.
Impact of PAR Timing on Cardiac Arrest or ICU Transfer Prediction When Combined With MEWS
Impact of PAR Timing on Cardiac Arrest or ICU Transfer Prediction When Combined With MEWS

The accuracy of Modified Early Warning Score (MEWS) vs the combined Patient Acuity Rating (PAR)-MEWS, for predicting cardiac arrest, intensive care unit (ICU) transfer, or rapid response team activation, using all the MEWS values available in the data set and carrying forward preceding PAR scores with decreasing allowable discrepancy in time between the 2 scores. Accuracy of each score is represented by the area under the receiver operator characteristic curve (AUC) with 95% confidence intervals (error bars).

1.
Winters  BD, Weaver  SJ, Pfoh  ER, Yang  T, Pham  JC, Dy  SM.  Rapid-response systems as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 pt 2):417-425.
PubMedArticle
2.
Churpek  MM, Yuen  TC, Edelson  DP.  Risk stratification of hospitalized patients on the wards. Chest. 2013;143(6):1758-1765.
PubMedArticle
3.
Edelson  DP, Retzer  E, Weidman  EK,  et al.  Patient acuity rating: quantifying clinical judgment regarding inpatient stability. J Hosp Med. 2011;6(8):475-479.
PubMedArticle
4.
Sinuff  T, Adhikari  NK, Cook  DJ,  et al.  Mortality predictions in the intensive care unit: comparing physicians with scoring systems. Crit Care Med. 2006;34(3):878-885.
PubMedArticle
5.
Chen  J, Bellomo  R, Hillman  K, Flabouris  A, Finfer  S; MERIT Study Investigators for the Simpson Centre and the ANZICS Clinical Trials Group.  Triggers for emergency team activation: a multicenter assessment. J Crit Care. 2010;25(2):359.e1-359.e7.
PubMedArticle
Views 1,863
Citations 0
Research Letter
March 2015

The Value of Clinical Judgment in the Detection of Clinical Deterioration

Author Affiliations
  • 1Division of Hospital Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois
  • 2Department of Medicine, University of Chicago, Chicago, Illinois
  • 3Center for Health Services Research, University of Kentucky, Lexington
JAMA Intern Med. 2015;175(3):456-458. doi:10.1001/jamainternmed.2014.7119

Hospitals routinely use rapid response teams (RRTs) to treat and triage unstable patients, but early and reliable identification of high-risk patients remains challenging.1 Objective, vital sign–based risk prediction scores, such as the Modified Early Warning Score (MEWS), have been developed but have limited accuracy.2 The Patient Acuity Rating (PAR), a subjective 7-point Likert scale assessment of the likelihood of cardiac arrest or intensive care unit (ICU) transfer within the next 24 hours, has been proposed as an alternate risk stratification tool.3 However, the PAR has not been externally validated or directly compared with objective metrics such as the MEWS.

Methods

This study was approved by the Northwestern University Institutional Review Board (CR3 STU00027683). Hospitalists at an academic medical center prospectively assigned a daily PAR score to consecutive medical in-patients as part of a standardized electronic medical record handoff. Covering physicians were blinded to PAR assignment. A corresponding MEWS was calculated using the closest vital signs. Outcomes were cardiac arrest or ICU transfer, RRT activation, and a composite of any 3, occurring within 24 hours of an observation. These were evaluated using area under the receiver operator characteristics curve (AUC). Logistic regression was used to combine paired PAR and MEWS scores.

Results

Between September 1, 2011, and August 31, 2012, we identified 51 eligible physicians, 34 of whom consented to participate. Participants gave written informed consent. There was no compensation for participation. Among the 34 physicians, 28 provided a total of 7244 PARs in 3249 distinct patients, with a median PAR of 2 (range, 1-7). Among these patients, we observed 2 cardiac arrests, 67 ICU transfers, and 105 RRT activations. In addition, 51 202 MEWS scores were calculated for these patients, with a median of 3 (range, 2-11). There was a median of 84 minutes between corresponding PAR and MEWS scores, which were poorly correlated (Spearman ρ = 0.10), even when the PAR-MEWS time discrepancy was less than or equal to 84 minutes (Spearman ρ = 0.13). The combination of PAR and MEWS (AUC, 0.70) was more accurate for the composite outcome than either MEWS (AUC, 0.65) or PAR alone (AUC, 0.67; P = .01 and .04, respectively). Similar trends were observed for the individual outcomes (Figure 1). Examination of all calculated MEWS scores (N = 51 202) with forward imputation of the preceding PAR, regardless of lag time, resulted in a combined risk score with improved accuracy over MEWS alone for predicting the composite end point in the next 24 hours (Figure 2).

Discussion

In this prospective, blinded validation study, we found that physician clinical judgment, as quantified by PAR, when added to the MEWS, improves its accuracy for predicting impending clinical deterioration on medical wards.

Clinical judgment-based risk assessments have been shown to be at least as accurate as, and often better than, objective risk scores for predicting mortality in critically ill patients.4 In ward patients, aggregated risk scores, such as the MEWS, are more predictive than single-parameter RRT activation criteria.3 However, these scores omit clinician worry, a frequent and early trigger in traditional systems.5 The PAR score quantifies this judgment such that it can be incorporated into risk scores.3

The accuracy of PAR for predicting cardiac arrest or ICU transfer in this study was notably lower than in the original study (AUC, 0.82). One potential explanation is that the current cohort had a lower rate of cardiac arrest or ICU transfer (1.0% vs 2.2%), which could reflect a healthier patient population, different ICU transfer or RRT activation thresholds, or the more frequent PAR assignment.

The weak correlation between the PAR and the MEWS is unlikely a factor of the variation in timing between the score assignments, as those close together were still poorly correlated. More likely, physicians were taking into account variables not contained in the MEWS, in assigning a PAR score.

We conclude that the combined use of PAR and MEWS is more accurate than MEWS alone. A generalizable way to incorporate clinical judgment into aggregated weighted scoring systems on the wards may improve detection of clinical deterioration and assist with timely mobilization of resources to address the etiology of clinical deterioration.

Back to top
Article Information

Corresponding Author: Dana P. Edelson, MD, MS, Department of Medicine, University of Chicago, 5841 S Maryland Ave, MC 5000, Chicago, IL 60637 (dperes@uchicago.edu).

Published Online: January 5, 2015. doi:10.1001/jamainternmed.2014.7119.

Author Contributions: Mr Zadravecz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Williams, Edelson.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Patel, Zadravecz, Edelson.

Critical revision of the manuscript for important intellectual content: Patel, Young, Williams, Churpek, Edelson.

Statistical analysis: Zadravecz, Churpek.

Administrative, technical, or material support: Patel, Young, Edelson.

Study supervision: Patel, Churpek, Edelson.

Conflict of Interest Disclosures: None reported.

Previous Presentation: This study was presented as a poster for the Hospital Medicine Annual Meeting; May 17, 2013; National Harbor, Maryland.

References
1.
Winters  BD, Weaver  SJ, Pfoh  ER, Yang  T, Pham  JC, Dy  SM.  Rapid-response systems as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 pt 2):417-425.
PubMedArticle
2.
Churpek  MM, Yuen  TC, Edelson  DP.  Risk stratification of hospitalized patients on the wards. Chest. 2013;143(6):1758-1765.
PubMedArticle
3.
Edelson  DP, Retzer  E, Weidman  EK,  et al.  Patient acuity rating: quantifying clinical judgment regarding inpatient stability. J Hosp Med. 2011;6(8):475-479.
PubMedArticle
4.
Sinuff  T, Adhikari  NK, Cook  DJ,  et al.  Mortality predictions in the intensive care unit: comparing physicians with scoring systems. Crit Care Med. 2006;34(3):878-885.
PubMedArticle
5.
Chen  J, Bellomo  R, Hillman  K, Flabouris  A, Finfer  S; MERIT Study Investigators for the Simpson Centre and the ANZICS Clinical Trials Group.  Triggers for emergency team activation: a multicenter assessment. J Crit Care. 2010;25(2):359.e1-359.e7.
PubMedArticle
×