Hospitals (n = 303) are ranked in the order of increasing DNR rates. Solid line indicates mean DNR rate of individual hospitals.
Regression results exclude 2 outlier hospitals with greater than 50% DNR use rates. A, Spearman r = 0.243 (P < .001). Correlation results were unchanged (Spearman r = 0.243; P < .001) when the 2 outlier hospitals were included. B, Spearman r = −0.09 (P = .12). Correlation results were similar (Spearman r = −0.08; P = .14) when the 2 outlier hospitals were included. C, Spearman r = −0.162 (P = .005). Correlation results were similar (Spearman r = −0.171; P = .003) when the 2 outlier hospitals were included. D, Spearman r = −0.162 (P = .005). Correlation results were similar (Spearman r = −0.149; P = .009) when the 2 outlier hospitals were included.
Line demonstrates theoretical line of unity where hospital mortality from the model with DNR adjustment would equal mortality without DNR adjustment (y = x). Diamonds represent hospitals that were not outliers. All triangles represent significant low-performing outlier hospitals from the model unadjusted for patient DNR status. All circles represent significant high-performing hospitals from the model unadjusted for patient DNR status. Open triangles and circles represent hospitals no longer identified as outlier hospitals after adjustment for DNR status.
eMethods. Supplemental Methods
eTable 1.International Classification of Diseases, 9th Edition, Clinical Modification (ICD-9-CM) Algorithms Used to Define Conditions and Procedures
eTable 2. Covariates Included in Models/Fixed Effects Output From Hospital Mortality Comparison Models, Primary Analysis Cohort
eTable 3. Model Performance Changes Based on Method of Inclusion of Early Do-Not-Resuscitate Status
eTable 4. Distribution of Comorbid Conditions, Acute Organ Failures, and Hospital Characteristics Across Quartiles of Hospital Early Do-Not-Resuscitate Order Rate
eTable 5. Results of Sensitivity Analyses
eFigure 1. Distribution of Hospital Do-Not-Resuscitate Rates
eFigure 2. Association Between Hospital DNR Rate and Composite Quality Measure of Hospital Processes of Care for Pneumonia
eFigure 3. Association Between Hospital DNR Rate and Composite Measure Patient Satisfaction
Customize your JAMA Network experience by selecting one or more topics from the list below.
Walkey AJ, Weinberg J, Wiener RS, Cooke CR, Lindenauer PK. Association of Do-Not-Resuscitate Orders and Hospital Mortality Rate Among Patients With Pneumonia. JAMA Intern Med. 2016;176(1):97–104. doi:10.1001/jamainternmed.2015.6324
Hospital quality measures that do not account for patient do-not-resuscitate (DNR) status may penalize hospitals admitting a greater proportion of patients with limits on life-sustaining treatments.
To evaluate the effect of analytic approaches accounting for DNR status on risk-adjusted hospital mortality rates and performance rankings.
Design, Setting, and Participants
A retrospective, population-based cohort study was conducted among adults hospitalized with pneumonia in 303 California hospitals between January 1 and December 31, 2011. We used hierarchical logistic regression to determine associations between patient DNR status, hospital-level DNR rates, and mortality measures. Changes in hospital risk-adjusted mortality rates after accounting for patient DNR status and interhospital variation in the association between DNR status and mortality were examined. Data analysis was conducted from January 16 to September 16, 2015.
Early DNR status (within 24 hours of admission).
Main Outcomes and Measures
In-hospital mortality, determined using hierarchical logistic regression.
A total of 90 644 pneumonia cases (5.4% of admissions) were identified among the 303 California hospitals evaluated during 2011; mean (SD) age of the patients was 72.5 (13.7) years, 51.5% were women, and 59.3% were white. Hospital DNR rates varied (median, 15.8%; 25th-75th percentile, 8.9%-22.3%). Without accounting for patient DNR status, higher hospital-level DNR rates were associated with increased patient mortality (adjusted odds ratio [OR] for highest-quartile DNR rate vs lowest quartile, 1.17; 95% CI, 1.04-1.32), corresponding to worse hospital mortality rankings. In contrast, after accounting for patient DNR status and between-hospital variation in the association between DNR status and mortality, hospitals with higher DNR rates had lower mortality (adjusted OR for highest-quartile DNR rate vs lowest quartile, 0.79; 95% CI, 0.70-0.89), with reversal of associations between hospital mortality rankings and DNR rates. Only 14 of 27 hospitals (51.9%) characterized as low-performing outliers without accounting for DNR status remained outliers after DNR adjustment. Hospital DNR rates were not significantly associated with composite quality measures of processes of care for pneumonia (r = 0.11; P = .052); however, DNR rates were positively correlated with patient satisfaction scores (r = 0.35; P < .001).
Conclusions and Relevance
Failure to account for DNR status may confound the evaluation of hospital quality using mortality outcomes, penalizing hospitals that admit a greater proportion of patients with limits on life-sustaining treatments. Stakeholders should seek to improve methods to standardize and report DNR status in hospital discharge records to allow further assessment of implications of adjusting for DNR in quality measures.
Do-not-resuscitate (DNR) orders are usually written by physicians to convey the wishes of patients to not receive cardiopulmonary resuscitation in the event of cardiac arrest. Do-not-resuscitate orders result from complex interactions between patients, physicians, families, and regional and hospital norms1-3 regarding end-of life care. Unlike DNR orders written later during hospitalization that generally signal a lack of response to therapy, early DNR orders (within 24 hours of hospitalization) largely reflect patients’ baseline burdens of comorbidity and/or prehospitalization preferences for end-of-life care.4 As a marker of baseline severity of illness and patients’ wishes to receive less-aggressive health care interventions, early DNR orders are among the strongest predictors of in-hospital5 and posthospitalization6 mortality. Therefore, early DNR status may result in potent unmeasured confounding effects if unaccounted for during the evaluation of health care delivery.5
Despite the potential for hospital variation in DNR orders to influence patients’ end-of-life experiences and outcomes, DNR status is generally unreported by hospitals and unaccounted for in hospital outcome measures. For example, the Centers for Medicare & Medicaid Services (CMS) do not adjust for DNR status in pneumonia Hospital Compare analyses, reasoning that adjustment for DNR status may be unnecessary owing to potential correlation with other comorbidity measures.7 Some8 have argued for exclusion of DNR status from hospital comparisons because of weaker associations between DNR orders and mortality at hospitals with higher DNR rates, characteristics that may bias conventional regression analyses by assigning the same strength of adjustment to DNR status across all hospitals. However, others suggest that not accounting for variation in patients’ treatment preferences may incentivize provision of care that is not patient centered9 and may confound evaluation of hospital quality based on mortality rates.10,11
To our knowledge, studies have not yet evaluated associations between measures of comorbidity and DNR rates in hospital comparisons, how alternative analytic approaches12-15 that allow associations between DNR status and mortality to vary across hospitals might alter hospital quality rankings, or how nonmortality measures of hospital quality relate to hospital DNR rates. We sought to examine associations between hospital DNR rates and measures of hospital quality and to determine how analyses that account for associations between hospital DNR rates, patient DNR status, and mortality may influence hospital mortality rankings. We hypothesized that accounting for the DNR status of individual patients in evaluation of hospital outcomes would result in changes to hospital mortality rankings with potential policy implications. We studied patients hospitalized with pneumonia, which is a frequent target of hospital quality evaluation.16
A detailed description of the methods is presented in the eMethods in the Supplement. The study was approved by the Boston University Medical Center institutional review board as exempt from review, and all procedures were performed using deidentified data. Using the Healthcare Cost and Utilization Project California State Inpatient Database (CA SID),17 we analyzed a cohort of adults 40 years or older hospitalized with pneumonia from January 1 to December 31, 2011. The CA SID consists of administrative claims data from all nonfederal hospitalizations in California and, rather uniquely, contains a validated variable that captures DNR orders written during the first 24 hours of hospitalization.18,19 We defined pneumonia hospitalizations using either principal diagnosis claims for pneumonia or principal diagnosis for acute respiratory failure or septicemia, with a secondary diagnosis of pneumonia that was present on admission (eTable 1 in the Supplement).20,21 Patients transferred from another hospital, discharged alive within 48 hours,16 or admitted to a hospital caring for fewer than 25 patients with pneumonia in 2011 were excluded.16
We characterized both patient-level early DNR status and hospital-level early DNR rates. Hospital DNR rate was defined as the percentage of patients with pneumonia with an early DNR order at each hospital. Risk-adjusted models were created using patient demographics, comorbidities,22 and acute organ failure present on admission (eTable 2 in the Supplement).23,24
The associations between patient DNR status and hospital DNR rates with patient in-hospital mortality were examined. Because between-hospital differences in discharge practices may be associated with in-hospital mortality rates,25 we also assessed associations of hospital DNR rates with hospital discharge destinations and length of stay among survivors.
Summary data across quartiles of hospital DNR rates were determined. To examine potential confounding by patient DNR status, contingency tables and adjusted regression models examining associations between hospital DNR rates and mortality were stratified by patient DNR status. We used hierarchical logistic regression models with hospital-level random intercepts to examine multivariable-adjusted associations between hospital DNR rates and patient mortality without accounting for patient DNR status.
Because accounting for between-hospital variation in associations between hospital DNR and hospital mortality rates with a random DNR slope improved model performance (eTable 3 in the Supplement), all models adjusting for patient DNR status also included DNR status as a random slope. Including DNR status as both a fixed effect and a random slope allowed the association between patient DNR status and mortality to vary for each hospital and was intended to attenuate bias that might be introduced if patients at some hospitals elected DNR status at lower levels of illness severity than at other hospitals.
Hospitals were ranked according to each institution’s effect estimate as determined by the random effects.16 Hospitals with a statistically significantly greater risk-standardized mortality compared with the mean were considered to be low-performing outliers. Hospital risk-standardized mortality rates were calculated from the ratio of the hospital risk-adjusted rate to the mean risk-adjusted rate, multiplied by the mean unadjusted hospital rate in California.16 We assessed the correlation between risk-standardized hospital early DNR rates and mortality outcomes quantitatively using Spearman rank correlation coefficients and visually using penalized B-spline regression.26 Differences in risk-standardized hospital DNR rates between hospitals ranked in the highest and lowest risk-standardized mortality quartiles were tested using Wilcoxon-Mann-Whitney tests. We evaluated the degree of hospital-level variation in early DNR use unexplained by measured patient characteristics by calculating intraclass correlation coefficients from hospital-level intercept variance estimates27-29 in multivariable-adjusted hierarchical logistic regression models, with early DNR status as the outcome of interest.
A sensitivity analysis with a model that adjusted for hospital characteristics (hospital ranking models do not routinely adjust for hospital characteristics) in addition to patient characteristics was conducted. We performed a second sensitivity analysis using methods similar to those of CMS Hospital Compare models and included patients aged 65 years or older with a principal diagnosis of pneumonia, without adjustment for race, income, or acute organ failure.
The association between hospital risk-standardized DNR rates and nonmortality measures of hospital quality were assessed to inform findings regarding changes in hospital rankings after accounting for DNR status. We linked year 2011 CMS Hospital Compare measures of pneumonia processes of care30 and patient satisfaction surveys31 to CA SID hospital data and created composites of hospital quality measures by determining the mean percentage of patients achieving the highest quality rating (eg, measure always performed). We assessed Spearman correlation between hospital DNR rates and hospital quality measures.
All statistical analyses were performed between January 16 and December 16, 2015, using SAS version, 9.3 (SAS Institute Inc). A 2-tailed α level of .05 was considered statistically significant for all analyses.
We identified 90 644 pneumonia cases (5.4% of admissions) among 303 California hospitals. Patients with pneumonia were a mean (SD) age of 72.5 (13.7) years, 51.5% were women, 59.3% were white, and 12.6% died in the hospital. Table 1 summarizes the demographic and clinical characteristics of patients hospitalized with pneumonia. Patients admitted to hospitals in the highest DNR rate quartile were older, more likely to be white, and more likely to be women and had higher median household incomes compared with those admitted to low DNR quartile hospitals. Although the distribution of specific comorbid conditions and acute organ failure (eTable 4 in the Supplement) differed based on hospital DNR rates, patients at high DNR quartile hospitals had a similar number of comorbidities and fewer cases of acute organ failure compared with patients at low DNR quartile hospitals.
The median hospital early DNR rate was 15.8% (25th-75th percentile, 8.9%-22.3%) (Figure 1 and eFigure 1 in the Supplement). The proportion of between-hospital variation for DNR orders unexplained by measured clinical covariates was 21% (95% CI, 19%-25%).
As expected, patients who received early DNR orders had a higher risk of mortality (adjusted odds ratio, 3.74; 95% CI, 3.55-3.93). Similarly, without accounting for patient DNR status, hospitals with higher DNR rates had higher patient mortality rates (Table 2 and Figure 2A). However, after stratification by patient DNR status, hospital DNR rates were inversely associated with mortality among patients without DNR orders (Table 2 and Figure 2B) and among patients with DNR orders (Table 2 and Figure 2C). Results of the multivariable-adjusted random hospital intercept with the random DNR coefficient model were similar to analyses stratified by DNR status, showing an inverse association between hospital DNR rates and patient mortality (Figure 2D) (adjusted OR for highest-quartile DNR rate vs lowest-quartile rate, 0.79; 95% CI, 0.70-0.89). Hospitals in the highest quartile of DNR rate had lower rates of discharge to long-term care facilities (28% vs 35%) and greater rates of discharge home (59% vs 53%) but shorter lengths of stay (median [25th-75th percentile], 5 [3-7] vs 6 [4-9] days) compared with hospitals with the lowest DNR rate.
In a model that did not account for patient DNR status, 27 hospitals were identified as significant low-performing outliers (greatest mortality), and hospitals ranked in the highest-performing quartile (lowest mortality) had lower median (25th-75th percentile) DNR rates (11.6% [6.8%-19.1%]) than the lowest-performing quartile (19.1% [12.3%-27.9%]) (P < .001). After accounting for patient DNR status, 20 hospitals were identified as significant low-performing outliers and 14 (51.9%) outlier hospitals were shared with the DNR-unadjusted analysis. Differences in median (25th-75th percentile) hospital DNR rates between hospitals ranked in the highest-performing quartile (17.7% [10.6%-24.0%]) and the lowest-performing quartile (12.1% [6.2%-21.0%]) (P = .01) were reversed after accounting for DNR status and between-hospital variation in associations between hospital DNR and hospital mortality rates. Changes in hospital mortality rates before and after adjusting for DNR status are shown in Figure 3; differences in effect estimates of model covariates are presented in eTable 2 in the Supplement.
Small hospitals, nonteaching hospitals, and nonprofit hospitals were more likely to have high DNR rates (eTable 4 in the Supplement); however, adjusting for hospital characteristics did not substantively change our results. Similarly, an analysis that more closely approximated CMS Hospital Compare models (n = 34 745) showed findings comparable to those of our primary models (eTable 5 in the Supplement). Hospital DNR rates were not significantly associated with composite quality measures of hospital processes of care for pneumonia (r = 0.11; P = .052) (eFigure 2 in the Supplement); however, DNR rates were positively correlated with patient satisfaction scores (r = 0.35; P < .001) (eFigure 3 in the Supplement).
Using a population-based cohort of adult pneumonia hospitalizations from California, we found a large variation in early DNR rates across hospitals. The DNR rates were strongly associated with patient demographics (eg, age, race/ethnicity, and median income) rather than measures of increased comorbidity burden. Without accounting for patient DNR status, higher hospital DNR rates appeared to be associated with lower survival, resulting in hospitals with higher DNR rates receiving worse mortality quality rankings. However, when risk adjustment accounted for patient DNR status, hospitals with higher DNR rates had better survival. Furthermore, hospitals with higher DNR rates had better patient satisfaction with similar quality of pneumonia processes of care. Our findings suggest that current methods of comparing hospitals, which do not account for patient DNR status, penalize potentially high-quality hospitals admitting a larger proportion of patients who had chosen to forego resuscitation. Therefore, accounting for DNR status in programs that compare hospital mortality outcomes may substantially affect publicly reportable hospital rankings and hospital reimbursements.
Our findings are most comparable with those of Tabak et al,8 Kelly et al,10 and Escobar et al,11 who examined associations between hospital variation in DNR orders with hospital mortality in other conditions. Although all studies found substantial changes in hospital mortality rankings after adjusting for early DNR status, conclusions differed. Tabak et al8 argued against using DNR status in hospital risk adjustment owing to potential bias introduced by applying the mean mortality rate associated with DNR status to an entire cohort when DNR-associated mortality varied between hospitals. Kelly et al10 found similar changes to hospital mortality rankings among patients with stroke. Escobar et al11 used standardized DNR orders and detailed physiologic data for patients hospitalized among 21 institutions in an integrated health care delivery system and concluded that adjustment for advance directives was “desirable” when DNR orders could be standardized. Standardized DNR reporting in the small hospital sample investigated by Escobar et al did not result in inverse associations between hospital DNR rates and mortality among DNR patients. Our methods differed from those of prior studies, focusing on pneumonia, using models that attenuated potential bias introduced from uniformly adjusting for less-standardized DNR orders across hospitals, and expanding hospital quality assessments beyond mortality. We advocate for further studies, including in other cohorts, to explore methods to accurately identify early DNR status from hospital discharge data and examine methodologies to account for variables, such as DNR status, that demonstrate variable associations with outcomes across hospitals.
Wide variation in early DNR orders between hospitals may be the result of a combination of patient preferences and hospital practices. Patient preferences may vary based on beliefs regarding end-of-life care; clustering of patients with similar beliefs may be reflected in a hospital’s DNR rate. As suggested in prior studies5,32-34 showing greater rates of DNR orders among white patients, we identified that hospitals with high DNR rates tended to admit more white patients with higher median incomes but without greater measures of comorbidity or acute illness severity. These findings suggest that socioeconomic and cultural factors result in different thresholds for choosing to forgo resuscitation. Currently, CMS models evaluating hospital quality do not adjust for race or socioeconomic status. Because hospital DNR rates are strongly associated with patient socioeconomic characteristics, additional studies are required to assess the potential ramifications of including DNR status in hospital ranking models.
Hospital practice norms may also influence when and how discussions of advance directives are initiated1 and the extent to which advance directives are carried out.3,35 Physicians and patients may define DNR differently, ranging from strict comfort measures only to care restricting only cardiopulmonary resuscitation.36-38 The inverse association between hospital DNR rates and acute organ failure supports a hypothesis that hospitals may vary in thresholds for initiating advance directive discussions, resulting in a lower level of acute severity of illness among patients with DNR orders at hospitals with high DNR rates. We demonstrated methods that may account for differences in DNR-associated care processes while more appropriately adjusting for previously unmeasured patient characteristics associated with DNR status.
The reversal in direction of the association between hospital DNR rates and patient mortality after accounting for patient DNR status demonstrates the Simpson paradox39 in a hospital quality measure. The Simpson paradox is a type of confounding that occurs when an association seen in an aggregate sample reverses in stratified analysis; it is generally caused by 2 factors: (1) a strong imbalance in distribution of exposures between subgroups and (2) imbalanced outcomes within the subgroup. In the present study, variation in hospital DNR rates and differing associations between patient DNR status and mortality across hospitals likely contributed to the Simpson paradox. Although simulation studies40,41 have theorized that the Simpson paradox may affect hospital quality measures, we are unaware of studies demonstrating the Simpson paradox occurring in a real-world performance measure.
Others may consider the reversal of direction in the association between hospital mortality and DNR rates after stratifying patients by DNR status as a Will Rogers phenomenon.42 This phenomenon may result from the reclassification of patients by DNR status because more marginal patients with a lower mortality risk likely had DNR orders at high–DNR rate hospitals (diluting the DNR pool with patients who were less ill), but these patients with DNR orders (who have greater-than-average risk of mortality) were redistributed out of the no-DNR pool at high–DNR rate hospitals. However, accounting for DNR status in models including all patients showed findings similar to the stratified analysis, suggesting that Will Rogers phenomenon alone does not fully account for our findings.
Our results show that variation in DNR rates may strongly affect hospital mortality rankings, even when accounting for between-hospital differences in the association between DNR status and mortality, and underscore the potential susceptibility of hospital comparison models to unmeasured factors. Concordance between hospital DNR rates, hospital mortality rates (after adjusting for patient DNR status), patient satisfaction, and pneumonia processes of care further argue for adjusting to include patient DNR status in hospital mortality measures. From the perspective of DNR status as a characteristic representing a composite of comorbidity, frailty, functional status, and wishes for less-aggressive care, our findings suggest that early DNR status is a confounder that should be accounted for in hospital mortality rankings (ie, the Simpson paradox). From the perspective that DNR orders also result from variable hospital norms and practice patterns that elicit advance directives, facilities that more actively seek to determine patient resuscitation wishes may differentially recategorize marginal patients by DNR status and alter mortality rankings (ie, Will Rogers phenomenon). Thus, before adopting DNR status into hospital comparison models, future studies should seek to determine the extent to which hospital-level DNR rates represent patient-driven phenomena or are determined by hospital processes norms in how advance directive discussions are approached.
Several strengths and limitations of our study should be considered in evaluating the findings. Although we sought to pattern our analysis after current CMS methods for hospital comparisons,16 our primary analysis differs in many ways from CMS models; however, the results did not substantially differ in sensitivity analysis, more closely approximating the CMS approach. It is possible that other unmeasured variables, such as functional status, disability, or prior hospitalizations, may also influence associations between DNR status and outcomes. In addition, early DNR measures in CA SID show good accuracy, but the potential for misclassification bias cannot be ruled out. Although use of a random slope method improved model performance and attenuated bias from variation in DNR and mortality associations across hospitals, residual bias in favor of high–DNR rate hospitals may not be eliminated. Thus, further work to standardize DNR reporting across hospitals may be needed before adopting DNR status into hospital quality comparison models. We were able to measure hospital (rather than 30-day) mortality, an outcome that may be affected by hospital discharge patterns to long-term acute care hospitals and length of stay.25,43 Because patients admitted to high–DNR rate hospitals were less likely to be discharged to long-term care facilities but had shorter lengths of stay, large bias in the difference between 30-day and hospital mortality resulting from differences in discharge practices is unlikely.11 Because event rates for mortality were not rare, odds ratios from our models may not approximate risk ratios. Finally, although we anticipate large hospital variation in DNR orders across the United States, generalization of our results to regions outside of California requires further study.
We have shown that failure to account for between-hospital variation in the rates of patients with limitations on life-sustaining treatments may confound the evaluation of hospital mortality outcomes. We describe a method for attenuating potential bias in efforts to measure the performance of hospitals that may be introduced by the Simpson paradox between hospital DNR rates and DNR-associated mortality. Our findings have potentially serious ramifications for methods used to assess patient outcomes and hospital quality. Given the proliferation of health care quality measures and the importance of accounting for confounding by early DNR status in the evaluation of outcomes, stakeholders should expedite efforts to standardize and report patient-level early DNR status in discharge data and pursue further studies to evaluate the ramifications of adjusting for DNR status in hospital quality measures.
Corresponding Author: Allan J. Walkey, MD, MSc, The Pulmonary Center, Division of Pulmonary and Critical Care Medicine, Boston University School of Medicine, 72 E Concord St, R-304, Boston, MA 02118 (firstname.lastname@example.org).
Accepted for Publication: September 24, 2015.
Published Online: December 14, 2015. doi:10.1001/jamainternmed.2015.6324.
Author Contributions: Dr Walkey had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Walkey, Cooke.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Walkey.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Walkey, Weinberg.
Obtained funding: Walkey.
Administrative, technical, or material support: Cooke.
Study supervision: Lindenauer.
Conflict of Interest Disclosures: None reported.
Funding/Support: The study was funded by grant K01HL116768 from the National Institutes of Health (NIH) National Heart, Lung, and Blood Institute (Dr Walkey), grant K08HS020672 from the Agency for Healthcare Research and Quality (Dr Cooke), and grant K07 CA138772 from the NIH (Dr Wiener). This study was also supported by resources from the Edith Nourse Rogers Memorial Veterans Affairs Hospital (Dr Wiener) and a Career Investment Award from the Boston University School of Medicine Department of Medicine (Dr Walkey).
Role of the Funder/Sponsor: The funding agencies had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.