Figure 1.
Range of Hospital Early Do-Not-Resuscitate (DNR) Order Rates

Hospitals (n = 303) are ranked in the order of increasing DNR rates. Solid line indicates mean DNR rate of individual hospitals.

Figure 2.
Association Between Risk-Standardized Hospital Do-Not-Resuscitate (DNR) Order Rates and Risk-Standardized Hospital Mortality Rates

Regression results exclude 2 outlier hospitals with greater than 50% DNR use rates. A, Spearman r = 0.243 (P < .001). Correlation results were unchanged (Spearman r = 0.243; P < .001) when the 2 outlier hospitals were included. B, Spearman r = −0.09 (P = .12). Correlation results were similar (Spearman r = −0.08; P = .14) when the 2 outlier hospitals were included. C, Spearman r = −0.162 (P = .005). Correlation results were similar (Spearman r = −0.171; P = .003) when the 2 outlier hospitals were included. D, Spearman r = −0.162 (P = .005). Correlation results were similar (Spearman r = −0.149; P = .009) when the 2 outlier hospitals were included.

Figure 3.
Risk-Standardized Hospital Mortality Rates Resulting From Model Adjusted and Unadjusted for Patient Do-Not-Resuscitate (DNR) Status

Line demonstrates theoretical line of unity where hospital mortality from the model with DNR adjustment would equal mortality without DNR adjustment (y = x). Diamonds represent hospitals that were not outliers. All triangles represent significant low-performing outlier hospitals from the model unadjusted for patient DNR status. All circles represent significant high-performing hospitals from the model unadjusted for patient DNR status. Open triangles and circles represent hospitals no longer identified as outlier hospitals after adjustment for DNR status.

Table 1.
Patient Characteristics According to Quartile of Hospital DNR Status Rate
Table 2.
Association Between Hospital DNR Rates and Patient Mortality Among All Patients With Pneumonia
1.
Barnato  AE, Mohan  D, Lane  RK,  et al.  Advance care planning norms may contribute to hospital variation in end-of-life ICU use: a simulation study.  Med Decis Making. 2014;34(4):473-484.PubMedArticle
2.
Dzeng  E, Colaianni  A, Roland  M,  et al.  Influence of institutional culture and policies on do-not-resuscitate decision making at the end of life.  JAMA Intern Med. 2015;175(5):812-819. PubMedArticle
3.
Zingmond  DS, Wenger  NS.  Regional and institutional variation in the initiation of early do-not-resuscitate orders.  Arch Intern Med. 2005;165(15):1705-1712.PubMedArticle
4.
Marrie  TJ, Fine  MJ, Kapoor  WN, Coley  CM, Singer  DE, Obrosky  DS.  Community-acquired pneumonia and do not resuscitate orders.  J Am Geriatr Soc. 2002;50(2):290-299.PubMedArticle
5.
Bradford  MA, Lindenauer  PK, Wiener  RS, Walkey  AJ.  Do-not-resuscitate status and observational comparative effectiveness research in patients with septic shock.  Crit Care Med. 2014;42(9):2042-2047.PubMedArticle
6.
Baldwin  MR, Narain  WR, Wunsch  H,  et al.  A prognostic model for 6-month mortality in elderly survivors of critical illness.  Chest. 2013;143(4):910-919.PubMedArticle
7.
National Quality Forum.  Pulmonary and critical care consensus standards endorsement maintenance: technical report: NQF pulmonary comment period: AHA letter points for 30-day COPD and pneumonia measures.http://www.qualityforum.org/Publications/2013/04/Pulmonary_and_Critical_Care_Consensus_Standards_Endorsement_Maintenance.aspx. Published April 2013. Accessed April 23, 2015.
8.
Tabak  YP, Johannes  RS, Silber  JH, Kurtz  SG.  Should Do-Not-Resuscitate status be included as a mortality risk adjustor? the impact of DNR variations on performance reporting.  Med Care. 2005;43(7):658-666.PubMedArticle
9.
Holloway  RG, Quill  TE.  Mortality as a measure of quality: implications for palliative and end-of-life care.  JAMA. 2007;298(7):802-804.PubMedArticle
10.
Kelly  AG, Zahuranec  DB, Holloway  RG, Morgenstern  LB, Burke  JF.  Variation in do-not-resuscitate orders for patients with ischemic stroke: implications for national hospital comparisons.  Stroke. 2014;45(3):822-827.PubMedArticle
11.
Escobar  GJ, Gardner  MN, Greene  JD, Draper  D, Kipnis  P.  Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system.  Med Care. 2013;51(5):446-453.PubMedArticle
12.
Localio  AR, Berlin  JA, Ten Have  TR, Kimmel  SE.  Adjustments for center in multicenter studies: an overview.  Ann Intern Med. 2001;135(2):112-123.PubMedArticle
13.
Finucane  MM, Samet  JH, Horton  NJ.  Translational methods in biostatistics: linear mixed effect regression models of alcohol consumption and HIV disease progression over time.  Epidemiol Perspect Innov. 2007;4:8.PubMedArticle
14.
Agresti  A, Hartzel  J.  Strategies for comparing treatments on a binary response with multi-centre data.  Stat Med. 2000;19(8):1115-1139.PubMedArticle
15.
Gould  AL.  Multi-centre trial analysis revisited.  Stat Med. 1998;17(15-16):1779-1797.PubMedArticle
16.
Bratzler  DW, Normand  SL, Wang  Y,  et al.  An administrative claims model for profiling hospital 30-day mortality rates for pneumonia patients.  PLoS One. 2011;6(4):e17401.PubMedArticle
17.
State Inpatient Database.  California: Healthcare Cost and Utilization Project: overview of the state inpatient databases. Rockville, MD: Agency for Healthcare Research and Quality; 2011. http://www.hcup-us.ahrq.gov/sidoverview.jsp. Published 2011. Accessed January 16, 2015.
18.
Goldman  LE, Chu  PW, Prothro  C, Osmond  D, Bindman  AB, eds.  Accuracy of condition present on admission, do not resuscitate, and e-codes in California Patient Discharge Data: prepared for the Office of Statewide Health Planning and Development, Healthcare Outcomes Center.http://www.oshpd.ca.gov/HID/Products/PatDischargeData/ResearchReports/PDDValidation/PDD_Validation_Study.pdf. Published Spring 2011. Accessed June 27, 2015.
19.
Goldman  LE, Chu  PW, Osmond  D, Bindman  A.  Accuracy of do not resuscitate (DNR) in administrative data.  Med Care Res Rev. 2013;70(1):98-112.PubMedArticle
20.
Walkey  AJ, Lagu  T, Lindenauer  PK.  Trends in sepsis and infection sources in the United States: a population-based study.  Ann Am Thorac Soc. 2015;12(2):216-220.PubMedArticle
21.
Rothberg  MB, Pekow  PS, Priya  A, Lindenauer  PK.  Variation in diagnostic coding of patients with pneumonia and its association with hospital risk-standardized mortality rates: a cross-sectional analysis.  Ann Intern Med. 2014;160(6):380-388.PubMedArticle
22.
Elixhauser  A, Steiner  C, Harris  DR, Coffey  RM.  Comorbidity measures for use with administrative data.  Med Care. 1998;36(1):8-27.PubMedArticle
23.
Martin  GS, Mannino  DM, Eaton  S, Moss  M.  The epidemiology of sepsis in the United States from 1979 through 2000.  N Engl J Med. 2003;348(16):1546-1554.PubMedArticle
24.
Angus  DC, Linde-Zwirble  WT, Lidicker  J, Clermont  G, Carcillo  J, Pinsky  MR.  Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care.  Crit Care Med. 2001;29(7):1303-1310.PubMedArticle
25.
Drye  EE, Normand  SL, Wang  Y,  et al.  Comparison of hospital risk-standardized mortality rates calculated by using in-hospital and 30-day models: an observational study with implications for hospital profiling.  Ann Intern Med. 2012;156(1, pt 1):19-26.PubMedArticle
26.
Eilers  PHC, Marx  BD.  Flexible smoothing with B-splines and penalties.  Stat Sci. 1996;11(2):89-121.Article
27.
Merlo  J, Chaix  B, Ohlsson  H,  et al.  A brief conceptual tutorial of multilevel analysis in social epidemiology: using measures of clustering in multilevel logistic regression to investigate contextual phenomena.  J Epidemiol Community Health. 2006;60(4):290-297.PubMedArticle
28.
Goldstein  H, Browne  W, Rasbash  J.  Multilevel modelling of medical data.  Stat Med. 2002;21(21):3291-3315.PubMedArticle
29.
Seymour  CW, Iwashyna  TJ, Ehlenbach  WJ, Wunsch  H, Cooke  CR.  Hospital-level variation in the use of intensive care.  Health Serv Res. 2012;47(5):2060-2080.PubMedArticle
30.
Centers for Medicare & Medicaid Services.  Hospital quality initiative overview.https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/downloads/HospitalOverview.pdf. Published July 2008. Accessed August 19, 2015.
31.
Centers for Medicare & Medicaid Services.  Hospital consumer assessment of healthcare providers and systems: quality survey.http://www.hcahpsonline.org/home.aspx. Accessed August 17, 2015.
32.
Bayer  W, Mallinger  JB, Krishnan  A, Shields  CG.  Attitudes toward life-sustaining interventions among ambulatory black and white patients.  Ethn Dis. 2006;16(4):914-919.PubMed
33.
Wenger  NS, Pearson  ML, Desmond  KA,  et al.  Epidemiology of do-not-resuscitate orders: disparity by age, diagnosis, gender, race, and functional impairment.  Arch Intern Med. 1995;155(19):2056-2062.PubMedArticle
34.
Chang  DW, Brass  EP.  Patient and hospital-level characteristics associated with the use of do-not-resuscitate orders in patients hospitalized for sepsis.  J Gen Intern Med. 2014;29(9):1256-1262.PubMedArticle
35.
Hart  JL, Harhay  MO, Gabler  NB, Ratcliffe  SJ, Quill  CM, Halpern  SD.  Variability among US intensive care units in managing the care of patients admitted with preexisting limits on life-sustaining therapies.  JAMA Intern Med. 2015;175(6):1019-1026.PubMedArticle
36.
National POLST.  Physician Orders for Life-Sustaining Treatment paradigm. http://www.polst.org/. Accessed April 11, 2014.
37.
Beach  MC, Morrison  RS.  The effect of do-not-resuscitate orders on physician decision-making.  J Am Geriatr Soc. 2002;50(12):2057-2061.PubMedArticle
38.
La Puma  J, Silverstein  MD, Stocking  CB, Roland  D, Siegler  M.  Life-sustaining treatment: a prospective study of patients with DNR orders in a teaching hospital.  Arch Intern Med. 1988;148(10):2193-2198.PubMedArticle
39.
Simpson  EH.  The interpretation of interaction in contingency tables.  J R Stat Soc B Methodol. 1951;13:238-241.
40.
Manktelow  BN, Evans  TA, Draper  ES.  Differences in case-mix can influence the comparison of standardised mortality ratios even with optimal risk adjustment: an analysis of data from paediatric intensive care.  BMJ Qual Saf. 2014;23(9):782-788.PubMedArticle
41.
Marang-van de Mheen  PJ, Shojania  KG.  Simpson’s paradox: how performance measurement can fail even with perfect risk adjustment.  BMJ Qual Saf. 2014;23(9):701-705.PubMedArticle
42.
Feinstein  AR, Sosin  DM, Wells  CK.  The Will Rogers phenomenon: stage migration and new diagnostic techniques as a source of misleading statistics for survival in cancer.  N Engl J Med. 1985;312(25):1604-1608.PubMedArticle
43.
Hall  WB, Willis  LE, Medvedev  S, Carson  SS.  The implications of long-term acute care hospital transfer practices for measures of in-hospital mortality and length of stay.  Am J Respir Crit Care Med. 2012;185(1):53-57.PubMedArticle
Views 3,518
• Cite This

### Citation

Walkey AJ, Weinberg J, Wiener RS, Cooke CR, Lindenauer PK. Association of Do-Not-Resuscitate Orders and Hospital Mortality Rate Among Patients With Pneumonia. JAMA Intern Med. 2016;176(1):97-104. doi:10.1001/jamainternmed.2015.6324

Original Investigation
January 2016

# Association of Do-Not-Resuscitate Orders and Hospital Mortality Rate Among Patients With Pneumonia

Author Affiliations
• 1The Pulmonary Center, Division of Pulmonary and Critical Care Medicine, Boston University School of Medicine, Boston, Massachusetts
• 2Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts
• 3Center for Healthcare Organization & Implementation Research, Edith Nourse Rogers Memorial Veterans Affairs Hospital, Bedford, Massachusetts
• 4Division of Pulmonary and Critical Care Medicine, Department of Medicine, University of Michigan, Ann Arbor
• 5Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor
• 6Center for Quality of Care Research, Division of General Internal Medicine, Baystate Medical Center, Springfield, Massachusetts
• 7Department of Medicine, Tufts University School of Medicine, Boston, Massachusetts
JAMA Intern Med. 2016;176(1):97-104. doi:10.1001/jamainternmed.2015.6324
Abstract

Importance  Hospital quality measures that do not account for patient do-not-resuscitate (DNR) status may penalize hospitals admitting a greater proportion of patients with limits on life-sustaining treatments.

Objective  To evaluate the effect of analytic approaches accounting for DNR status on risk-adjusted hospital mortality rates and performance rankings.

Design, Setting, and Participants  A retrospective, population-based cohort study was conducted among adults hospitalized with pneumonia in 303 California hospitals between January 1 and December 31, 2011. We used hierarchical logistic regression to determine associations between patient DNR status, hospital-level DNR rates, and mortality measures. Changes in hospital risk-adjusted mortality rates after accounting for patient DNR status and interhospital variation in the association between DNR status and mortality were examined. Data analysis was conducted from January 16 to September 16, 2015.

Exposures  Early DNR status (within 24 hours of admission).

Main Outcomes and Measures  In-hospital mortality, determined using hierarchical logistic regression.

Results  A total of 90 644 pneumonia cases (5.4% of admissions) were identified among the 303 California hospitals evaluated during 2011; mean (SD) age of the patients was 72.5 (13.7) years, 51.5% were women, and 59.3% were white. Hospital DNR rates varied (median, 15.8%; 25th-75th percentile, 8.9%-22.3%). Without accounting for patient DNR status, higher hospital-level DNR rates were associated with increased patient mortality (adjusted odds ratio [OR] for highest-quartile DNR rate vs lowest quartile, 1.17; 95% CI, 1.04-1.32), corresponding to worse hospital mortality rankings. In contrast, after accounting for patient DNR status and between-hospital variation in the association between DNR status and mortality, hospitals with higher DNR rates had lower mortality (adjusted OR for highest-quartile DNR rate vs lowest quartile, 0.79; 95% CI, 0.70-0.89), with reversal of associations between hospital mortality rankings and DNR rates. Only 14 of 27 hospitals (51.9%) characterized as low-performing outliers without accounting for DNR status remained outliers after DNR adjustment. Hospital DNR rates were not significantly associated with composite quality measures of processes of care for pneumonia (r = 0.11; P = .052); however, DNR rates were positively correlated with patient satisfaction scores (r = 0.35; P < .001).

Conclusions and Relevance  Failure to account for DNR status may confound the evaluation of hospital quality using mortality outcomes, penalizing hospitals that admit a greater proportion of patients with limits on life-sustaining treatments. Stakeholders should seek to improve methods to standardize and report DNR status in hospital discharge records to allow further assessment of implications of adjusting for DNR in quality measures.

Introduction

Do-not-resuscitate (DNR) orders are usually written by physicians to convey the wishes of patients to not receive cardiopulmonary resuscitation in the event of cardiac arrest. Do-not-resuscitate orders result from complex interactions between patients, physicians, families, and regional and hospital norms13 regarding end-of life care. Unlike DNR orders written later during hospitalization that generally signal a lack of response to therapy, early DNR orders (within 24 hours of hospitalization) largely reflect patients’ baseline burdens of comorbidity and/or prehospitalization preferences for end-of-life care.4 As a marker of baseline severity of illness and patients’ wishes to receive less-aggressive health care interventions, early DNR orders are among the strongest predictors of in-hospital5 and posthospitalization6 mortality. Therefore, early DNR status may result in potent unmeasured confounding effects if unaccounted for during the evaluation of health care delivery.5

Despite the potential for hospital variation in DNR orders to influence patients’ end-of-life experiences and outcomes, DNR status is generally unreported by hospitals and unaccounted for in hospital outcome measures. For example, the Centers for Medicare & Medicaid Services (CMS) do not adjust for DNR status in pneumonia Hospital Compare analyses, reasoning that adjustment for DNR status may be unnecessary owing to potential correlation with other comorbidity measures.7 Some8 have argued for exclusion of DNR status from hospital comparisons because of weaker associations between DNR orders and mortality at hospitals with higher DNR rates, characteristics that may bias conventional regression analyses by assigning the same strength of adjustment to DNR status across all hospitals. However, others suggest that not accounting for variation in patients’ treatment preferences may incentivize provision of care that is not patient centered9 and may confound evaluation of hospital quality based on mortality rates.10,11

To our knowledge, studies have not yet evaluated associations between measures of comorbidity and DNR rates in hospital comparisons, how alternative analytic approaches1215 that allow associations between DNR status and mortality to vary across hospitals might alter hospital quality rankings, or how nonmortality measures of hospital quality relate to hospital DNR rates. We sought to examine associations between hospital DNR rates and measures of hospital quality and to determine how analyses that account for associations between hospital DNR rates, patient DNR status, and mortality may influence hospital mortality rankings. We hypothesized that accounting for the DNR status of individual patients in evaluation of hospital outcomes would result in changes to hospital mortality rankings with potential policy implications. We studied patients hospitalized with pneumonia, which is a frequent target of hospital quality evaluation.16

Methods
Cohort

A detailed description of the methods is presented in the eMethods in the Supplement. The study was approved by the Boston University Medical Center institutional review board as exempt from review, and all procedures were performed using deidentified data. Using the Healthcare Cost and Utilization Project California State Inpatient Database (CA SID),17 we analyzed a cohort of adults 40 years or older hospitalized with pneumonia from January 1 to December 31, 2011. The CA SID consists of administrative claims data from all nonfederal hospitalizations in California and, rather uniquely, contains a validated variable that captures DNR orders written during the first 24 hours of hospitalization.18,19 We defined pneumonia hospitalizations using either principal diagnosis claims for pneumonia or principal diagnosis for acute respiratory failure or septicemia, with a secondary diagnosis of pneumonia that was present on admission (eTable 1 in the Supplement).20,21 Patients transferred from another hospital, discharged alive within 48 hours,16 or admitted to a hospital caring for fewer than 25 patients with pneumonia in 2011 were excluded.16

Early DNR Measures and Covariates

We characterized both patient-level early DNR status and hospital-level early DNR rates. Hospital DNR rate was defined as the percentage of patients with pneumonia with an early DNR order at each hospital. Risk-adjusted models were created using patient demographics, comorbidities,22 and acute organ failure present on admission (eTable 2 in the Supplement).23,24

Outcomes

The associations between patient DNR status and hospital DNR rates with patient in-hospital mortality were examined. Because between-hospital differences in discharge practices may be associated with in-hospital mortality rates,25 we also assessed associations of hospital DNR rates with hospital discharge destinations and length of stay among survivors.

Statistical Analysis

Summary data across quartiles of hospital DNR rates were determined. To examine potential confounding by patient DNR status, contingency tables and adjusted regression models examining associations between hospital DNR rates and mortality were stratified by patient DNR status. We used hierarchical logistic regression models with hospital-level random intercepts to examine multivariable-adjusted associations between hospital DNR rates and patient mortality without accounting for patient DNR status.

Because accounting for between-hospital variation in associations between hospital DNR and hospital mortality rates with a random DNR slope improved model performance (eTable 3 in the Supplement), all models adjusting for patient DNR status also included DNR status as a random slope. Including DNR status as both a fixed effect and a random slope allowed the association between patient DNR status and mortality to vary for each hospital and was intended to attenuate bias that might be introduced if patients at some hospitals elected DNR status at lower levels of illness severity than at other hospitals.

Hospitals were ranked according to each institution’s effect estimate as determined by the random effects.16 Hospitals with a statistically significantly greater risk-standardized mortality compared with the mean were considered to be low-performing outliers. Hospital risk-standardized mortality rates were calculated from the ratio of the hospital risk-adjusted rate to the mean risk-adjusted rate, multiplied by the mean unadjusted hospital rate in California.16 We assessed the correlation between risk-standardized hospital early DNR rates and mortality outcomes quantitatively using Spearman rank correlation coefficients and visually using penalized B-spline regression.26 Differences in risk-standardized hospital DNR rates between hospitals ranked in the highest and lowest risk-standardized mortality quartiles were tested using Wilcoxon-Mann-Whitney tests. We evaluated the degree of hospital-level variation in early DNR use unexplained by measured patient characteristics by calculating intraclass correlation coefficients from hospital-level intercept variance estimates2729 in multivariable-adjusted hierarchical logistic regression models, with early DNR status as the outcome of interest.

Sensitivity Analyses

A sensitivity analysis with a model that adjusted for hospital characteristics (hospital ranking models do not routinely adjust for hospital characteristics) in addition to patient characteristics was conducted. We performed a second sensitivity analysis using methods similar to those of CMS Hospital Compare models and included patients aged 65 years or older with a principal diagnosis of pneumonia, without adjustment for race, income, or acute organ failure.

The association between hospital risk-standardized DNR rates and nonmortality measures of hospital quality were assessed to inform findings regarding changes in hospital rankings after accounting for DNR status. We linked year 2011 CMS Hospital Compare measures of pneumonia processes of care30 and patient satisfaction surveys31 to CA SID hospital data and created composites of hospital quality measures by determining the mean percentage of patients achieving the highest quality rating (eg, measure always performed). We assessed Spearman correlation between hospital DNR rates and hospital quality measures.

All statistical analyses were performed between January 16 and December 16, 2015, using SAS version, 9.3 (SAS Institute Inc). A 2-tailed α level of .05 was considered statistically significant for all analyses.

Results

We identified 90 644 pneumonia cases (5.4% of admissions) among 303 California hospitals. Patients with pneumonia were a mean (SD) age of 72.5 (13.7) years, 51.5% were women, 59.3% were white, and 12.6% died in the hospital. Table 1 summarizes the demographic and clinical characteristics of patients hospitalized with pneumonia. Patients admitted to hospitals in the highest DNR rate quartile were older, more likely to be white, and more likely to be women and had higher median household incomes compared with those admitted to low DNR quartile hospitals. Although the distribution of specific comorbid conditions and acute organ failure (eTable 4 in the Supplement) differed based on hospital DNR rates, patients at high DNR quartile hospitals had a similar number of comorbidities and fewer cases of acute organ failure compared with patients at low DNR quartile hospitals.

Hospital Variation in Early DNR Rates

The median hospital early DNR rate was 15.8% (25th-75th percentile, 8.9%-22.3%) (Figure 1 and eFigure 1 in the Supplement). The proportion of between-hospital variation for DNR orders unexplained by measured clinical covariates was 21% (95% CI, 19%-25%).

Early DNR Orders and Patient Outcomes

As expected, patients who received early DNR orders had a higher risk of mortality (adjusted odds ratio, 3.74; 95% CI, 3.55-3.93). Similarly, without accounting for patient DNR status, hospitals with higher DNR rates had higher patient mortality rates (Table 2 and Figure 2A). However, after stratification by patient DNR status, hospital DNR rates were inversely associated with mortality among patients without DNR orders (Table 2 and Figure 2B) and among patients with DNR orders (Table 2 and Figure 2C). Results of the multivariable-adjusted random hospital intercept with the random DNR coefficient model were similar to analyses stratified by DNR status, showing an inverse association between hospital DNR rates and patient mortality (Figure 2D) (adjusted OR for highest-quartile DNR rate vs lowest-quartile rate, 0.79; 95% CI, 0.70-0.89). Hospitals in the highest quartile of DNR rate had lower rates of discharge to long-term care facilities (28% vs 35%) and greater rates of discharge home (59% vs 53%) but shorter lengths of stay (median [25th-75th percentile], 5 [3-7] vs 6 [4-9] days) compared with hospitals with the lowest DNR rate.

Early DNR Orders and Hospital Mortality Rankings

In a model that did not account for patient DNR status, 27 hospitals were identified as significant low-performing outliers (greatest mortality), and hospitals ranked in the highest-performing quartile (lowest mortality) had lower median (25th-75th percentile) DNR rates (11.6% [6.8%-19.1%]) than the lowest-performing quartile (19.1% [12.3%-27.9%]) (P < .001). After accounting for patient DNR status, 20 hospitals were identified as significant low-performing outliers and 14 (51.9%) outlier hospitals were shared with the DNR-unadjusted analysis. Differences in median (25th-75th percentile) hospital DNR rates between hospitals ranked in the highest-performing quartile (17.7% [10.6%-24.0%]) and the lowest-performing quartile (12.1% [6.2%-21.0%]) (P = .01) were reversed after accounting for DNR status and between-hospital variation in associations between hospital DNR and hospital mortality rates. Changes in hospital mortality rates before and after adjusting for DNR status are shown in Figure 3; differences in effect estimates of model covariates are presented in eTable 2 in the Supplement.

Sensitivity Analyses

Small hospitals, nonteaching hospitals, and nonprofit hospitals were more likely to have high DNR rates (eTable 4 in the Supplement); however, adjusting for hospital characteristics did not substantively change our results. Similarly, an analysis that more closely approximated CMS Hospital Compare models (n = 34 745) showed findings comparable to those of our primary models (eTable 5 in the Supplement). Hospital DNR rates were not significantly associated with composite quality measures of hospital processes of care for pneumonia (r = 0.11; P = .052) (eFigure 2 in the Supplement); however, DNR rates were positively correlated with patient satisfaction scores (r = 0.35; P < .001) (eFigure 3 in the Supplement).

Discussion

Using a population-based cohort of adult pneumonia hospitalizations from California, we found a large variation in early DNR rates across hospitals. The DNR rates were strongly associated with patient demographics (eg, age, race/ethnicity, and median income) rather than measures of increased comorbidity burden. Without accounting for patient DNR status, higher hospital DNR rates appeared to be associated with lower survival, resulting in hospitals with higher DNR rates receiving worse mortality quality rankings. However, when risk adjustment accounted for patient DNR status, hospitals with higher DNR rates had better survival. Furthermore, hospitals with higher DNR rates had better patient satisfaction with similar quality of pneumonia processes of care. Our findings suggest that current methods of comparing hospitals, which do not account for patient DNR status, penalize potentially high-quality hospitals admitting a larger proportion of patients who had chosen to forego resuscitation. Therefore, accounting for DNR status in programs that compare hospital mortality outcomes may substantially affect publicly reportable hospital rankings and hospital reimbursements.

Our findings are most comparable with those of Tabak et al,8 Kelly et al,10 and Escobar et al,11 who examined associations between hospital variation in DNR orders with hospital mortality in other conditions. Although all studies found substantial changes in hospital mortality rankings after adjusting for early DNR status, conclusions differed. Tabak et al8 argued against using DNR status in hospital risk adjustment owing to potential bias introduced by applying the mean mortality rate associated with DNR status to an entire cohort when DNR-associated mortality varied between hospitals. Kelly et al10 found similar changes to hospital mortality rankings among patients with stroke. Escobar et al11 used standardized DNR orders and detailed physiologic data for patients hospitalized among 21 institutions in an integrated health care delivery system and concluded that adjustment for advance directives was “desirable” when DNR orders could be standardized. Standardized DNR reporting in the small hospital sample investigated by Escobar et al did not result in inverse associations between hospital DNR rates and mortality among DNR patients. Our methods differed from those of prior studies, focusing on pneumonia, using models that attenuated potential bias introduced from uniformly adjusting for less-standardized DNR orders across hospitals, and expanding hospital quality assessments beyond mortality. We advocate for further studies, including in other cohorts, to explore methods to accurately identify early DNR status from hospital discharge data and examine methodologies to account for variables, such as DNR status, that demonstrate variable associations with outcomes across hospitals.

Wide variation in early DNR orders between hospitals may be the result of a combination of patient preferences and hospital practices. Patient preferences may vary based on beliefs regarding end-of-life care; clustering of patients with similar beliefs may be reflected in a hospital’s DNR rate. As suggested in prior studies5,3234 showing greater rates of DNR orders among white patients, we identified that hospitals with high DNR rates tended to admit more white patients with higher median incomes but without greater measures of comorbidity or acute illness severity. These findings suggest that socioeconomic and cultural factors result in different thresholds for choosing to forgo resuscitation. Currently, CMS models evaluating hospital quality do not adjust for race or socioeconomic status. Because hospital DNR rates are strongly associated with patient socioeconomic characteristics, additional studies are required to assess the potential ramifications of including DNR status in hospital ranking models.

Hospital practice norms may also influence when and how discussions of advance directives are initiated1 and the extent to which advance directives are carried out.3,35 Physicians and patients may define DNR differently, ranging from strict comfort measures only to care restricting only cardiopulmonary resuscitation.3638 The inverse association between hospital DNR rates and acute organ failure supports a hypothesis that hospitals may vary in thresholds for initiating advance directive discussions, resulting in a lower level of acute severity of illness among patients with DNR orders at hospitals with high DNR rates. We demonstrated methods that may account for differences in DNR-associated care processes while more appropriately adjusting for previously unmeasured patient characteristics associated with DNR status.

The reversal in direction of the association between hospital DNR rates and patient mortality after accounting for patient DNR status demonstrates the Simpson paradox39 in a hospital quality measure. The Simpson paradox is a type of confounding that occurs when an association seen in an aggregate sample reverses in stratified analysis; it is generally caused by 2 factors: (1) a strong imbalance in distribution of exposures between subgroups and (2) imbalanced outcomes within the subgroup. In the present study, variation in hospital DNR rates and differing associations between patient DNR status and mortality across hospitals likely contributed to the Simpson paradox. Although simulation studies40,41 have theorized that the Simpson paradox may affect hospital quality measures, we are unaware of studies demonstrating the Simpson paradox occurring in a real-world performance measure.

Others may consider the reversal of direction in the association between hospital mortality and DNR rates after stratifying patients by DNR status as a Will Rogers phenomenon.42 This phenomenon may result from the reclassification of patients by DNR status because more marginal patients with a lower mortality risk likely had DNR orders at high–DNR rate hospitals (diluting the DNR pool with patients who were less ill), but these patients with DNR orders (who have greater-than-average risk of mortality) were redistributed out of the no-DNR pool at high–DNR rate hospitals. However, accounting for DNR status in models including all patients showed findings similar to the stratified analysis, suggesting that Will Rogers phenomenon alone does not fully account for our findings.

Our results show that variation in DNR rates may strongly affect hospital mortality rankings, even when accounting for between-hospital differences in the association between DNR status and mortality, and underscore the potential susceptibility of hospital comparison models to unmeasured factors. Concordance between hospital DNR rates, hospital mortality rates (after adjusting for patient DNR status), patient satisfaction, and pneumonia processes of care further argue for adjusting to include patient DNR status in hospital mortality measures. From the perspective of DNR status as a characteristic representing a composite of comorbidity, frailty, functional status, and wishes for less-aggressive care, our findings suggest that early DNR status is a confounder that should be accounted for in hospital mortality rankings (ie, the Simpson paradox). From the perspective that DNR orders also result from variable hospital norms and practice patterns that elicit advance directives, facilities that more actively seek to determine patient resuscitation wishes may differentially recategorize marginal patients by DNR status and alter mortality rankings (ie, Will Rogers phenomenon). Thus, before adopting DNR status into hospital comparison models, future studies should seek to determine the extent to which hospital-level DNR rates represent patient-driven phenomena or are determined by hospital processes norms in how advance directive discussions are approached.

Several strengths and limitations of our study should be considered in evaluating the findings. Although we sought to pattern our analysis after current CMS methods for hospital comparisons,16 our primary analysis differs in many ways from CMS models; however, the results did not substantially differ in sensitivity analysis, more closely approximating the CMS approach. It is possible that other unmeasured variables, such as functional status, disability, or prior hospitalizations, may also influence associations between DNR status and outcomes. In addition, early DNR measures in CA SID show good accuracy, but the potential for misclassification bias cannot be ruled out. Although use of a random slope method improved model performance and attenuated bias from variation in DNR and mortality associations across hospitals, residual bias in favor of high–DNR rate hospitals may not be eliminated. Thus, further work to standardize DNR reporting across hospitals may be needed before adopting DNR status into hospital quality comparison models. We were able to measure hospital (rather than 30-day) mortality, an outcome that may be affected by hospital discharge patterns to long-term acute care hospitals and length of stay.25,43 Because patients admitted to high–DNR rate hospitals were less likely to be discharged to long-term care facilities but had shorter lengths of stay, large bias in the difference between 30-day and hospital mortality resulting from differences in discharge practices is unlikely.11 Because event rates for mortality were not rare, odds ratios from our models may not approximate risk ratios. Finally, although we anticipate large hospital variation in DNR orders across the United States, generalization of our results to regions outside of California requires further study.

Conclusions

We have shown that failure to account for between-hospital variation in the rates of patients with limitations on life-sustaining treatments may confound the evaluation of hospital mortality outcomes. We describe a method for attenuating potential bias in efforts to measure the performance of hospitals that may be introduced by the Simpson paradox between hospital DNR rates and DNR-associated mortality. Our findings have potentially serious ramifications for methods used to assess patient outcomes and hospital quality. Given the proliferation of health care quality measures and the importance of accounting for confounding by early DNR status in the evaluation of outcomes, stakeholders should expedite efforts to standardize and report patient-level early DNR status in discharge data and pursue further studies to evaluate the ramifications of adjusting for DNR status in hospital quality measures.

Article Information

Corresponding Author: Allan J. Walkey, MD, MSc, The Pulmonary Center, Division of Pulmonary and Critical Care Medicine, Boston University School of Medicine, 72 E Concord St, R-304, Boston, MA 02118 (alwalkey@bu.edu).

Accepted for Publication: September 24, 2015.

Published Online: December 14, 2015. doi:10.1001/jamainternmed.2015.6324.

Author Contributions: Dr Walkey had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Walkey, Cooke.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Walkey.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Walkey, Weinberg.

Obtained funding: Walkey.

Administrative, technical, or material support: Cooke.

Study supervision: Lindenauer.

Conflict of Interest Disclosures: None reported.

Funding/Support: The study was funded by grant K01HL116768 from the National Institutes of Health (NIH) National Heart, Lung, and Blood Institute (Dr Walkey), grant K08HS020672 from the Agency for Healthcare Research and Quality (Dr Cooke), and grant K07 CA138772 from the NIH (Dr Wiener). This study was also supported by resources from the Edith Nourse Rogers Memorial Veterans Affairs Hospital (Dr Wiener) and a Career Investment Award from the Boston University School of Medicine Department of Medicine (Dr Walkey).

Role of the Funder/Sponsor: The funding agencies had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Barnato  AE, Mohan  D, Lane  RK,  et al.  Advance care planning norms may contribute to hospital variation in end-of-life ICU use: a simulation study.  Med Decis Making. 2014;34(4):473-484.PubMedArticle
2.
Dzeng  E, Colaianni  A, Roland  M,  et al.  Influence of institutional culture and policies on do-not-resuscitate decision making at the end of life.  JAMA Intern Med. 2015;175(5):812-819. PubMedArticle
3.
Zingmond  DS, Wenger  NS.  Regional and institutional variation in the initiation of early do-not-resuscitate orders.  Arch Intern Med. 2005;165(15):1705-1712.PubMedArticle
4.
Marrie  TJ, Fine  MJ, Kapoor  WN, Coley  CM, Singer  DE, Obrosky  DS.  Community-acquired pneumonia and do not resuscitate orders.  J Am Geriatr Soc. 2002;50(2):290-299.PubMedArticle
5.
Bradford  MA, Lindenauer  PK, Wiener  RS, Walkey  AJ.  Do-not-resuscitate status and observational comparative effectiveness research in patients with septic shock.  Crit Care Med. 2014;42(9):2042-2047.PubMedArticle
6.
Baldwin  MR, Narain  WR, Wunsch  H,  et al.  A prognostic model for 6-month mortality in elderly survivors of critical illness.  Chest. 2013;143(4):910-919.PubMedArticle
7.
National Quality Forum.  Pulmonary and critical care consensus standards endorsement maintenance: technical report: NQF pulmonary comment period: AHA letter points for 30-day COPD and pneumonia measures.http://www.qualityforum.org/Publications/2013/04/Pulmonary_and_Critical_Care_Consensus_Standards_Endorsement_Maintenance.aspx. Published April 2013. Accessed April 23, 2015.
8.
Tabak  YP, Johannes  RS, Silber  JH, Kurtz  SG.  Should Do-Not-Resuscitate status be included as a mortality risk adjustor? the impact of DNR variations on performance reporting.  Med Care. 2005;43(7):658-666.PubMedArticle
9.
Holloway  RG, Quill  TE.  Mortality as a measure of quality: implications for palliative and end-of-life care.  JAMA. 2007;298(7):802-804.PubMedArticle
10.
Kelly  AG, Zahuranec  DB, Holloway  RG, Morgenstern  LB, Burke  JF.  Variation in do-not-resuscitate orders for patients with ischemic stroke: implications for national hospital comparisons.  Stroke. 2014;45(3):822-827.PubMedArticle
11.
Escobar  GJ, Gardner  MN, Greene  JD, Draper  D, Kipnis  P.  Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system.  Med Care. 2013;51(5):446-453.PubMedArticle
12.
Localio  AR, Berlin  JA, Ten Have  TR, Kimmel  SE.  Adjustments for center in multicenter studies: an overview.  Ann Intern Med. 2001;135(2):112-123.PubMedArticle
13.
Finucane  MM, Samet  JH, Horton  NJ.  Translational methods in biostatistics: linear mixed effect regression models of alcohol consumption and HIV disease progression over time.  Epidemiol Perspect Innov. 2007;4:8.PubMedArticle
14.
Agresti  A, Hartzel  J.  Strategies for comparing treatments on a binary response with multi-centre data.  Stat Med. 2000;19(8):1115-1139.PubMedArticle
15.
Gould  AL.  Multi-centre trial analysis revisited.  Stat Med. 1998;17(15-16):1779-1797.PubMedArticle
16.
Bratzler  DW, Normand  SL, Wang  Y,  et al.  An administrative claims model for profiling hospital 30-day mortality rates for pneumonia patients.  PLoS One. 2011;6(4):e17401.PubMedArticle
17.
State Inpatient Database.  California: Healthcare Cost and Utilization Project: overview of the state inpatient databases. Rockville, MD: Agency for Healthcare Research and Quality; 2011. http://www.hcup-us.ahrq.gov/sidoverview.jsp. Published 2011. Accessed January 16, 2015.
18.
Goldman  LE, Chu  PW, Prothro  C, Osmond  D, Bindman  AB, eds.  Accuracy of condition present on admission, do not resuscitate, and e-codes in California Patient Discharge Data: prepared for the Office of Statewide Health Planning and Development, Healthcare Outcomes Center.http://www.oshpd.ca.gov/HID/Products/PatDischargeData/ResearchReports/PDDValidation/PDD_Validation_Study.pdf. Published Spring 2011. Accessed June 27, 2015.
19.
Goldman  LE, Chu  PW, Osmond  D, Bindman  A.  Accuracy of do not resuscitate (DNR) in administrative data.  Med Care Res Rev. 2013;70(1):98-112.PubMedArticle
20.
Walkey  AJ, Lagu  T, Lindenauer  PK.  Trends in sepsis and infection sources in the United States: a population-based study.  Ann Am Thorac Soc. 2015;12(2):216-220.PubMedArticle
21.
Rothberg  MB, Pekow  PS, Priya  A, Lindenauer  PK.  Variation in diagnostic coding of patients with pneumonia and its association with hospital risk-standardized mortality rates: a cross-sectional analysis.  Ann Intern Med. 2014;160(6):380-388.PubMedArticle
22.
Elixhauser  A, Steiner  C, Harris  DR, Coffey  RM.  Comorbidity measures for use with administrative data.  Med Care. 1998;36(1):8-27.PubMedArticle
23.
Martin  GS, Mannino  DM, Eaton  S, Moss  M.  The epidemiology of sepsis in the United States from 1979 through 2000.  N Engl J Med. 2003;348(16):1546-1554.PubMedArticle
24.
Angus  DC, Linde-Zwirble  WT, Lidicker  J, Clermont  G, Carcillo  J, Pinsky  MR.  Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care.  Crit Care Med. 2001;29(7):1303-1310.PubMedArticle
25.
Drye  EE, Normand  SL, Wang  Y,  et al.  Comparison of hospital risk-standardized mortality rates calculated by using in-hospital and 30-day models: an observational study with implications for hospital profiling.  Ann Intern Med. 2012;156(1, pt 1):19-26.PubMedArticle
26.
Eilers  PHC, Marx  BD.  Flexible smoothing with B-splines and penalties.  Stat Sci. 1996;11(2):89-121.Article
27.
Merlo  J, Chaix  B, Ohlsson  H,  et al.  A brief conceptual tutorial of multilevel analysis in social epidemiology: using measures of clustering in multilevel logistic regression to investigate contextual phenomena.  J Epidemiol Community Health. 2006;60(4):290-297.PubMedArticle
28.
Goldstein  H, Browne  W, Rasbash  J.  Multilevel modelling of medical data.  Stat Med. 2002;21(21):3291-3315.PubMedArticle
29.
Seymour  CW, Iwashyna  TJ, Ehlenbach  WJ, Wunsch  H, Cooke  CR.  Hospital-level variation in the use of intensive care.  Health Serv Res. 2012;47(5):2060-2080.PubMedArticle
30.
Centers for Medicare & Medicaid Services.  Hospital quality initiative overview.https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/downloads/HospitalOverview.pdf. Published July 2008. Accessed August 19, 2015.
31.
Centers for Medicare & Medicaid Services.  Hospital consumer assessment of healthcare providers and systems: quality survey.http://www.hcahpsonline.org/home.aspx. Accessed August 17, 2015.
32.
Bayer  W, Mallinger  JB, Krishnan  A, Shields  CG.  Attitudes toward life-sustaining interventions among ambulatory black and white patients.  Ethn Dis. 2006;16(4):914-919.PubMed
33.
Wenger  NS, Pearson  ML, Desmond  KA,  et al.  Epidemiology of do-not-resuscitate orders: disparity by age, diagnosis, gender, race, and functional impairment.  Arch Intern Med. 1995;155(19):2056-2062.PubMedArticle
34.
Chang  DW, Brass  EP.  Patient and hospital-level characteristics associated with the use of do-not-resuscitate orders in patients hospitalized for sepsis.  J Gen Intern Med. 2014;29(9):1256-1262.PubMedArticle
35.
Hart  JL, Harhay  MO, Gabler  NB, Ratcliffe  SJ, Quill  CM, Halpern  SD.  Variability among US intensive care units in managing the care of patients admitted with preexisting limits on life-sustaining therapies.  JAMA Intern Med. 2015;175(6):1019-1026.PubMedArticle
36.
National POLST.  Physician Orders for Life-Sustaining Treatment paradigm. http://www.polst.org/. Accessed April 11, 2014.
37.
Beach  MC, Morrison  RS.  The effect of do-not-resuscitate orders on physician decision-making.  J Am Geriatr Soc. 2002;50(12):2057-2061.PubMedArticle
38.
La Puma  J, Silverstein  MD, Stocking  CB, Roland  D, Siegler  M.  Life-sustaining treatment: a prospective study of patients with DNR orders in a teaching hospital.  Arch Intern Med. 1988;148(10):2193-2198.PubMedArticle
39.
Simpson  EH.  The interpretation of interaction in contingency tables.  J R Stat Soc B Methodol. 1951;13:238-241.
40.
Manktelow  BN, Evans  TA, Draper  ES.  Differences in case-mix can influence the comparison of standardised mortality ratios even with optimal risk adjustment: an analysis of data from paediatric intensive care.  BMJ Qual Saf. 2014;23(9):782-788.PubMedArticle
41.
Marang-van de Mheen  PJ, Shojania  KG.  Simpson’s paradox: how performance measurement can fail even with perfect risk adjustment.  BMJ Qual Saf. 2014;23(9):701-705.PubMedArticle
42.
Feinstein  AR, Sosin  DM, Wells  CK.  The Will Rogers phenomenon: stage migration and new diagnostic techniques as a source of misleading statistics for survival in cancer.  N Engl J Med. 1985;312(25):1604-1608.PubMedArticle
43.
Hall  WB, Willis  LE, Medvedev  S, Carson  SS.  The implications of long-term acute care hospital transfer practices for measures of in-hospital mortality and length of stay.  Am J Respir Crit Care Med. 2012;185(1):53-57.PubMedArticle