Comparison of the Value of Nursing Work Environments in Hospitals Across Different Levels of Patient Risk

Importance  The literature suggests that hospitals with better nursing work environments provide better quality of care. Less is known about value (cost vs quality).

Objectives  To test whether hospitals with better nursing work environments displayed better value than those with worse nursing environments and to determine patient risk groups associated with the greatest value.

Design, Setting, and Participants  A retrospective matched-cohort design, comparing the outcomes and cost of patients at focal hospitals recognized nationally as having good nurse working environments and nurse-to-bed ratios of 1 or greater with patients at control group hospitals without such recognition and with nurse-to-bed ratios less than 1. This study included 25 752 elderly Medicare general surgery patients treated at focal hospitals and 62 882 patients treated at control hospitals during 2004-2006 in Illinois, New York, and Texas. The study was conducted between January 1, 2004, and November 30, 2006; this analysis was conducted from April to August 2015.

Exposures  Focal vs control hospitals (better vs worse nursing environment).

Main Outcomes and Measures  Thirty-day mortality and costs reflecting resource utilization.

Results  This study was conducted at 35 focal hospitals (mean nurse-to-bed ratio, 1.51) and 293 control hospitals (mean nurse-to-bed ratio, 0.69). Focal hospitals were larger and more teaching and technology intensive than control hospitals. Thirty-day mortality in focal hospitals was 4.8% vs 5.8% in control hospitals (P < .001), while the cost per patient was similar: the focal-control was −$163 (95% CI = −$542 to $215; P = .40), suggesting better value in the focal group. For the focal vs control hospitals, the greatest mortality benefit (17.3% vs 19.9%; P < .001) occurred in patients in the highest risk quintile, with a nonsignificant cost difference of $941 per patient ($53 701 vs $52 760; P = .25). The greatest difference in value between focal and control hospitals appeared in patients in the second-highest risk quintile, with mortality of 4.2% vs 5.8% (P < .001), with a nonsignificant cost difference of −$862 ($33 513 vs $34 375; P = .12).

Conclusions and Relevance  Hospitals with better nursing environments and above-average staffing levels were associated with better value (lower mortality with similar costs) compared with hospitals without nursing environment recognition and with below-average staffing, especially for higher-risk patients. These results do not suggest that improving any specific hospital’s nursing environment will necessarily improve its value, but they do show that patients undergoing general surgery at hospitals with better nursing environments generally receive care of higher value.

Past studies have shown that hospitals with excellent nursing environments, as confirmed in a national peer-assessed recognition program, have lower mortality^1-5 and lower failure-to-rescue rates,⁶ yet others have reported unclear patient cost and revenue benefits associated with hospitals known to have good nursing work environments.^7,8

This study asks whether selecting hospitals based solely on excellent nursing environments (defined by having both national peer-assessed recognition and above-average nurse staffing) identifies a set of hospitals that display better outcomes and value, a question most relevant to a patient seeking advice on where to go for care. Our approach was different from previous studies. We did not ask whether a specific hospital would benefit from improving its nursing environment, a question relevant to an administrator capable of changing the environment of the hospital. Therefore, we purposely did not match on individual hospital characteristics, instead seeking to compare 2 groups of hospitals with very different nursing environments but very similar patients and allowing other hospital characteristics to vary naturally with the 2 groups.

Furthermore, by closely matching pairs of patients from hospitals with better and worse nursing environments, we explored whether better nursing environments especially benefit patients of higher initial risk.

The data set comprised Medicare fee-for-service claims for elderly patients admitted for general surgery in Illinois, New York, and Texas from 2004-2006. We acquired the following files: the Master Beneficiary Summary File, inpatient claims, outpatient claims, and Carrier/Part B bills. The study was conducted between January 1, 2004, and November 30, 2006; this analysis was conducted from April to August 2015. This research protocol was reviewed by the Children’s Hospital of Philadelphia institutional review board and judged to be non–human subjects research, and informed consent was not required.

Patient characteristics were defined using the index admission and a 90-day look-back in all utilization files. Variables included patient age, year of admission, sex, race, emergency department admission status, transfer-in status, and 31 comorbidities (eAppendix 1 in the Supplement). Patient probability of 30-day death was estimated by a model fit to an external data set that was not used for matching (eAppendix 2 in the Supplement). We estimated a propensity score using all of the matching covariates^9,10 for attending a hospital with a good work environment. We also required an exact match within pairs on all 4-digit principal procedure codes (N = 130; see eAppendix 3 in the Supplement for the complete list).

We defined each hospital’s nursing environment using the 2007 list of a national voluntary accreditation program for nursing environment excellence that has been found by many studies to identify hospitals with significantly better nursing environments.^2,11 Each hospital’s nurse-to-bed (NTB) ratio, resident-to-bed ratio, nurse mix, technology level, and number of beds were determined using the Medicare Provider of Service file. The NTB ratio was defined by dividing the number of full-time–equivalent registered nurses and licensed practical nurses by the number of total beds. Likewise, the resident-to-bed ratio was defined by dividing the number of residents by the number of total beds. Nurse mix was the proportion of registered nurses among the total number of registered nurses and licensed practical nurses. Technology level was considered high by the presence of a burn unit or the provision of coronary artery bypass graft surgery or organ transplantation.^12,13

Thirty-day mortality was our primary quality-related outcome. We also report in-hospital mortality, in-hospital and 30-day complications (38 common complications that occur after surgery, as defined in previous work; eAppendix 4 in the Supplement),^14-17 in-hospital and 30-day failure to rescue,^14,18 all-cause readmissions within 30 days of discharge, length of stay, and intensive care unit (ICU) use.

We used 2 approaches to assess economic performance: costs and Centers for Medicare and Medicaid Services (CMS) payments. Our primary metric was 30-day cost. We calculated each patient’s in-hospital and in-hospital plus 30-day costs (hereafter referred to as 30-day costs) based on resource utilization.^15,19 As in previous studies,^15,16 in-hospital costs accounted for any resources used for the patient’s care during the period of the index hospitalization. Thirty-day costs included in-hospital costs, plus any emergency department, outpatient visit, or office visit costs, as well as any costs arising from a rehospitalization that began within 30 days of the index admission date (counting all costs from the entirety of the readmission, including beyond 30 days). Our costing function was based on data available in Medicare claims. Cost was a function of days in the hospital and level of care (ICU vs floor) for each day, total relative value units determined from all bills, all procedures for which a bill was identified and charged to CMS (including operating room cost and anesthesia), and any bill observed using the description provided here. Finally, we added an estimate of costs directly associated with above- or below-average NTB ratio. The costing algorithm used salary data from the Bureau of Labor and Statistics, adjusted for fringe benefits²⁰ to create adjusted costs reflecting the hospital’s positive or negative deviation from the average NTB ratio, and assigned to each patient an additional cost or cost reduction reflecting the extra or reduced nursing costs per day multiplied by days spent on the general floor (eAppendix 5 in the Supplement). We also report a second cost metric (cost without NTB adjustment) that did not include adjustments to cost based on differing NTB ratios.

Another approach to evaluating value was through Medicare payments associated with the hospital admission (eAppendix 5 and eAppendix 6 in the Supplement). We report payments using 2 definitions. One included all the payments provided by CMS. A second definition omitted the geography adjustment (because we did not want possibly different pricing environments between focal and control hospitals to confound the comparisons of payments) and the indirect medical expenditure adjustment (because we did not want hospitals with an increased educational burden to be penalized for teaching when comparing payments).

Each focal patient was treated at a hospital recognized nationally as having a good nurse working environment and an NTB ratio of 1 or greater and was matched to a control patient treated at a hospital without such recognition and with an NTB ratio less than 1. The optimal match²¹ was calculated using the ASSIGN procedure in SAS (SAS Institute).²² Our algorithm exactly matched 1 of 130 procedures inside each pair and then attempted to balance 42 patient covariates by minimizing the Mahalanobis distance^15,23-26 between cases and control patients, including age, year of admission, sex, race, emergency admission status, transfer-in status, the propensity score, the risk score, and 31 comorbidities (eAppendix 1 in the Supplement).

Matches were performed first without viewing outcomes.^27,28 We aimed to attain standardized differences in covariate means below 0.1. We also assessed balance using Fisher exact test for binary covariates²⁹ and Wilcoxon rank sum test for continuous ones.³⁰

Outcomes were compared using paired methods: for binary outcomes, McNemar test³¹; for continuous outcomes, m-statistics,^32-35 including the permutational t test.^32,33,36 We also used the jackknife procedure to explore the potential effect of hospital-level clustering on reported P values.^37,38

Using a data set not overlapping with our matched sample (eAppendix 2 in the Supplement), we constructed a 30-day mortality model to calculate each study patient’s mortality probability. After matching, we ranked each matched pair by its average risk of mortality, forming quintiles of increasing risk, and compared outcomes between focal and control patients inside each quintile. Graphs of focal-control outcome differences by risk level were produced using LOWESS in R,³⁹ its pointwise 95% bootstrap CI,⁴⁰ and 95% joint confidence ellipse for Hotelling T².^41-44

We identified 172 225 patients who underwent general surgery in the 3 states in 606 short-term, acute-care hospitals. The focal group had 25 752 patients in 35 hospitals recognized nationally as having both good nurse working environments and NTB ratios of 1 or greater (mean NTB ratio, 1.51). Matched controls were drawn from a cohort of 62 882 patients treated at 298 nonrecognized hospitals with NTB ratios below 1 (mean NTB ratio, 0.69).

Focal hospitals with excellent nursing environments differed from controls in many ways, as seen in Table 1. For example, 21.5% of patients (n = 5400) in the focal group attended hospitals that were major teaching hospitals with resident-to-bed ratios above 0.25 compared with 5.7% of matched control patients (n = 1420). More focal patients attended hospitals that had high-level technology available (21 823 patients [87.0%] vs 14 827 [59.1%]), and more focal patients attended large hospitals, as measured by bed size greater than 250 patients (22 286 patients [88.9%] vs 15 934 [63.5%]) (see eAppendix 7 in the Supplement for the characteristics of the hospitals where focal and control patients were treated).

Using the 25 752 general surgery patients treated in the 35 focal hospitals, we formed 25 076 pairs matched exactly for the 130 surgical procedures (97.4% of the available focal patients), with 293 of the 298 available control hospitals represented in the match. Table 2 displays some of the variables used in the match. All 130 principal procedures were matched exactly, and all other patient covariates (n = 42) were balanced, with no standardized difference after matching exceeding 0.05 SD. See eAppendix 8 in the Supplement for complete details of this extremely balanced match, including frequencies of principal procedure codes.

Table 3 compares outcomes of focal patients and matched control patients. Focal patients had lower 30-day mortality rates than control patients (4.8% vs 5.8%; odds ratio [OR], 0.79; 95% CI, 0.73-0.86; P < .001; clustered P value = .005) (see eAppendix 9 in the Supplement for jackknife results; see eAppendix 10 in the Supplement for sensitivity analysis results). Focal patients also had lower 30-day failure-to-rescue rates (7.5% vs 8.9%; OR, 0.83; 95% CI, 0.76-0.90; P < .001) and were in the ICU less often (32.9% vs 42.9%; OR, 0.55; 95% CI, 0.52-0.57; P < .001). Length of stay was slightly shorter among focal patients than matched control patients (m-estimate, 8.4 vs 8.6 days; paired difference, −0.1; 95% CI, −0.3 to −0.0; P = .01). Results for in-hospital outcomes were generally similar to 30-day results (eAppendix 11 in the Supplement).

Did better quality cost more? As measured by resource utilization, focal patients had similar in-hospital and 30-day costs per patient as their controls. Thirty-day cost per patient was $27 131 vs $27 292 (focal vs control), a difference of −$163 per patient pair (95% CI, −$542 to $215; P = .40). If we take away the NTB adjustment, we see a focal-control difference of −$2038 per patient pair (P < .001; clustered P value < .001).

Payments from Medicare were higher in focal patients. Estimated 30-day payments for focal patients were $26 091 per patient vs $25 067, a paired difference of $1001 per patient (95% CI, $710 to $1292; P < .001; clustered P value = .30). However, when both the geography payment adjustment and indirect medical expenditure payments were removed, 30-day payment was actually $851 less per focal vs control patient (95% CI, −$1113 to −$589; P < .001; clustered P value = .03). In-hospital payment results were similar to the 30-day results (eAppendix 6 in the Supplement).

The central question of this study asks whether value differences exist across hospitals selected for better or worse nursing environments and NTB ratio, but a different question is the extent to which the nurses themselves are the cause of the value differences. An explanatory variable of interest is teaching status, and all hospital characteristics associated with teaching status. We divided the 25 076 matched pairs into 4 possible combinations of teaching status of hospitals attended by each of the 2 patients in each matched pair (focal vs control): teaching vs teaching, nonteaching vs nonteaching, teaching vs nonteaching, and nonteaching vs teaching. The resulting outcome differences can be seen in Table 4. For 30-day mortality, the focal patient advantage was maintained in all comparison combinations except when the focal patient attended a nonteaching hospital and the control attended a teaching hospital—when the odds of mortality become similar. Thirty-day costs were generally similar in the focal hospitals and controls for all 4 comparisons. Intensive care unit use was consistently less in focal vs control patients, with a very large and significant reduction in the odds of using the ICU compared with matched control patients. To verify the stability of these findings, we repeated the analysis excluding any pairs in which either patient attended a major teaching hospital (resident-to-bed ratio >0.25); as can be seen on the right side of Table 4, the results were similar.

Stratifying the patient pairs by hospital size or technology generally yielded results similar to those of the teaching analysis (eAppendix 12 in the Supplement).

Overall, focal patients had better outcomes with costs similar to those of control patients. Do some types of patients benefit more than others? The right side of Table 5 divides the matched pairs into quintiles based on the predicted risk of 30-day mortality, that is, the risk score that was closely matched in each pair. Focal patients had lower mortality than control patients in all risk quintiles, but the difference was larger and statistically significant among higher-risk patients. In the second-highest risk quintile, mortality was 1.6% lower at focal hospitals than at control hospitals (4.2% vs 5.8%; P < .001), and in the highest-risk quintile, mortality was 2.6% lower at focal hospitals (17.3% vs 19.9%; P < .001). This trend was statistically significant (P < .001). Focal patients had 30-day cost similar to that of controls in all risk quintiles (second-highest quintile: $33 513 vs $34 375 [difference of −$862]; P = .12; highest quintile: $53 701 vs $52 760 [difference of $941]; P = .25), as well as similar lengths of stay. Costs without NTB adjustment were lower at focal hospitals, and ICU use was far lower across all risk quintiles. Figure 1 displays the difference between focal and control matched pairs plotted against the initial risk of each matched pair. For 30-day mortality, focal patients are consistently below the line of equivalence that denotes a 0 difference between the focal minus control patient outcome (Figure 1A). For cost, it appears there is very little difference between groups (Figure 1B). Not adjusting for NTB differences, costs appear lower in the focal group, with savings increasing with risk (Figure 1C).

In Figure 2, we compare value in the matched pairs of patients in the control (worse) and focal (better) nursing environments by patient risk. The x-axis represents the control minus focal paired difference in 30-day costs for each matched pair. The y-axis represents the control minus focal difference in 30-day mortality. The ellipses on these graphs represent the 95% joint confidence region for cost and quality mean differences.

We display 6 ellipses: 5 numbered ones including about 5015 matched sets of patients by risk quintile, and a central ellipse with a centered dot, which is based on all patients (N = 25 076) (see Appendix 13 in the Supplement for further explanation of the size of the ellipses). Ellipses crossing the horizontal axis at 0 suggest no difference in quality. Ellipses crossing the vertical axis at 0 suggest no difference in cost. For Figure 2A, most ellipses are above the horizontal line, suggesting better quality at the focal hospitals (lower mortality than matched controls). At the same time, most ellipses also cross the vertical axis, suggesting no difference in cost. Together, there is a strong case for better value (similar cost with lower mortality) in the focal group compared with the matched controls. In Figure 2B, we see a somewhat different pattern. When we did not adjust for different NTB ratios, the second-highest risk group displayed both significantly better quality and significantly lower resource utilization.

In this study, we defined a hospital with recognized excellent nursing environments if this recognition was achieved either before or including 2007 because it reflects conditions in the hospital in the recent past, as our patients were admitted between 2004 and 2006. To examine this definition more closely, we performed an analysis that excluded the subset of focal patients whose admissions occurred in hospitals that would be certified by 2007 but had not yet been certified by the year the patient was admitted. That is, for a patient to be included in the new analysis, his or her hospital had to be certified by the year the patient was admitted. Our results were unchanged. After exclusions, there were 18 212 matched pairs. The original 30-day mortality OR was 0.79 (95% CI, 0.73-0.86; P < .001; Table 3); after exclusions, the OR was 0.77 (95% CI, 0.68-0.88; P < .001). For 30-day cost, we previously found a difference of −$163 (95% CI, −$542 to $215; P = .40); after exclusions, the difference was −$138 (95% CI, −$584 to $307; P = .54). For ICU use, the previous OR was 0.55 (95% CI, 0.52-0.57; P < .001); after exclusions, the OR was also 0.55 (95% CI, 0.52-0.57; P < .001).

While there is considerable evidence that a better nursing work environment is associated with better outcomes,^1-7 the question of value has remained uncertain. In this study, we asked whether 2 large groups of hospitals, defined only by different nursing environments and NTB ratios, displayed different value. We chose to examine certified hospitals with good NTB ratios because these were 2 well-known and important factors identified with better nursing environment—the accreditation of the hospital with respect to nursing environment and the most common and fundamental nurse staffing variable. Patients and referring physicians can easily observe such characteristics. When examining 30-day mortality and cost, we found that focal patients treated in better nursing environments and NTB ratios greater than or equal to 1 displayed a clear-cut advantage in value over patients treated at control hospitals. Focal patients have lower mortality with similar costs and, therefore, better value.

We also found that while all patients may benefit from hospitals that have a good nursing environment, sicker patients benefit more. Patients in the highest quintile of risk have the largest reduction in mortality rates, but not lower costs, confirming that improved outcomes are possible for high-risk patients but expensive. Patients in the second-highest quintile of risk have substantial reduction in mortality and the largest reduction in cost, producing the highest value.

Focal hospitals also had dramatically lower rates of ICU use. This finding could be consistent with better nursing care on the floor, acting as a substitute for ICU care or other resource utilization for some patients, possibly leading to lower overall resource utilization and contributing to the business case for improving nursing environments.^7,8

Our analysis of value based on 30-day mortality and Medicare payments displayed results generally similar to those of the cost analyses. We observed that the 1% improvement in absolute mortality in the focal vs control population (4.8% vs 5.8%) was associated with a statistically significant CMS payment increase of about $1000 ($26 091 vs $25 067), still a strong argument for excellent value.

Because our study asked whether a better nursing environment as defined by national recognition and NTB ratio could identify hospitals with better value, we purposely did not match on hospital characteristics. We found that using these 2 variables associated with the nursing environment produced 2 sets of hospitals with very different characteristics. Had we asked a different question related to whether a hospital administrator should improve a hospital’s nursing environment, as other studies have asked, then a different matching algorithm using both patient and hospital characteristics as well as a propensity score for being a recognized hospital could be used.

A limitation of our study was the use of a voluntary program of accreditation for good nursing environments as an indicator of hospital nursing work environment. Although hospitals with formal accreditation have been shown, on average, to have significantly better work environments than those without accreditation, there is known overlap in measured environments between hospitals with and without formal accreditation.^2,11 However, our study did not use formal accreditation alone to define different nursing environments but also separated hospitals by their NTB ratio, thereby helping to reducing this overlap.

Patients who undergo surgery in hospitals with better nursing environments typically display lower mortality, with similar costs suggesting that better nursing environments are associated with higher value. Our results do not address whether hospitals can necessarily improve their value by improving the nursing environment; other research has investigated that question. While better outcomes and value may be owing to other features of hospitals with good nursing, excellent nursing environments appear to provide a strong signal to patients and referring physicians for better quality, lower cost, and higher value. This is especially true for higher-risk patients, where the value of a better nursing environment appears to be greatest.

Corresponding Author: Jeffrey H. Silber, MD, PhD, Center for Outcomes Research, Children’s Hospital of Philadelphia, 3535 Market St, Ste 1029, Philadelphia, PA 19104 (silberj@wharton.upenn.edu).

Accepted for Publication: October 8, 2015.

Published Online: January 20, 2016. doi:10.1001/jamasurg.2015.4908.

Author Contributions: Dr Silber had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Silber, Rosenbaum, McHugh, Smith, Even-Shoshan, Aiken.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Silber, Rosenbaum, McHugh, Niknam, Kelz, Aiken.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Silber, Rosenbaum, McHugh, Ludwig, Smith, Niknam, Aiken.

Obtained funding: Silber, Even-Shoshan.

Administrative, technical, or material support: Silber, McHugh, Niknam, Even-Shoshan, Aiken.

Study supervision: Silber, McHugh, Even-Shoshan, Aiken.

Conflict of Interest Disclosures: None reported.

Funding/Support: This research was funded by grant R01-HS018338 from the Agency for Healthcare Research and Quality, grant R01-NR014855 from the National Institute of Nursing Research, and grant NSF SBS-10038744 from the National Science Foundation.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The findings and conclusions of this report are those of the authors and do not necessarily represent the official position of the Agency for Healthcare Research and Quality, National Institute of Nursing Research, or the National Science Foundation.

Previous Presentation: This study was presented at the 2015 AcademyHealth Annual Research Meeting; June 13, 2015; Minneapolis, Minnesota.

Additional Contributions: Traci Frank, AA, and Alex Hill, BS, Center for Outcomes Research, Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania, provided assistance with this research; they received no compensation.