Box-and-whisker plots with the left and right margins of the box indicating the first and third quartiles. The band inside the box is the second quartile (the median), and the whiskers indicate the minimum and maximum.
Each hospital is indicated by a letter along the x-axis, in sequence, based on hospital observed-to-expected PPR ratio in the 15-day baseline model. In adjusting for social determinants of health, hospitals that change from penalized to not penalized are indicated by solid black vertical lines, and hospitals that change from not penalized to penalized are indicated by dotted black vertical lines.
eTable. Most Common All Patients Refined Diagnosis Related Groups (APR-DRGs) for Initial Hospitalizations without and with a Potentially Preventable Readmission and Return Hospitalizations for a Potentially Preventable Readmission
eFigure. Number of Hospitals Changing Penalty Status with Risk-Adjustment for Social Determinants of Health by Observed-to-Expected Ratio Threshold for Potentially Preventable Readmissions
Sills MR, Hall M, Colvin JD, Macy ML, Cutler GJ, Bettenhausen JL, Morse RB, Auger KA, Raphael JL, Gottlieb LM, Fieldston ES, Shah SS. Association of Social Determinants With Children’s Hospitals’ Preventable Readmissions Performance. JAMA Pediatr. 2016;170(4):350-358. doi:10.1001/jamapediatrics.2015.4440
Performance-measure risk adjustment is of great interest to hospital stakeholders who face substantial financial penalties from readmissions pay-for-performance (P4P) measures. Despite evidence of the association between social determinants of health (SDH) and individual patient readmission risk, the effect of risk adjusting for SDH on readmissions P4P penalties to hospitals is not well understood.
To determine whether risk adjustment for commonly available SDH measures affects the readmissions-based P4P penalty status of a national cohort of children’s hospitals.
Design, Setting, and Participants
Retrospective cohort study of 43 free-standing children’s hospitals within the Pediatric Health Information System database in the calendar year 2013. We evaluated hospital discharges from 2013 that met criteria for 3M Health Information Systems’ potentially preventable readmissions measure for calendar year 2013. The analysis was conducted from July 2015 to August 2015.
Two risk-adjustment models: a baseline model adjusted for severity of illness and an SDH-enhanced model that adjusted for severity of illness and the following 4 SDH variables: race, ethnicity, payer, and median household income for the patient’s home zip code.
Main Outcomes and Measures
Change in a hospital’s potentially preventable readmissions penalty status (ie, change in whether a hospital exceeded the penalty threshold) using an observed-to-expected potentially preventable readmissions ratio of 1.0 as a penalty threshold.
For the 179 400 hospital discharges from the 43 hospitals meeting inclusion criteria, median (interquartile range [IQR]) hospital-level percentages for the SDH variables were 39.2% nonwhite (n = 71 300; IQR, 28.6%-54.6%), 17.9% Hispanic (n = 32 060; IQR, 6.7%-37.0%), and 58.7% publicly insured (n = 106 116; IQR, 50.4%-67.8%). The hospital median household income for the patient’s home zip code was $40 674 (IQR, $35 912-$46 190). When compared with the baseline model, adjustment for SDH resulted in a change in penalty status for 3 hospitals within the 15-day window (2 were no longer above the penalty threshold and 1 was newly penalized) and 5 hospitals within the 30-day window (3 were no longer above the penalty threshold and 2 were newly penalized).
Conclusions and Relevance
Risk adjustment for SDH changed hospitals’ penalty status on a readmissions-based P4P measure. Without adjusting P4P measures for SDH, hospitals may receive penalties partially related to patient SDH factors beyond the quality of hospital care.
The use of readmission rates as the basis for pay for performance (P4P) penalties has generated controversy, particularly with regard to the extent that readmissions reflect hospital quality.1 For adult Medicare patients, the Centers for Medicare and Medicaid Services’ (CMS) Hospital Readmission Reduction Program reduces payments to hospitals with excess 30-day Medicare readmissions. For children, several state Medicaid programs have implemented2- 5 or plan to implement6,7 readmission penalties for hospitals with excess pediatric readmissions, using 3M’s Potentially Preventable Readmission (PPR) measure.
Expected PPR rates, against which each hospital’s observed PPR rates are compared, are calculated at the state level after state-specific risk-adjustment calculations are performed. The risk-adjustment factors CMS permits in readmissions P4P policies include sex, age, comorbidities, and medical frailty,8 but not social determinants of health (SDH) such as race, ethnicity, education, income, and payer.9 The goal of risk adjustment is to help distinguish quality of patient care from the effects of patient characteristics that are not under the control of the clinician.1 A 2014 review of predictors of pediatric readmissions10,11 showed that several SDH were predictors: race/ethnicity, public insurance, and median household income for the patient’s home zip code (zip-MHHI).12 Social determinants of health affect health directly as well as indirectly by impairing access to care and adherence to health care recommendations, both of which could raise readmission risk.11,13- 15These findings have prompted public debate on whether the readmission measures used by CMS for P4P should control for SDH. Supporters of CMS’ existing policies argue that adjusting for SDH in readmission risk might obscure the detection of disparities and diminish incentives to reduce disparities.16 The opposing argument, which supports risk adjustment to avoid penalizing physicians serving socioeconomically disadvantaged populations, is supported by several studies17,18 and is the impetus for legislation introduced in Congress.19,20
In response to this debate, CMS asked a National Quality Forum expert panel to evaluate whether P4P measures, such as readmissions, should include risk adjustments for SDH. Despite evidence of an association between SDH and readmission risk, the effect of including SDH in models quantifying readmission rates is not well understood.10 Although the National Quality Forum panel recommended risk adjustment for SDH for performance measures used for P4P,21 the National Quality Forum board of directors recommended further study before adopting the panel recommendations.22
In this study, we respond to the National Quality Forum’s call for additional research on the effects of SDH risk adjusment on P4P measurements by comparing baseline and SDH–risk-adjusted PPR rates to determine whether adjustment for SDH could affect the P4P penalty status of a national sample of children’s hospitals.
Question: Does risk adjustment for social determinants of health affect hospitals’ penalty status on a readmissions-based pay-for-performance measure in a national cohort of children’s hospitals?
Findings: In this retrospective cohort study of 179 400 discharges from 43 hospitals, we calculated hospital rates on a readmissions-based pay-for-performance measure. Risk adjusting the readmissions measure for social determinants of health changed the penalty status of 7.0% and 11.6% of hospitals using 15-day and 30-day readmission windows, respectively.
Meaning: Social determinants of health are risk factors for readmissions-related penalties for children’s hospitals, and thus, risk adjustment for social determinants may reduce penalizing hospitals for patient factors beyond their control.
We performed a multicenter retrospective cohort study of children with inpatient and observation status hospitalizations using data from the 43 hospitals in the Pediatric Health Information System (PHIS) for calendar year 2013.23 Analyses were performed at the hospital level and conducted from July 2015 to August 2015.
The PHIS study hospitals provide patient demographic data, patient’s home zip code, International Classification of Diseases, Ninth Revision, Clinical Modification diagnoses and procedure codes, and hospital charges. Data are deidentified prior to inclusion in the database, but unique identifiers permit longitudinal analyses across visits. Data quality and reliability are assured jointly by the Children’s Hospital Association (Overland Park, Kansas), participating hospitals, and Truven Health Analytics (New York, New York).24 In accordance with the Common Rule (45 CFR 46.102[f]) and the policies of Cincinnati Children’s Institutional Review Board, this study using a deidentified data set was not considered human participants research. For this reason, it was not subject to institutional review board approval, and consent was not obtained from the patients.
This analysis included all patients discharged from PHIS hospitals during calendar year 2013 who met PPR inclusion criteria.
Our primary outcome was change in hospital PPR penalty status (ie, change in whether a hospital exceeded the penalty threshold) using an observed-to-expected PPR ratio of 1.0 as a penalty threshold. An observed-to-expected ratio greater than 1.0 indicated that a hospital had more observed readmissions than would be expected for the characteristics of its patients. Each state uses different and complex criteria for assessing a penalty in relation to the penalty threshold; for our analysis, we simplified these to a hypothetical policy that assigns a penalty to any hospital with an observed-to-expected PPR ratio greater than 1.0. Observed-to-expected ratio thresholds used by state Medicaid agencies with publicly available methods range from 0.85 to 1.12- 5 with 2 states using a threshold of 1.0.4,5
The primary rationale for selecting the 3M PPR measure (3M Health Information Systems, version 32.0) was that it is used for P4P by several state Medicaid agencies2- 5 and thus already directly affects many hospitals. The 3M PPRs use a rate-based approach to identify PPRs using the All Patient Refined Diagnosis Related Group (APR-DRG) categories of the index and return encounters. The crude PPR rate for a hospital is the ratio of the number of returns deemed preventable divided by the number of index encounters at risk for a PPR. The 3M PPRs exclude certain conditions in which the readmission was considered mostly planned, clinically unrelated to the index admission, or clinically related but seldom preventable. In addition, 3M PPRs have age-specific exclusion criteria (eg, urinary tract infections in children <1 year).25 The 3M PPRs also exclude index discharges that had any disposition other than discharge home (eg, died or transferred). All patient hospitalizations during the study period were categorized using APR-DRGs (3M Health Information Systems, version 24) based on patient age and International Classification of Diseases, Ninth Revision, Clinical Modification diagnoses and procedure codes assigned during each episode of care.
We examined the change in PPR penalty status, comparing 2 models that incorporated stepwise adjustment for severity of illness and SDH.
To approximate the risk adjustment used by several state Medicaid agencies, we compiled the risk-adjustment factors from states with publicly available methods.2- 5 Each state uses a combination of 3 risk-adjustment factors: severity of illness, age, and mental health comorbidity. In our baseline model, we included only APR-DRG severity of illness to calculate adjusted observed-to-expected PPR ratios. The APR-DRG system provides a severity-of-illness score (1 [minor] to 4 [extreme]) from which we calculated each hospital’s case mix index. We excluded age from risk adjustment because all 4 states used a binary child-adult variable not relevant to our children’s hospital study population. All 4 states also adjust for the presence of a secondary mental health or substance abuse diagnosis, but because only 4.6% of discharges in our sample had these diagnoses, adjusting for this variable resulted in model convergence issues, and thus it was excluded.
Our SDH-enhanced model resulted from adding risk adjustment for 4 SDH factors to our baseline model. Our selection of SDH was informed by the SDH factors recommended by the Institute of Medicine’s report Capturing Social and Behavioral Domains and Measures in Electronic Health Records.26 Of the recommended 11 variables measuring social and behavioral factors, 3 variables are available in PHIS (race, ethnicity, and, as a proxy for “financial strain,” payer), and a fourth is available via zip code links (zip-MHHI).
Demographic data and insurance status were defined at the time of the index hospitalization as assigned through each hospital’s patient registration process. Race categories included white, black or African American, Asian, American Indian or Alaska Native, native Hawaiian or other Pacific Islander, and other. The “other” category included unreported or missing data or any category not previously mentioned; to this category we also added American Indian, Alaska Native, native Hawaiian, and Pacific Islander. Ethnicity categories included Hispanic or Latino and other. The payer category “public” included Medicaid, Medicaid-managed care, and Title V. “Commercial” payer included employer-based (including TRICARE) and privately purchased health insurance. “Other” payer category included self-pay, charity, Medicare, worker’s compensation, other governmental insurance, missing payer information, and no charge.
We used patient-level 5-digit zip codes to extract the zip-MHHI variable from the Geolytics data set (Geolytics Inc). When individual-level data are unavailable, zip-level data have been shown to be a valid proxy for SDH.27,28 Given the expected colinearity between the 4 SDH variables, we included the 2-term interactions in the SDH-enhanced model. We used zip-MHHI as a continuous variable in our models.
To calculate an observed-to-expected ratio for a hospital, we used the observed (ie, actual) number of PPRs at the hospital and divided by the expected (ie, predicted) number of PPRs. The expected number of PPRs was derived from pooled data from all 43 study hospitals and estimated for each hospital (based on its mix of patients) using a generalized linear mixed-effect model that included fixed effects for the specified severity of illness and SDH covariates for the risk adjustment. In other words, we calculated each hospital’s expected PPR by starting with the mean PPR for all 43 hospitals and then adjusting for the individual hospital’s PPR population patient characteristics. In practice, PPR is compared against other hospitals in the same state; we used our nationally distributed cohort of children’s hospitals to calculate expected PPR rates.
We calculated observed-to-expected PPR ratios for 15-day and 30-day PPR using the baseline and SDH-enhanced models. We elected to use both the 15-day and 30-day windows in our analysis because these are the timeframes most commonly used in studies29 and in PPR-based state Medicaid policies.2- 5 We identified hospitals that fell above the penalty threshold of an observed-to-expected ratio of 1.0.30 To assess the effect of adjusting for SDH, we reported the count of hospitals that changed penalty status overall when moving from the baseline to the SDH-enhanced model and further categorized the hospitals by the direction of change (no longer penalized or now penalized).
In the 43 PHIS study hospitals, 179 400 index discharges met inclusion criteria (eligible for PPR rate calculation) during the study period. Of these, 9484 (5.3%) were associated with a 30-day PPR (Table 1). Discharge characteristics were 59.2% publicly insured, 60.3% white, and 17.9% Hispanic. The zip-MHHI was $36 422 (Table 1).
In unadjusted comparisons (Table 1), discharges with 30-day PPRs were more common among patients who were older (>12 years), female, non-Hispanic, and other race/ethnicity, had other insurance, and had complex chronic conditions. Those patients with higher zip-MHHI were at greater risk for 30-day PPR, although the difference ($810) was small. The most common index hospitalization APR-DRGs for patients with a PPR were seizure, bronchiolitis, and gastroenteritis; the most common for those without a PPR were asthma, seizure, and bronchiolitis; and the most common among PPR return hospitalizations were seizure, other digestive system diagnosis, and gastroenteritis (eTable in the Supplement).
At the hospital level, PHIS-derived SDH demonstrated wide variation in race/ethnicity, with a median percentage of 60.8% white patients (interquartile range [IQR], 45.4%-71.4%) and a median percentage of 14.1% Hispanic patients (IQR, 6.7%-37.0%). Hospitals were more similar to each other in the proportion of publicly insured patients, with a median of 58.7% (IQR, 50.4%-67.8%), and zip-MHHI patients, with a median of $40 674 (IQR, $35 912-$46 190) (Table 2). The median 15-day and 30-day PPR rates were 3.7% (IQR, 3.2%-4.2%) and 5.2% (IQR, 4.5%-5.9%), respectively.
In the baseline model, hospitals varied in severity-adjusted PPR rates, with a 15-day PPR median of 3.8% (IQR, 3.4%-4.2%; range, 2.8%-4.8%) and a 30-day PPR median of 5.3% (IQR, 4.7%-5.9%; range, 4.1%-6.8%) (Figure 1). As expected based on methods used to risk adjust across the study hospitals, the hospital-level median PPR did not change significantly between models for either timeframe.
With a 15-day timeframe, 22 hospitals were penalized in the baseline model, and SDH adjustment changed penalty status for 3 hospitals (7.0% of study hospitals): 2 were no longer penalized and 1 was newly penalized (Figure 2). With a 30-day timeframe, 23 hospitals were penalized in the baseline model, and SDH-adjustment changed penalty status for 5 hospitals (11.6%): 3 were no longer penalized and 2 were newly penalized (Figure 2). Although we examined the effect of adjusting for SDH using a single observed-to-expected PPR ratio (1.0), a sensitivity analysis showed that observed-to-expected PPR ratio penalty thresholds between 0.75 and 1.30 would result in 0 to 5 hospitals changing penalty status through SDH risk adjustment (eFigure in the Supplement).
Based on the methods used for risk adjustment, those hospitals whose PPR performance worsened with SDH adjustment (ie, the observed-to-expected ratio increased), including those newly penalized, have a patient population less at risk than average, and vice versa. In multivariable modeling of the case mix index and SDH variables used for risk adjustment in the SDH-enhanced model, higher case mix index and public insurance increased risk of 30-day PPR; other race and other insurance decreased risk; and ethnicity and zip-MHHI were not associated with 30-day PPR (Table 3).
Prior studies have shown the effect of SDH on outcomes for hospitalized children, including readmissions, but, to our knowledge, the effect of risk adjustment for these factors on hospital performance has not been explored in a population of children. We compared hospital performance on a PPR-based P4P model derived from state Medicaid policies before and after adjustment for SDH. We found that SDH adjustment changed the PPR penalty status of 7.0% and 11.6% of hospitals using 15-day and 30-day readmission windows, respectively. These findings demonstrate the contribution of select, easily available SDH measures to hospital P4P measures, such as readmissions, and show that SDH adjustment can affect which hospitals are penalized.
Although other studies have shown that SDH are risk factors for readmission of individual children,24,25 our findings show that SDH are also risk factors for readmissions at the hospital level. To our knowledge, our analysis is the first to show the effect of Medicaid penalty policies. Two prior studies found that hospital-level variation in performance on Hospital Readmission Reduction Program measures narrows with SDH adjustment in a Medicare population.10,31,32Although our study expressed the outcome differently (in terms of change in hospital penalty status rather than change in distribution of hospital-level readmission rates), we had a similar finding, ie, that SDH affect hospital-level risk in addition to individual patient risk.
A third study found that teaching hospitals and safety net hospitals are at risk of higher penalties than other hospitals.32 Although this study did not examine the association between SDH and penalty status, high-penalty-risk hospitals care for more socioeconomically disadvantaged patients. Our study hospitals were all teaching hospitals, and our study patients were more socioeconomically disadvantaged than children hospitalized elsewhere. Specifically, compared with children hospitalized nationally in all settings, our study population was more likely to have public insurance (49.0% nationally vs 59.2% in our study population)33 and had a lower median household income (29.5% with zip-MHHI below $39 000 nationally vs 57.6% in our study population).33
Our study also adds to the evidence that children’s hospitals have higher pediatric readmission rates than other hospitals. The 5.2% 30-day PPR rate found in our study approximates the 6.3% to 6.5% 30-day readmission rates reported in children’s hospitals.25,29 When compared with New York state’s 30-day all-cause readmission rates for children, our findings were higher than same-hospital rates (2.5%) for children in all hospitals, but similar to the same-hospital rates for the state’s 12 children’s hospitals (5.3%).30
Adjustment for SDH affected more hospitals’ penalty status using a 30-day readmissions window compared with a 15-day readmissions window. This is consistent with other studies that have concluded that earlier readmissions are more reflective of hospital care quality (eg, care coordination), while later readmissions reflect more of a mix of care quality, personal health status, and SDH.25,34
The financial effect of P4P penalties is substantial, can worsen the financial challenges already facing safety net hospitals, and can lead to unintended consequences such as hospital closure. For example, the Texas Medicaid penalties include a reimbursement adjustment of 1% for hospitals with an observed-to-expected PPR ratio from 1.11 through 1.25 and of 2% for hospitals with an observed-to-expected PPR ratio greater than 1.25. Because hospitals average an operating margin of 3.4% (8.5% in children’s hospitals),35 penalties of this magnitude could threaten the financial solvency of low-resourced hospitals caring for patient populations facing social adversity. As an example of penalty magnitude, the 1 children’s hospital in our sample for which PPR penalty was publicly reported was assessed a total payment reduction of $2 236 624 in 2013.36
Further research is needed to fully examine the effect of SDH risk adjustment on hospital P4P programs. Although this analysis focused on the 4 SDH variables that were recommended by the Institute of Medicine26 and were readily available in the PHIS data set, others have found that a broader array of SDH were predictive of variability in readmission rate.10,37 Broader analysis of the effect of diverse SDH factors on P4P measures can help stakeholders, including hospital administrators, health care professionals, and payers, better target interventions for those patient populations most vulnerable to readmission. Because of the correlation between SDH factors, further research into analytic methods, including factor analysis, can help identify factor groupings that are clinically useful and mitigate analytic issues related to collinearity.
A key limitation is that our sample of hospitals differs substantially from samples used in actual PPR P4P programs, where hospitals are compared with other hospitals in the same state. The children’s hospitals in this study may be more similar to one another than to other hospitals in their same state. The smaller variation in socioeconomic status between hospitals in our study likely biased our findings toward the null. In other words, by using a data set of similar, tertiary, and quaternary children's hospitals, our study likely masked a more significant effect of SDH on PPR penalty status.
Second, the PHIS database does not allow tracking patients across multiple hospitals and thus excludes the contribution of readmissions to other hospitals.30 It is unclear how our use of same-hospital instead of all-hospital readmissions would bias study findings.
We also note that our selection of SDH, informed by the Institute of Medicine report’s recommendations,26 was limited by data routinely available to hospitals. Of the 11 SDH variables that the Institute of Medicine recommends hospitals routinely collect, only 4 were available in PHIS. Policies that support efforts to create standardized methods for representing and quantifying the effect of SDH on metrics used for quality reporting could make SDH risk adjustment more robust and help hospital stakeholders better understand which of their patients are more vulnerable to readmission.10
The results of our analysis show that adjustment for SDH changes hospitals’ penalty status on a readmissions-based P4P measure. Without adjusting P4P measures for SDH, hospitals that care for more vulnerable patients may receive penalties in part related to patient factors beyond the control of the hospital and unrelated to the quality of hospital care. Further work to characterize the effects of SDH on performance measures may assist efforts to improve care quality and deliver more equitable care.
Corresponding Author: Marion R. Sills, MD, MPH, Department of Pediatrics, University of Colorado School of Medicine, 13123 E 16th Ave, B251, Aurora, CO 80045 (firstname.lastname@example.org).
Published Online: February 15, 2016. doi:10.1001/jamapediatrics.2015.4440.
Author Contributions: Dr Hall had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: All authors.
Acquisition, analysis, or interpretation of data: Sills, Hall, Macy, Cutler, Bettenhausen, Morse, Auger, Raphael, Gottlieb, Fieldston, Shah.
Drafting of the manuscript: Sills, Hall, Cutler, Bettenhausen, Morse, Gottlieb.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Hall.
Administrative, technical, or material support: Sills, Hall, Fieldston.
Study supervision: Shah.
Conflict of Interest Disclosures: None reported.