Reliability of the American Community Survey Estimates of Risk-Adjusted Readmission Rankings for Hospitals Before and After Peer Group Stratification

IMPORTANCE Since the transition to the American Community Survey, data uncertainty has complicated its use for policy making and research, despite the ongoing need to identify disparities in health care outcomes. The US Centers for Medicare & Medicaid Services’ new, stratified payment adjustment method for its Hospital Readmissions Reduction Program may be able to reduce the reliance on data linkages to socioeconomic survey estimates. OBJECTIVE To determine whether there are differences in the reliability of socioeconomically riskadjusted hospital readmission rates among hospitals that serve a disproportionate share of low-income populations after stratifying hospitals into peer group–based classification groups. DESIGN, SETTING, AND PARTICIPANTS This cross-sectional study uses data from the 2014 New York State Health Cost and Utilization Project State Inpatient Database for 96 278 hospital admissions for acute myocardial infarction, pneumonia, and congestive heart failure. The analysis included patients aged 18 years and older who were not transferred to another hospital, who were discharged alive, who did not leave the hospital against medical advice, and who were discharged before December 2014. MAIN OUTCOMES AND MEASURES The main outcomes were 30-day hospital readmissions after acute myocardial infarction, pneumonia, and congestive heart failure assessed using hierarchical logistic regression. RESULTS The mean (SD) age of the patients was 69.6 (16.0) years for the safety-net hospitals and 74.9 (14.7) years for the non–safety-net hospitals; 9382 (48.8%) and 7003 (48.5%) patients, respectively, were female. For safety net designations, 20% (3 of 15) of all evaluations concealed and distorted differences in risk, with factors such as poverty failing to identify similar risk of acute myocardial infarction readmission until unreliable estimates were excluded from the analysis (OR, 1.23 [95% CI, 1.00-1.52], P = .02; vs OR, 1.17 [95% CI, 0.94-1.46], P = .15). By comparison, 2 of the 60 models (3%) for the peer group–based classification altered the association between socioeconomic status and readmission risk, concealing similarities in congestive heart failure readmission when adjusted using high school completion rates (OR, 1.27 [95% CI 1.02-1.58], P = .04; vs OR, 1.23 [95% CI, 0.98-1.53], P = .06) and distorting similarities in pneumonia readmissions when accounting for the proportion of lone-parent families (OR, 1.27 [95% CI, 0.98-1.66], P = .07; vs OR, 1.35 [95% CI, 1.02-1.80], P = .04) between the lowest and highest socioeconomic status hospitals in quartile 1. CONCLUSIONS AND RELEVANCE There was greater precision in socioeconomic adjusted readmission estimates when hospitals were stratified into the new payment adjustment criteria (continued) Key Points Question Is the quality of American Community Survey estimates associated with changes in hospital readmission rankings after socioeconomic risk adjustment? Findings In this cross-sectional study of 96 278 hospital admissions for acute myocardial infarction, pneumonia, and congestive heart failure included in the 2014 New York State Health Cost and Utilization Project, compared with previous stratification by safety-net hospital designation, the new peer group–based stratification system was associated with improved reliability of American Community Survey socioeconomic status estimates. Meaning Poor reliability in the American Community Survey can conceal and distort readmission rates for hospitals, and the use of peer group– based performance measurement is associated with a reduction, but not elimination, of the impact of measurement error on risk-adjusted rates. Author affiliations and article information are listed at the end of this article. Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2019;2(10):e1912727. doi:10.1001/jamanetworkopen.2019.12727 (Reprinted) October 9, 2019 1/15 Downloaded From: https://jamanetwork.com/ by a Non-Human Traffic (NHT) User on 03/15/2021 Abstract (continued)continued) compared with safety net designations. A contributing factor for improved reliability of American Community Survey estimates under the new payment criteria was the merging of patients from low-income neighborhoods with greater homogeneity in survey estimates into groupings similar to those for higher-income patients, whose neighborhoods often exhibit greater estimate variability. Additional efforts are needed to explore the effect of measurement error on American Community Survey–adjusted readmissions using the new peer group–based classification methods. JAMA Network Open. 2019;2(10):e1912727. doi:10.1001/jamanetworkopen.2019.12727


Introduction
Adhering to the Centers for Medicare & Medicaid Services (CMS) Hospital Readmissions Reduction Program (HRRP) performance and accountability metrics presents a profound and ongoing challenge for many safety-net hospitals (SNHs). [1][2][3] Because these hospitals care for a disproportionate number of socially and economically vulnerable patients, their readmission rates are often higher than those of non-SNHs, driven by challenges brought on by poverty, low health literacy, poor housing conditions, and a lack of social support and access to care. [4][5][6][7][8] These conditions often coincide with the primary reasons most patients cite as contributing to relapse and readmission. 9 In particular, research 10 indicates that differences in hospital readmission rates between SNHs and non-SNHs often become negligible after accounting for patient socioeconomic position, or socioeconomic status (SES), thus emphasizing the substantive association between the socioeconomic case mix of a hospital's patient population and interpretations of its quality and performance.
In an attempt to address this concern, CMS recently (fiscal year 2019) introduced a new peer group-based payment adjustment method into the HRRP to account for differences in readmission risk attributed to differences in patient socioeconomic case mix. 11 Because hospitals lack individuallevel data on patient socioeconomic case mix, CMS chose to classify institutions according to their proportion of fee-for-service Medicare and Medicare Advanced hospitalizations, for which the patient is eligible for both Medicare and Medicaid reimbursement. Initial evaluations of the new payment model suggest that a peer group-based method measured at the hospital level offers some reprieve to SNHs but does not eliminate the cost imbalances associated with the disproportionate burden these facilitates experience because they serve low-income populations. 12 To our knowledge, there has been no evaluation of CMS's new payment model structure relative to other measures of SES. Given the ever-changing insurance situation in the United States, insurance categories may not consistently be associated with income and other socioeconomic measures.
However, because hospitals typically do not collect patient-level socioeconomic data, any measure of SES beyond insurance status must be derived by geocoding hospitalization records to the US Census. This approach is problematic because it attempts to infer specific meaning from individual events that are drawn from aggregate data. Because of this reliance, a second and potentially more profound challenge has now emerged. At issue is that the recent transition from the long-form decennial US Census to the American Community Survey (ACS) fundamentally changed the quality of demographic, socioeconomic, and housing data collected from the population. For example, some studies 13 have found the margins of error in ACS poverty estimates to be so large that a neighborhood can switch from the least to the most deprived SES quartile. Accordingly, in this study, we tested whether proxy measures of patient SES were significantly associated with readmission risk before and after the switch to the peer group-based stratification model. Our primary focus was whether the reliability of the ACS estimates was associated with instances where patients' SES explained the differences in readmission risk.

Study Population and Outcomes
We evaluated readmissions for acute myocardial infarction (AMI), pneumonia, and congestive heart failure (CHF) because these were the 3 medical conditions linked to HRRP payments during this year.

Socioeconomic Status Estimation
We expressed the relative uncertainty for each ACS estimate by following US Census Bureau guidelines for calculating its coefficient of variation (CV) and margin of error. 22 The margin of error for unemployment status was calculated using guidelines for derived proportions. The measure of educational attainment and female lone-parent families required multiple numerators or denominators to build the proportions. For these, the numerators and denominators were aggregated within the equation for calculating the margin of error for the derived estimate. Median household income and poverty status did not require further calculation because these estimates are published as percentages or monetary values. We used an adjustment factor by multiplying the margin of error for each estimate or its numerator or denominator by a factor equal to 1.960 or 1.64 to derive the SE for each estimate with 95% confidence.
The National Research Council 23 defines a reasonable standard of precision for each ACS estimate as a CV score less than 12%. Categorical rankings of CV precision are also defined as high reliability when CV scores are less than 12%, moderate reliability for CV scores between 12% and 40%, and low or unreliable when CV scores exceed 40%. 24 The process required for evaluating the precision of the ACS estimates is described in greater detail by the US Census Bureau. 25 We used US Census Bureau thresholds for defining estimates as either reliable (CV Յ40%) or unreliable (CV >40%). The CV scores are often used as litmus test for the ACS because they allow uncertainty to be expressed in relative terms (eg, a CV score of 50% for a median household income of $50 000 would be ± $25 000 in the estimate). We excluded homeless populations from all analyses because they lack a geographic identifier that can link them to US Census data, the lack of a geographic identifier makes it infeasible to test estimate reliability, and the definition of homelessness can vary substantially across health care systems. More than 96% of patients defined as homeless in the New York SID were treated for AMI, CHF, or pneumonia by SNHs.

Statistical Analysis
The primary objective of our analysis was to assess whether ACS reliability was associated with the global interpretation of SES-adjusted 30-day readmission risk when institutions were defined as SNHs compared with the new CMS peer group-based method. For our first test, we used a hierarchical logistical regression model to estimate the independent association between riskadjusted 30-day readmission rates and hospital grouping designation after adjusting for patient age, sex, and health status using the Elixhauser algorithm. We then extended this model to include adjustment for SES regardless of the estimate's precision. In our last test, we limited the analysis to adjustment using only reliable estimates. Using an approach similar to the method used by CMS for its readmission rankings, we estimated readmission risk using hierarchical logistic regression specifying hospitals as a random effect to account for clustering.
Social, economic, and demographic factors are perceived to be associated with readmission risk if differences between facilities become negligible after their inclusion in the adjustment model. 26,27 We inferred that measurement reliability had an impact on this association if the removal of unreliable estimates altered the observed effect. We used adjusted odds ratios (ORs) to assess whether measurement error was associated with the likelihood that a patient would be readmitted to an SNH, compared with patients in non-SNHs. For the SNH analysis, non-SNH hospitals (lowest quartile) were the reference category. For the peer-group analysis, the highest-SES hospitals in each stratum were the reference group. The analysis included patients aged 18 years and older who were not transferred to another hospital, who were discharged alive, who did not leave the hospital against medical advice, and who were discharged before December 2014. All models were run using the GLIMMIX procedure in SAS statistical software version 9.4 for Windows (SAS Institute) using maximum likelihood estimation based on Laplace approximation. Two-sided P < .05 was considered significant.

Results
As shown in the Figure, approximately 40% of ACS zip code estimates for high school completion, female lone-parent families, poverty, and unemployment for the state of New York are considered to be unreliable according to their CV scores. Median household income was the most reliable measure in all comparisons. The Figure also illustrates that when SES scores were stratified by hospital grouping designation, SNHs served patients who had more reliable SES data. For all measures, no more than 1% of the SES estimates for SNH patients had levels of error that were deemed to be unreliable. By comparison, approximately 11% of estimates assigned to non-SNHs could be considered too imprecise to use. These trends remained when assessed across specific AMI, pneumonia, and CHF panels (data not shown). In contrast, the Figure also shows there was little discernable difference in the proportion of unreliable data assigned to each patient when hospitals were stratified using CMS's new peer group-based criteria. By comparison, only 11 of the 207 institutions defined as SNHs were similarly grouped into CMS's lowest-SES peer group (21.2%), whereas 41.5% of SNH hospitals were designated into the highest SES peer group (χ 2 = 20.78; P = .01). Given the differences in estimate reliability across hospital classification groups, we examined SES-adjusted readmission risk before and after accounting for the precision of each measure.     after adjustment for area unemployment rates. In total, 3 of the 15 SES models based on SNH designation (20%) were susceptible to error due to the inclusion of ACS SES estimates.     .02

JAMA Network Open | Health Policy
Abbreviations: CV, coefficient of variation; OR, odds ratio; SES, socioeconomic status.

Discussion
The use of patient SES to risk adjust hospital readmission rates remains an area of ongoing debate.
However, equivalent discourse over the quality of data available for SES adjustment has been lacking.
To date, there have been no fewer than 8 studies [28][29][30][31][32][33][34][35] that have attempted to contrast hospital readmission rates against socioeconomic data derived from the ACS. However, to our knowledge, no study has attempted to disentangle its findings from the reliability of the estimates. At issue is that approximately 40% of 2015 ACS zip code poverty estimates for the entire country-the de facto scale for SES risk adjustment owing to patient privacy constraints-have CVs classified as very unreliable. 24 Uncertainty of this magnitude jeopardizes efforts to expand risk adjustment, particularly if it results in the misclassification and mislabeling of a hospital's performance.
Our evaluation of ACS-adjusted readmission rates reveals subtle, but significant, differences in hospital scores owing to poor precision in the data. Before the new group-based payment class, approximately 20% of SNH comparisons adjusted for high school completion rates, median household income, poverty, female lone-parent families, or area unemployment were so imprecise that they either concealed or distorted differences in readmission risk. These findings suggest a possible double jeopardy of expanding the HRRP on the basis of SNH designation: account for factors known to predispose patients to readmission in ways unrelated to the quality of care they received, but risk misclassifying hospital performance. Although an alternative method to increase homogeneity in the estimates is to expand the analysis to US Census Blocks or Tracts, there are 2 hurdles to this approach. First, patient zip codes, when recorded, are typically the finest geographic footprint that hospitals release. Second, and more problematic, is that the reliability of the ACS typically gets worse as the geographic area gets smaller. 36 A twist on these findings was that, with few exceptions, the peer group-based criteria largely reduced the need for additional socioeconomic adjustment. Key exceptions were found for CHF readmissions adjusted with area poverty and high school completion rates. However, the lack of consistency in association between readmission risk and SES when these same variables were used to adjust AMI and pneumonia rates raises questions as to the sensitivity of aggregate data for representing the specific social and economic conditions widely held to be associated with health outcomes.
An additional challenge in using the ACS that was not discussed in this study is whether its reliability is suitable for annual data surveillance. It takes 5 years of surveys to build the block, tract, and zip code estimates that are released for public use. However, 80% of the data contained in the 2014 five-year estimates overlap the 2013 survey estimates. 25 This means that annual changes in an area's social, economic, or housing profile will often be driven by differences in data from the nonoverlapping years. Our team's previous evaluations 37 have shown that annual changes in ACS-reported socioeconomic conditions are more likely to arise because of sampling errors, as opposed to actual changes in living conditions. Additional tests are still needed to assess the fitness of the ACS for use on an annual basis. Even if one agrees with expanding HRRP criteria to include other measures of patient SES, many advocates also believe that SES adjustment may have a negative effect on SNH rankings. 40 One concern is that without upstream changes to the root cause of health inequalities, the added adjustment will force hospitals to become even more accountable to economically vulnerable patients. 40 At issue is that SNHs already operate on a minimal or negative budget, which is often stretched because of cost imbalances resulting from the disproportionate share of reimbursement program cutbacks and revenue gained from expanded insurance coverage mandated by the Affordable Care Act. 41,42 Another concern is that SES adjustment will "adjust away" instances when economically vulnerable patients experience treatment bias or receive treatment based on what they can afford. 43 Until recently, this has been the point of view of CMS and one of the reasons the HRRP had yet to expand its performance criteria. 39 Others 44,45 contend that further adjustment is unfair because it requires holding hospitals accountable for results of care that are beyond their control.

JAMA Network Open | Health Policy
This critique becomes magnified owing to the lack of direct measurement for many of the factors deemed to be instrumental to increased risk of rehospitalization. That our findings show inconsistency in the association between SES and readmission risk across all 3 medical conditions lends support to this argument. However, it nevertheless remains difficult to substantiate many of these concerns because broader measures of social and economic conditions have yet to be included within CMS performance measures.

Limitations
Our findings should be interpreted in light of the limitations of this study. Reliance on administrative data to conceptualize specific social processes at an individual level has well-known limitations.
Nonetheless, the ACS is currently the best available resource to account for social risk factors owing to the lack of patient-level data in hospital registries. Similarly, our findings were limited to readmissions contained in a single state's SID file. As a result, we were unable to determine whether these rates represented regional patterns that might have been further identified through additional data linkages. Although we suspect that our findings represent trends that would also appear in other states using different measures or other medical and surgical conditions, we did not determine whether our findings were an exception or rule. In addition, we did not account for whether any hospitals included in our analysis were classified as critical access hospitals. Although critical access hospitals are exempt from the HRRP, hospital readmissions are a central focus on its internal performance reports to CMS. Subsequent analyses will further address these limitations, as well as contribute new questions that can add to this discussion.

Conclusions
Studies are now emerging about the reliability of the ACS and what is at stake for communities that are having to make planning decisions based on bad data. Issues of data uncertainty in the ACS have not yet permeated into evaluations of risk-adjusted hospital performance measurement. If the HRRP were to continue to expand its adjustment criteria to account for patient SES case mix, it is possible that some health care systems would be buried further in debt simply because data with poor estimate reliability were used to adjust their scores. Uncertainty of this magnitude jeopardizes efforts to expand risk-adjustment criteria, particularly if they incorrectly penalize or reward health care systems that may be undeserving of either. One alternative to these challenges is to improve the demographic and socioeconomic data collected from patients, but the fact remains that there is no system in the United States for such information to be systematically collected, standardized, and effectively used.