Importance
Medicare penalizes hospitals with higher than expected readmission rates by up to 3% of annual inpatient payments. Expected rates are adjusted only for patients’ age, sex, discharge diagnosis, and recent diagnoses.
Objective
To assess the extent to which a comprehensive set of patient characteristics accounts for differences in hospital readmission rates.
Design, Setting, and Participants
Using survey data from the nationally representative Health and Retirement Study (HRS) and linked Medicare claims for HRS participants enrolled in Medicare who were hospitalized from 2009 to 2012 (n = 8067 admissions), we assessed 29 patient characteristics from survey data and claims as potential predictors of 30-day readmission when added to standard Medicare adjustments of hospital readmission rates. We then compared the distribution of these characteristics between participants admitted to hospitals with higher vs lower hospital-wide readmission rates reported by Medicare. Finally, we estimated differences in the probability of readmission between these groups of participants before vs after adjusting for the additional patient characteristics.
Main Outcomes and Measures
All-cause readmission within 30 days of discharge.
Results
Of the additional 29 patient characteristics assessed, 22 significantly predicted readmission beyond standard adjustments, and 17 of these were distributed differently between hospitals in the highest vs lowest quintiles of publicly reported hospital-wide readmission rates (P ≤ .04 for all comparisons). Almost all of these differences (16 of 17) indicated that participants admitted to hospitals in the highest quintile of readmission rates were more likely to have characteristics that were associated with a higher probability of readmission. The difference in the probability of readmission between participants admitted to hospitals in the highest vs lowest quintile of hospital-wide readmission rates was reduced by 48% from 4.41 percentage points with standard adjustments used by Medicare to 2.29 percentage points after adjustment for all patient characteristics assessed (reduction in difference: −2.12; 95% CI, −3.33 to −0.67; P = .003).
Conclusions and Relevance
Patient characteristics not included in Medicare’s current risk-adjustment methods explained much of the difference in readmission risk between patients admitted to hospitals with higher vs lower readmission rates. Hospitals with high readmission rates may be penalized to a large extent based on the patients they serve.
The Medicare Hospital Readmissions Reduction Program (HRRP) financially penalizes hospitals with higher than expected 30-day readmission rates for Medicare patients by reducing annual reimbursements by up to 3%. In 2014, the second year of the program, 2610 hospitals were fined a total of $428 million for excess readmissions.1Quiz Ref ID In setting an expected readmission rate for each hospital, the Centers for Medicare and Medicaid Services (CMS) adjusts only for patients’ age, sex, discharge diagnosis, and diagnoses present in claims during the 12 months prior to admission.2 This limited adjustment has raised concerns that hospitals may be penalized because they disproportionately serve patients with clinical and social characteristics that predispose them to hospitalization or rehospitalization.3,4
Prior research has identified several patient factors that are predictive of readmission and not included in the HRRP’s risk-adjustment model.5 Individual studies have addressed only a sparse set of factors, however, because detailed patient information is typically lacking in databases identifying hospitalizations.6 Moreover, the most policy-relevant question is not whether patient characteristics omitted from the HRRP’s risk-adjustment model predict readmission. Rather, it is whether those characteristics are distributed unevenly across hospitals and thereby account for differences in excess readmissions—and penalties—determined by the CMS. Few studies have addressed this question in Medicare directly by examining the effects of adjustment for patient characteristics on differences in hospital readmission rates, and these studies have been restricted to a small number of characteristics.7-10 Therefore, the extent to which adjustment for a comprehensive set of patient characteristics would account for differences in hospital readmission rates remains unclear.
Using detailed survey data from the Health and Retirement Study (HRS) and linked Medicare claims, we conducted 3 related analyses. First, using data from 2000 to 2012, we analyzed an extensive set of clinical and social characteristics as potential predictors of all-cause 30-day readmission among hospitalized survey participants, including claims and survey variables not used by the CMS in risk adjustment of readmission rates. Second, using data from 2009 to 2012 to align the study period with the first publicly reported readmission rates, we compared these characteristics between participants admitted to hospitals with high vs low readmission rates. Third, again using 2009-2012 data, we then compared differences in the probability of readmission between participants admitted to hospitals with high vs low publicly reported readmission rates before vs after adjustment for the additional patient characteristics.
We analyzed data from the 2000-2010 biennial waves of the HRS, a nationally representative longitudinal survey of adults older than 50 years in the continental United States (average response rate, 88%), and linked Medicare claims from 2000 to 2012.11-13 Our study sample included HRS survey respondents who were eligible for Medicare and provided their Medicare identification numbers for linkage to claims and enrollment files (91% of eligible participants). We excluded participants residing in nursing homes because the HRS samples households and provides sampling weights only for community-dwelling adults. For each survey year, we limited our sample to participants who were hospitalized after survey completion during the survey year or 2 subsequent years. We analyzed all admissions during this span for each participant (median time between survey and admission, 462 days), using the participant admission as the unit of analysis. Our study was approved by the Harvard Medical School Committee on Human Studies.
We examined readmissions for all hospitalizations as defined in the hospital-wide readmission rate measure,14 rather than the condition-specific measures used in the HRRP, to maximize statistical power for analyzing readmissions in the HRS sample; the conditions included in the HRRP (congestive heart failure [CHF], myocardial infarction, and pneumonia) represent less than 20% of all Medicare admissions.15 Following CMS specifications for calculating hospital-wide readmission rates,14,15 we defined index admissions as all admissions to nonfederal acute care hospitals without transfer to another acute care facility or discharge against medical advice, and we excluded admissions for certain primary diagnoses or to certain facilities, using principal discharge diagnoses and procedure codes to define reasons for admission.14 We also excluded index admissions during which the patient died and admissions for patients without 12 months of enrollment in fee-for-service Medicare prior to admission.16 Patients who died within 30 days after discharge were not excluded per CMS specifications.
For each index admission, we used Medicare inpatient claims to assess whether the participant had an unplanned readmission within 30 days of discharge, excluding planned readmissions, such as scheduled procedures or chemotherapy per CMS specifications.14 In a sensitivity analysis, we also excluded index admissions that were also readmissions; this restriction is applied by the CMS in determining readmissions for the HRRP but not in calculating hospital-wide readmission rates.14,16
Categorizing Participants by Readmission Rate of Admitting Hospital
For comparisons of participants admitted to hospitals with high vs low readmission rates, we categorized index admissions in our study sample into quintiles according to the admitting hospital’s publicly reported hospital-wide readmission rate from 2011 to 2012 (the earliest reporting period for this measure).17 Like the condition-specific readmission rates reported by the HRRP, publicly reported hospital-wide readmission rates are adjusted for age, sex, discharge diagnosis, and specific diagnoses present in claims during the 12 months prior to admission.16 Among the 1896 hospitals captured in our study sample, publicly reported hospital-wide readmission rates for 2011 to 2012 were strongly correlated with case-weighted averages of readmission rates reported by the HRRP from 2009 to 2012 for myocardial infarction, pneumonia, and CHF (r = 0.70; P < .001).17 This strong correlation supports the steps we took to generate adequate statistical power for our research objectives—specifically, considering readmissions for all index hospitalizations and using publicly reported hospital-wide readmission rates from 2011 to 2012 to categorize participants admitted from 2009 to 2012.
Because the HRS is a nationally representative sample, participants admitted to hospitals in the highest or lowest quintiles of readmission rates, for example, should constitute representative samples of the national populations of patients admitted to hospitals in the highest or lowest quintile. In a supplementary analysis (eAppendix 1 in the Supplement), we confirmed that differences between these quintiles in patient characteristics assessed from claims were largely similar for the HRS study sample and a 20% random sample of all similarly aged fee-for-service Medicare beneficiaries.
Clinical and Social Characteristics
From administrative and survey data for each participant, we assessed a broad range of prespecified demographic, financial, clinical, and social characteristics, including variables used by the CMS for risk adjustment of hospital readmission rates and additional variables not included in those methods (Table 1).
Demographics and Eligibility Categories From Medicare Enrollment File
From Medicare enrollment files, we determined age, sex, Medicaid enrollment, whether disability was the original reason for Medicare eligibility, and whether the participant had end-stage renal disease.
Clinical Characteristics From Claims
From linked Medicare claims, we assessed the discharge diagnosis and 31 condition indicators used by the CMS for adjustment of hospital-wide readmission rates.14 Consistent with methods used by the HRRP, we derived these indicators from diagnoses present in inpatient claims for the index admission or in inpatient or outpatient claims during the 12 months prior to admission.16 We similarly assessed additional condition indicators used for adjusting condition-specific readmission rates in the HRRP but did not include these in our main analyses because they affected our results minimally.
For each admission of each participant, we also determined a hierarchical condition category (HCC) risk score from the 12 months of claims prior to admission, and we determined at the start of the year the presence of 26 conditions from the Chronic Condition Data Warehouse (CCW), which uses claims since 1999 to describe Medicare beneficiaries’ accumulated chronic disease burden.18,19
Clinical and Social Characteristics From HRS Surveys
From HRS surveys, we selected 24 variables potentially predictive of readmission in elderly patients according to previously developed conceptual models.6,20 As listed in Table 1, these variables included race/ethnicity, education, labor force status, household income and assets, supplemental and prescription drug coverage, smoking status, alcohol consumption, general health status, physical functioning, difficulties with activities of daily living (ADLs) and instrumental ADLs (IADLs), work limitations due to health, depressive symptoms based on the Center for Epidemiologic Studies Depression Scale,21 cognition based on the Telephone Interview for Cognitive Status,22 whether participants required a proxy to respond on their behalf, and measures of household structure and social supports (see eAppendix 2 in the Supplement).
Linked survey data were missing for at least 1 item of interest for 9.9% of admissions in our study sample. In our main analysis, we carried values forward from prior surveys to reduce this proportion to 1.5% and excluded these remaining 1.5% of admissions.
In unadjusted analyses of 2000-2012 data, we compared the proportion of admissions that were followed by readmission across different categories of each patient characteristic. We then fitted a logistic regression model predicting 30-day readmission as a function of the variables used by the CMS for risk adjustment of hospital readmission rates (age, sex, discharge diagnosis, and condition indicators), alternately adding each additional characteristic to test whether it independently predicted readmission after standard adjustments by the CMS. In these models, we also included indicators for the quintile of the admitting hospital’s publicly reported hospital-wide readmission rate to hold hospital performance constant, because the focus of this analysis was the within-quintile association between each additional characteristic and readmission. That is, if a characteristic were more common among hospitals with readmission rates that are high because of poor quality of care, we would not want to conclude from such clustering that the characteristic is a consistent predictor of readmission for which CMS might consider adjustment. In a sensitivity analysis, we modeled the interaction between these characteristics and the hospital quintile to test whether the association between each characteristic and readmission was similar across quintiles (see eAppendix 3 in the Supplement). We assumed similarity across quintiles when subsequently examining the effects of additional adjustments on between-quintile differences in the probability of readmission.
In unadjusted analyses focusing on admissions from 2009 to 2012, we then compared the distribution of patient characteristics between hospitals in the highest vs lowest quintile of publicly reported hospital-wide readmission rates. Finally, we estimated the difference in the probability of readmission between participants admitted to hospitals with higher vs lower hospital-wide readmission rates by including indicators for the admitting hospital’s quintile in a logistic regression model of readmission. To examine how this difference was affected by adjustment for additional patient characteristics, we sequentially added to this model subsets of characteristics as covariates (see eMethods in the Supplement for model specification). We report differences in the probability of readmission between participants admitted to hospitals in the highest vs lowest quintile of readmission rates (see eMethods in the Supplement) because we expected small differences in readmission probabilities among the middle quintiles based on publicly reported rates and because hospitals in the highest quintile were substantially more likely to receive a high penalty than other hospitals (see eAppendix 4 in the Supplement).23 We also report the reduction in the between-quintile difference in the probability of readmission due to each successive subset of characteristics, using bootstrap methods to estimate 95% CIs for the reductions.
We performed several sensitivity analyses (eMethods in the Supplement). First, we weighted analyses to address the lack of linkage of some participants to Medicare data. Second, we repeated our analyses without survey weights, alternately including and excluding nursing home residents to assess their impact on results. Third, for hospitals with at least 20 admissions in our sample, we estimated a multilevel model of readmission with hospital random effects to estimate changes in hospital variation in readmission rates associated with adjustment for additional patient characteristics (eMethods in the Supplement).24 Fourth, using publicly available data from the CMS,25 we assessed the distribution of HRRP penalties in 2014 (which use data from 2009 to 2012) across quintiles of hospitals (defined by hospital-wide readmission rates) for all US hospitals vs the hospitals captured in our study sample (eAppendix 4 in the Supplement). Finally, we repeated analyses using multiple imputation instead of carrying the last observation forward to handle missing data.26
In a supplementary analysis, we assessed the extent to which a zip code–level composite index of 17 sociodemographic indicators of deprivation reduced the difference in the probability of readmission between participants admitted to hospitals in the highest vs lowest quintile of readmission rates, when added to standard CMS adjustments.27-29 In all analyses, we used robust design-based variance estimators to account for clustering within geographic areas, hospitals, or participants and HRS survey weights to account for the survey design and survey nonresponse.30 All analyses were performed with the survey package (version 3.30-3) in R (version 3.1.2; R Foundation).31,32
Our study sample included 33 158 index admissions from 2000 to 2012 for 8767 Medicare beneficiaries in the HRS and 8067 index admissions from 2009 to 2012 for 3470 beneficiaries in the HRS. In unadjusted analyses of the 2000-2012 sample (Table 1 and eAppendix 5 in the Supplement), the proportion of admissions followed by readmission significantly differed across categories for 27 of the 29 patient characteristics not included in CMS adjustments (P ≤ .02 for all comparisons). Of these characteristics, 22 remained significantly predictive of readmission after standard CMS adjustments (P ≤ .04). Associations between these characteristics and readmission were similar across quintiles of the admitting hospital’s publicly reported readmission rate (eAppendix 3 in the Supplement).
In unadjusted analyses of admissions from 2009 to 2012, the characteristics of participants with index admissions to hospitals in the highest quintile of publicly reported readmission rates differed substantially from those with index admissions to hospitals in the lowest quintile of readmission rates (Table 2). Of the 22 characteristics significantly predictive of readmission after standard CMS adjustments, 17 were distributed differently between the highest and lowest quintiles (P ≤ .04), with almost all of these differences (16 of 17) indicating that participants admitted to hospitals in the highest quintile of readmission rates were more likely to have characteristics associated with a higher probability of readmission. Quiz Ref IDFor example, participants admitted to hospitals in the highest quintile had higher HCC scores, more chronic conditions, less education, fewer assets, worse self-reported health status, more depressive symptoms, worse cognition, worse physical functioning, and more difficulties with ADLs and IADLs than participants admitted to hospitals in the lowest quintile. Differences between quintiles in patient characteristics assessed from Medicare enrollment and claims data were similar when estimated using a 20% sample of Medicare beneficiaries from 2009 to 2012 (eAppendix 1 in the Supplement).
Table 3 describes the effects of successive adjustments for patient characteristics on the difference in the probability of readmission between participants admitted to hospitals in the highest vs lowest quintile of readmission rates. Quiz Ref IDThis difference decreased from 5.86 percentage points without any adjustment to 4.41 percentage points after standard CMS adjustments (reduction in difference: −1.45 percentage points; 95% CI, −2.63 to −0.48), to 3.50 percentage points after adjustment for additional variables from Medicare enrollment and claims data (additional reduction: −0.91; 95% CI, −1.78 to −0.04), to 2.29 after additional adjustment for variables from HRS surveys (additional reduction: −1.21; 95% CI, −2.07 to −0.21). The fully adjusted difference constituted a 61% reduction relative to the unadjusted difference and a 48% reduction relative to the difference adjusted for variables already used by the CMS for risk adjustment of readmission rates, or an absolute reduction of −2.12 percentage points (95% CI, −3.33 to −0.67; P = .003). Similar reductions were observed in a sensitivity analysis excluding index admissions that were also readmissions. Adding the area deprivation index to the model with standard CMS adjustments reduced the between-quintile difference minimally.
A multilevel model estimating between-hospital variation in readmission rates in the sample similarly demonstrated a substantial reduction in between-hospital variation in readmission rates after adjustment for more patient characteristics (eAppendix 6 and eFigure in the Supplement). The distribution of penalties assessed by the HRRP in 2014 across all US hospitals, when categorized into quintiles based on hospital-wide readmission rates, was similar to the distribution of penalties across quintiles of hospitals in our study sample (eAppendix 4 in the Supplement). Weighting analyses to account for incomplete linkage to Medicare claims, including nursing home residents in analyses without survey weights, and use of multiple imputation to address item nonresponse did not substantively alter our conclusions.
Quiz Ref IDIn this nationally representative study of readmissions in the Medicare population, many patient characteristics not currently included in risk adjustment of hospital readmission rates were significantly predictive of readmission and more prevalent at hospitals with higher publicly reported readmission rates. In our study sample, additional adjustment for these characteristics accounted for approximately half of the observed difference in the probability of readmission between patients admitted to hospitals in the highest vs lowest quintiles of publicly reported readmission rates. These findings suggest that differences in patient characteristics between hospitals may contribute substantially to the penalties levied by Medicare on hospitals with high readmission rates.
The higher prevalence of clinical and social predictors of readmission among patients admitted to hospitals with higher readmission rates is likely driven by factors largely outside of a hospital’s influence. Our findings therefore call into question the extent to which variation in hospital readmission rates reflects quality of care and, by extension, the extent to which this variation should serve as the basis for financial penalties.33,34 The differences in patient characteristics between hospitals with high vs low readmission rates also suggest that the HRRP imposes substantially greater costs on hospitals disproportionately serving patients more likely to be readmitted. Hospitals serving healthier, more socially advantaged patients may not have to devote any resources to achieving a penalty-free readmission rate, whereas hospitals serving sicker, more socially disadvantaged patients may have to devote considerable resources to avoid a penalty. By selectively increasing costs or lowering revenue for hospitals serving patients at greater risk of readmission, the HRRP therefore threatens to deplete hospital resources available to improve overall quality for populations at high risk of poor outcomes.
More detailed risk adjustment by the CMS could help mitigate this risk of exacerbating disparities. Arguments against additional adjustments contend that adjusting for some risk factors—such as race/ethnicity or income—would hold hospitals serving more disadvantaged patients to a lower standard of quality or obscure the poorer quality they might provide.35,36 Appropriate case mix adjustment for more clinical and social factors, however, should not raise these concerns because it would only help to isolate the portion of between-hospital variation in readmissions that is due to differences in hospital quality.33,34,37 After adjustment for income, for example, hypothetically poorer quality provided by a hospital disproportionately serving low-income patients would still be evident (see hypothetical example in eAppendix 7 in the Supplement).
In response to the prospect of penalties, a hospital may target patients at highest risk in its efforts to reduce readmissions—for example, through better discharge planning—thereby potentially reducing disparities to some extent while lowering its overall readmission rate.38 Incentives to reduce readmission rates and within-hospital disparities, however, need not be at cross purposes with the goals of risk adjustment.34 Thus, our findings support legislation calling for the adjustment of readmission rates and other quality measures for patients’ socioeconomic status and more health-related variables.39,40
Because the detailed risk adjustment available for HRS respondents may not be feasible for the CMS on a large scale, alternative payment models may be required to preserve strong incentives to lower readmissions without unfairly penalizing hospitals based on the populations they serve and consequently risking deterioration in quality for patients at high risk of readmission. For example, a hospital’s expected readmission rate could be set at its historical average, with financial rewards for achieving a rate below the historical average and penalties for exceeding it. The expected rate would have to be held constant or constrained gradually over time, since incentives to reduce readmissions would be diminished by a policy requiring continual improvement over the prior year’s performance.41 Alternatively, growth in similarly designed payment models that cover the full spectrum of care and allow providers discretion in identifying avoidable care to target, such as accountable care organization programs, might obviate the need for payment incentives wedded specifically to readmissions.42
Our study had several limitations. Because our study sample was limited to HRS participants, we were unable to assess the impact of additional risk adjustment on readmission rates for individual hospitals. Because the HRS sample is nationally representative, however, we were able to compare samples of patients admitted to hospitals with high vs low readmission rates, and we confirmed that differences between these groups of patients were reflected in the full population of fee-for-service Medicare beneficiaries (eAppendix 1 in the Supplement). In addition, our conclusions were supported by a multilevel model of hospital-level variation in our study sample. Quiz Ref IDThe size of the HRS sample also limited the precision with which we could estimate differences in the probability of readmission between participants admitted to hospitals with high vs low readmission rates or the reduction in this difference due to adjustment for additional patient characteristics. We would not expect the survey design, however, to cause sampling of systematically sicker and more disadvantaged patients when admitted to a hospital with a high readmission rate.
Accounting for a comprehensive array of clinical and social characteristics substantially decreased the difference in patients’ probability of readmission between hospitals with higher vs lower readmission rates. This finding suggests that Medicare is penalizing hospitals to a large extent based on the patients they serve.
Corresponding Author: J. Michael McWilliams, MD, PhD, Department of Health Care Policy, Harvard Medical School, 180 Longwood Ave, Boston, MA 02115 (mcwilliams@hcp.med.harvard.edu).
Published Online: September 14, 2015. doi:10.1001/jamainternmed.2015.4660.
Author Contributions: Dr Barnett had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Barnett, McWilliams.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Barnett.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Barnett, McWilliams.
Obtained funding: McWilliams.
Administrative, technical, or material support: Barnett.
Study supervision: Hsu, McWilliams.
Conflict of Interest Disclosures: Dr Barnett serves as medical advisor for Ginger.io, which has no relationship with this study. No other disclosures are reported.
Funding/Support: This study was supported by grants from the National Institute on Aging (P01 AG032952) and Health Resources and Services Administration (HRSA) (T32-HP10251).
Role of the Funder/Sponsor: The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the HRSA. The Health and Retirement Study is sponsored by the National Institute on Aging (U01AG009740) and is conducted by the University of Michigan.
Additional Contributions: We thank Bruce E. Landon, MD, MBA, MSc, Ateev Mehrotra, MD, MPH, and Alan M. Zaslavsky, PhD, all from the Department of Health Care Policy, Harvard Medical School, for their helpful comments on an earlier draft of this manuscript.
2.Krumholz
HM, Lin
Z, Drye
EE,
et al. An administrative claims measure suitable for profiling hospital performance based on 30-day all-cause readmission rates among patients with acute myocardial infarction.
Circ Cardiovasc Qual Outcomes. 2011;4(2):243-252. doi:
10.1161/CIRCOUTCOMES.110.957498.
PubMedGoogle ScholarCrossref 9.Gu
Q, Koenig
L, Faerberg
J, Steinberg
CR, Vaz
C, Wheatley
MP. The Medicare Hospital Readmissions Reduction Program: potential unintended consequences for hospitals serving vulnerable populations.
Health Serv Res. 2014;49(3):818-837. doi:
10.1111/1475-6773.12150.
PubMedGoogle ScholarCrossref 10.Singh
S, Lin
Y-L, Kuo
Y-F, Nattinger
AB, Goodwin
JS. Variation in the risk of readmission among hospitals: the relative contribution of patient, hospital and inpatient provider characteristics.
J Gen Intern Med. 2014;29(4):572-578. doi:
10.1007/s11606-013-2723-7.
PubMedGoogle ScholarCrossref 13.Health and Retirement Study, public use dataset. Produced and distributed by the University of Michigan with funding from the National Institute on Aging (grant No. NIA U01AG009740). Ann Arbor: University of Michigan.
15.Horwitz
LI, Partovian
C, Lin
Z,
et al. Development and use of an administrative claims measure for profiling hospital-wide performance on 30-day unplanned readmission.
Ann Intern Med. 2014;161(10)(suppl):S66-S75. doi:
10.7326/M13-3000.
PubMedGoogle ScholarCrossref 18.Center for Medicare and Medicaid Services. Chronic Conditions Data Warehouse. 2014.
https://www.ccwdata.org/. Accessed March 25, 2015.
20.Arbaje
AI, Wolff
JL, Yu
Q, Powe
NR, Anderson
GF, Boult
C. Postdischarge environmental and socioeconomic factors and the likelihood of early hospital readmission among community-dwelling Medicare beneficiaries.
Gerontologist. 2008;48(4):495-504. doi:
10.1093/geront/48.4.495.
PubMedGoogle ScholarCrossref 21.Radloff
LS. The CES-D scale a self-report depression scale for research in the general population.
Appl Psychol Meas. 1977;1(3):385-401.
Google ScholarCrossref 24.Bates
D, Mächler
M, Bolker
B, Walker
S. fitting linear mixed-effects models using lme4. ArXiv StatCO. June 2014.
http://arxiv.org/abs/1406.5823. Accessed March 24, 2015.
31.Lumley
T. Analysis of complex survey samples.
J Stat Softw. 2004;9(1):1-19.
Google Scholar 32.R Core Team.
R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2014.
http://www.R-project.org/. Accessed March 25, 2015.