[Skip to Content]
[Skip to Content Landing]
Figure 1.  Risk-Standardized 30-Day Mortality Rates Among Veterans Hospitalized With Heart Failure and Pneumonia From Claims-Based and Clinical Risk-Adjustment Models
Risk-Standardized 30-Day Mortality Rates Among Veterans Hospitalized With Heart Failure and Pneumonia From Claims-Based and Clinical Risk-Adjustment Models

Each point corresponds to the risk-standardized 30-day mortality rate in each quarter between January to March 2009 and July to September 2015 for the reference population of interest. The claims-based model presented for pneumonia did not include an indicator for primary diagnosis.

Figure 2.  Quarterly Trends in Predicted Risk of 30-Day Mortality, as Derived From Clinical and Claims-Based Risk-Adjustment Models
Quarterly Trends in Predicted Risk of 30-Day Mortality, as Derived From Clinical and Claims-Based Risk-Adjustment Models

Each point corresponds to the predicted risk of 30-day mortality in each quarter between January to March 2009 and July to September 2015. The claims-based model presented for pneumonia did not include an indicator for primary diagnosis.

Table 1.  Characteristics of Veterans Hospitalized With Heart Failure and Pneumonia in the First and Last Years of the Studya
Characteristics of Veterans Hospitalized With Heart Failure and Pneumonia in the First and Last Years of the Studya
Table 2.  Changes in 30-Day Mortality Among Veterans Admitted With Heart Failure and Pneumoniaa
Changes in 30-Day Mortality Among Veterans Admitted With Heart Failure and Pneumoniaa
Supplement.

eFigure 1. Exclusion Flowchart for the Study Population

eFigure 2. Number of Admissions per Quarter for the Study Population

eFigure 3. Percent of Admissions per Quarter with a Primary Diagnosis of Sepsis/RF [with Pneumonia as Secondary Diagnosis] in Extended Pneumonia Cohort

eFigure 4. Unadjusted 30-day Mortality Rates for Veterans Hospitalized with Heart Failure and Pneumonia

eFigure 5. Unadjusted 30-day Mortality Rates for Veterans Hospitalized with Heart Failure, Pneumonia, Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]

eFigure 6. Bayesian Structural Time Series Analyses

eFigure 7. Risk-Standardized 30-Day Mortality Rates Among Veterans Hospitalized with Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis] from Claims-Based and Clinical Risk-Adjustment Models

eFigure 8. Quarterly Trends in Predicted Risk of 30-day Mortality, as Derived from Clinical and Claims-Based Risk-adjustment Models for Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]

eTable 1. ICD-9 Codes for Primary Diagnoses

eTable 2. Summary of Missing Data for Adjustment/Summary Variables

eTable 3. Characteristics of Veterans Hospitalized with Pneumonia [excluding primary diagnosis of Sepsis/RF] Between the First and Last Years of the Study

eTable 4. Characteristics of Veterans Hospitalized with Sepsis/RF [with Pneumonia as Secondary Diagnosis] Between the First and Last Years of the Study

eTable 5. Characteristics of Veterans Hospitalized with Heart Failure Between the First and Last Years of the Study

eTable 6. Characteristics of Veterans Hospitalized with Pneumonia Between the First and Last Years of the Study

eTable 7. Comparison of Clinical Characteristics Between First and Last Years of Study for Patients Admitted with Heart Failure

eTable 8. Comparison of Clinical Characteristics Between First and Last Years of Study for Patients Admitted with Pneumonia

eTable 9. Comparison of Sociodemographic Characteristics Between First and Last Years for Patients Admitted with Heart Failure or Pneumonia

eTable 10. Changes in 30-day Mortality among Veterans Admitted with Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]

eTable 11. Analyses for Area under the ROC curve (AUC)/Goodness of Fit for Claims-based and Clinical Risk-adjustment Models

eTable 12. Sensitivity Analyses for Multiple Admissions – First Admission

eTable 13. Sensitivity Analyses for Multiple Admissions – Random Admission

1.
Kizer  KW, Dudley  RA.  Extreme makeover: Transformation of the veterans health care system.  Annu Rev Public Health. 2009;30(1):313-339. doi:10.1146/annurev.publhealth.29.020907.090940PubMedGoogle ScholarCrossref
2.
Joynt  KE, Orav  EJ, Zheng  J, Jha  AK.  Public Reporting of Mortality Rates for Hospitalized Medicare Patients and Trends in Mortality for Reported Conditions.  Ann Intern Med. 2016;165(3):153-160. doi:10.7326/M15-1462PubMedGoogle ScholarCrossref
3.
US Department of Veterans Affairs. Strategic Analytics for Improvement and Learning (SAIL). https://www.va.gov/QUALITYOFCARE/measure-up/Strategic_Analytics_for_Improvement_and_Learning_SAIL.asp. Accessed November 16, 2019.
4.
Lee  JS, Nsa  W, Hausmann  LRM,  et al.  Quality of care for elderly patients hospitalized for pneumonia in the United States, 2006 to 2010.  JAMA Intern Med. 2014;174(11):1806-1814. doi:10.1001/jamainternmed.2014.4501PubMedGoogle ScholarCrossref
5.
Chen  J, Normand  S-LT, Wang  Y, Krumholz  HM.  National and regional trends in heart failure hospitalization and mortality rates for Medicare beneficiaries, 1998-2008.  JAMA. 2011;306(15):1669-1678. doi:10.1001/jama.2011.1474PubMedGoogle ScholarCrossref
6.
Kaboli  PJ, Go  JT, Hockenberry  J,  et al.  Associations between reduced hospital length of stay and 30-day readmission rate and mortality: 14-year experience in 129 Veterans Affairs hospitals.  Ann Intern Med. 2012;157(12):837-845. doi:10.7326/0003-4819-157-12-201212180-00003PubMedGoogle ScholarCrossref
7.
Trivedi  AN, Grebla  RC.  Quality and equity of care in the veterans affairs health-care system and in medicare advantage health plans.  Med Care. 2011;49(6):560-568. doi:10.1097/MLR.0b013e31820fb0f6PubMedGoogle ScholarCrossref
8.
Shah  RU, Tsai  V, Klein  L, Heidenreich  PA.  Characteristics and outcomes of very elderly patients after first hospitalization for heart failure.  Circ Heart Fail. 2011;4(3):301-307. doi:10.1161/CIRCHEARTFAILURE.110.959114PubMedGoogle ScholarCrossref
9.
Heidenreich  PA, Sahay  A, Kapoor  JR, Pham  MX, Massie  B.  Divergent trends in survival and readmission following a hospitalization for heart failure in the Veterans Affairs health care system 2002 to 2006.  J Am Coll Cardiol. 2010;56(5):362-368. doi:10.1016/j.jacc.2010.02.053PubMedGoogle ScholarCrossref
10.
Lindenauer  PK, Lagu  T, Shieh  MS, Pekow  PS, Rothberg  MB.  Association of diagnostic coding with trends in hospitalizations and mortality of patients with pneumonia, 2003-2009.  JAMA. 2012;307(13):1405-1413. doi:10.1001/jama.2012.384PubMedGoogle ScholarCrossref
11.
Ibrahim  AM, Dimick  JB, Sinha  SS, Hollingsworth  JM, Nuliyalu  U, Ryan  AM.  Association of coded severity with readmission reduction after the Hospital Readmissions Reduction Program.  JAMA Intern Med. 2018;178(2):290-292. doi:10.1001/jamainternmed.2017.6148PubMedGoogle ScholarCrossref
12.
Sukul  D, Hoffman  GJ, Nuliyalu  U,  et al.  Association between Medicare policy reforms and changes in hospitalized Medicare beneficiaries’ severity of illness.  JAMA Netw Open. 2019;2(5):e193290. doi:10.1001/jamanetworkopen.2019.3290PubMedGoogle Scholar
13.
Shahian  DM, Silverstein  T, Lovett  AF, Wolf  RE, Normand  S-LT.  Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards.  Circulation. 2007;115(12):1518-1527. doi:10.1161/CIRCULATIONAHA.106.633008PubMedGoogle ScholarCrossref
14.
Krumholz  HM, Wang  Y, Mattera  JA,  et al.  An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure.  Circulation. 2006;113(13):1693-1701. doi:10.1161/CIRCULATIONAHA.105.611194PubMedGoogle ScholarCrossref
15.
Fonarow  GC, Pan  W, Saver  JL,  et al.  Comparison of 30-day mortality models for profiling hospital performance in acute ischemic stroke with vs without adjustment for stroke severity.  JAMA. 2012;308(3):257-264. doi:10.1001/jama.2012.7870PubMedGoogle ScholarCrossref
16.
Iezzoni  L. Risk Adjustment for Measuring Healthcare Outcomes, Fourth Edition. Health Administration Press; Chicago, IL. 2012.
17.
Volpp  KG, Rosen  AK, Rosenbaum  PR,  et al.  Mortality among patients in VA hospitals in the first 2 years following ACGME resident duty hour reform.  JAMA. 2007;298(9):984-992.PubMedGoogle ScholarCrossref
18.
Ross  JS, Maynard  C, Krumholz  HM,  et al.  Use of administrative claims models to assess 30-day mortality among Veterans Health Administration hospitals.  Med Care. 2010;48(7):652-658. doi:10.1097/MLR.0b013e3181dbe35dPubMedGoogle ScholarCrossref
19.
Sohn  M-W, Arnold  N, Maynard  C, Hynes  DM.  Accuracy and completeness of mortality data in the Department of Veterans Affairs.  Popul Health Metr. 2006;4:2. doi:10.1186/1478-7954-4-2PubMedGoogle ScholarCrossref
20.
Fine  MJ, Auble  TE, Yealy  DM,  et al.  A prediction rule to identify low-risk patients with community-acquired pneumonia.  N Engl J Med. 1997;336(4):243-250. doi:10.1056/NEJM199701233360402PubMedGoogle ScholarCrossref
21.
Rahimi  K, Bennett  D, Conrad  N,  et al.  Risk prediction in patients with heart failure: a systematic review and analysis.  JACC Heart Fail. 2014;2(5):440-446. doi:10.1016/j.jchf.2014.04.008PubMedGoogle ScholarCrossref
22.
Garvin  JH, DuVall  SL, South  BR,  et al.  Automated extraction of ejection fraction for quality measurement using regular expressions in unstructured information management architecture (UIMA) for heart failure.  J Am Med Inform Assoc. 2012;19(5):859-866. doi:10.1136/amiajnl-2011-000535PubMedGoogle ScholarCrossref
23.
Rubin  DB, ed.  Multiple Imputation for Nonresponse in Surveys. Wiley; Hoboken, New Jersey. 1987. doi:10.1002/9780470316696
24.
Kind  AJH, Buckingham  WR.  Making neighborhood-disadvantage metrics accessible - the Neighborhood Atlas.  N Engl J Med. 2018;378(26):2456-2458. doi:10.1056/NEJMp1802313PubMedGoogle ScholarCrossref
25.
Kind  AJH, Jencks  S, Brock  J,  et al.  Neighborhood socioeconomic disadvantage and 30-day rehospitalization: a retrospective cohort study.  Ann Intern Med. 2014;161(11):765-774. doi:10.7326/M13-2946PubMedGoogle ScholarCrossref
26.
Singh  GK.  Area deprivation and widening inequalities in US mortality, 1969-1998.  Am J Public Health. 2003;93(7):1137-1143. doi:10.2105/AJPH.93.7.1137PubMedGoogle ScholarCrossref
27.
Scott  SL, Varian  HR.  Predicting the present with Bayesian structural time series.  IJMMNO. 2014;5(1/2):4. doi:10.1504/IJMMNO.2014.059942Google ScholarCrossref
28.
Sjoding  MW, Iwashyna  TJ, Dimick  JB, Cooke  CR.  Gaming hospital-level pneumonia 30-day mortality and readmission measures by legitimate changes to diagnostic coding.  Crit Care Med. 2015;43(5):989-995. doi:10.1097/CCM.0000000000000862PubMedGoogle ScholarCrossref
29.
Trivedi  AN, Nsa  W, Hausmann  LRM,  et al.  Quality and equity of care in U.S. hospitals.  N Engl J Med. 2014;371(24):2298-2308. doi:10.1056/NEJMsa1405003PubMedGoogle ScholarCrossref
30.
Ody  C, Msall  L, Dafny  LS, Grabowski  DC, Cutler  DM.  Decreases in readmissions credited to Medicare’s program to reduce hospital readmissions have been overstated.  Health Aff (Millwood). 2019;38(1):36-43. doi:10.1377/hlthaff.2018.05178PubMedGoogle ScholarCrossref
31.
Radomski  TR, Fine  MJ, Gellad  WF.  Outcome after admission at Veterans Affairs vs non-Veterans Affairs hospitals.  JAMA. 2016;316(3):345-346. doi:10.1001/jama.2016.5391PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 2,156
    Citations 0
    Original Investigation
    December 20, 2019

    Mortality Trends for Veterans Hospitalized With Heart Failure and Pneumonia Using Claims-Based vs Clinical Risk-Adjustment Variables

    Author Affiliations
    • 1Department of Biostatistics, Brown University School of Public Health, Providence, Rhode Island
    • 2Providence VA Medical Center, Providence, Rhode Island
    • 3Department of Health Services, Policy and Practice, Brown University School of Public Health, Providence, Rhode Island
    • 4Center for Health Equity Research and Promotion, Virginia Pittsburgh Healthcare System, Pittsburgh, Pennsylvania
    • 5School of Medicine, Division of General Internal Medicine, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania
    • 6Center for Healthcare Organization and Implementation Research, Virginia Boston Healthcare System, Boston, Massachusetts
    • 7Boston University School of Medicine, Boston, Massachusetts
    JAMA Intern Med. 2020;180(3):347-355. doi:10.1001/jamainternmed.2019.5970
    Key Points

    Question  Do trends in 30-day mortality rates among veterans hospitalized with heart failure (HF) and pneumonia differ according to whether claims-based or clinical variables are used in risk-adjustment models?

    Findings  In this observational time-trend study that included 146 924 hospitalizations for HF and 131 325 for pneumonia from 2009 to 2015, the estimated decline in 30-day mortality rates per quarter was substantially smaller when using clinical variables for risk adjustment (HF, −0.017%; pneumonia, −0.026%) than when using claims-based variables (HF, −0.051%; pneumonia, −0.084%).

    Meaning  Compared with clinical risk adjustment, claims-based adjustment is likely to overestimate temporal declines in mortality for veterans hospitalized with HF and pneumonia.

    Abstract

    Importance  Prior studies have reported declines in mortality for patients admitted to Veterans Health Administration (VA) and non-VA hospitals using claims-based risk adjustment. These apparent mortality reductions may be influenced by changes in coding practices.

    Objective  To compare trends in the VA for 30-day mortality following hospitalization for heart failure (HF) and pneumonia using claims-based and clinical risk-adjustment models.

    Design, Setting, and Participants  This observational time-trend study analyzed admissions to a VA Medical Center with a principal diagnosis of HF, pneumonia, or sepsis/respiratory failure (RF) with a secondary diagnosis of pneumonia. Exclusion criteria included having less than 12 months of VA enrollment, being discharged alive within 24 hours, leaving against medical advice, and hospice utilization.

    Exposures  Admission to a VA hospital from January 2009 through September 2015.

    Main Outcomes and Measures  The primary outcome was 30-day, all-cause mortality. All models included age and sex. Claims-based covariates included 22 (30) comorbidities for HF (pneumonia). Clinical covariates included vital signs, laboratory values, and ejection fraction.

    Results  Among the 146 924 HF admissions, the mean (SD) age was 71.6 (11.4) years and 144 502 (98.4%) were men; among the 131 325 admissions for pneumonia, the mean (SD) age was 70.8 (12.3) years and 127 491 (97.1%) were men. Unadjusted 30-day mortality rates were 6.45% (HF) and 11.22% (pneumonia). Claims-based models showed an increased predicted risk of 30-day mortality over time (0.019 percentage points per quarter for HF [95% CI, 0.015 to 0.023]; 0.053 percentage points per quarter for pneumonia [95% CI, 0.043 to 0.063]). Clinical models showed declines or no change in predicted risk (−0.014 percentage points per quarter for HF [95% CI, −0.020 to −0.008]; −0.004 percentage points per quarter for pneumonia [95% CI, −0.017 to 0.008]). Claims-based risk adjustment yielded declines in 30-day mortality of 0.051 percentage points per quarter for HF (95% CI, −0.074 to −0.027) and 0.084 percentage points per quarter for pneumonia (95% CI, −0.111 to −0.056). Models adjusting for clinical covariates attenuated or eliminated these changes for HF (−0.017 percentage points per quarter; 95% CI, −0.039 to 0.006) and for pneumonia (−0.026 percentage points per quarter; 95% CI, −0.052 to 0.001). Compared with the claims-based models, the clinical models for HF and pneumonia more accurately differentiated between patients who died after 30 days and those who did not.

    Conclusions and Relevance  Among HF and pneumonia hospitalizations, adjusting for clinical covariates attenuated declines in mortality rates identified using claims-based models. Assessments of temporal trends in 30-day mortality using claims-based risk adjustment should be interpreted with caution.

    Introduction

    Over the past decade, health care systems in the United States have invested substantial resources to measure and reduce mortality rates for hospitalized patients with common and costly medical conditions.1 For example, since 2008 the Centers for Medicare & Medicaid Services (CMS) have publicly reported 30-day mortality rates for admissions for acute myocardial infarction (AMI), heart failure (HF), and pneumonia to virtually all acute-care hospitals in the United States.2 The Centers for Medicare & Medicaid Services include mortality measures in their Hospital Value-based Payment Program, which financially rewards these hospitals for better quality of care. In the Veterans Health Administration (VA), the largest integrated delivery system in the United States, hospital mortality rates (calculated using the CMS approach) inform the Strategic Analytics for Improvement and Learning (SAIL) composite quality score. The VA uses SAIL to monitor and rank performance across VA Medical Centers (VAMCs) on a national level.3

    Previous studies have identified significant improvements in hospital and population-level outcomes for AMI, HF, and pneumonia among Medicare beneficiaries and in the VA Healthcare System.2,4-10 To account for differences in the characteristics of patients across hospitals and/or over time, these studies have adjusted for the presence of comorbid conditions identified in claims or other administrative data. A key concern in the use of claims-based models is that they may be influenced by secular changes in hospitals’ diagnostic coding practices, particularly because comorbid diagnoses often have flexible clinical criteria. Thus, apparent reductions in hospital- and population-level mortality rates may reflect changes in coding practices rather than actual improvements in outcomes.10,11 Prior work has found that the number of comorbidities among hospitalized patients has increased over time.5,6,9,12

    Risk adjustment using clinical data (eg, vital signs, laboratory data) provides an alternative to claims-based risk-adjustment. Clinical models often outperform claims-based models in predicting hospital mortality at the individual patient level and have greater face validity among clinicians.13-15 Furthermore, whereas claims-based models assess the burden of comorbid conditions, clinical data can provide a measure of the severity of the primary illness for which a patient is admitted.16 However, it is unknown whether the use of clinical models changes VA system–level trends in hospital mortality over time. In this study, we leverage the availability of clinical data from the VA Healthcare System to examine trends in 30-day mortality following hospitalization for HF or pneumonia from 2009 to 2015 using 2 alternative risk-adjustment models: (1) an approach using claims-based comorbidities specified by the CMS model and (2) a model using objective clinical covariates.

    Methods
    Data Sources/Study Population

    We retrieved VA administrative claims data and clinical records from the VA’s Corporate Data Warehouse (CDW). The data included VA hospital admissions, vital signs, laboratory values, outpatient claims, vital status, and fee-basis files of VA-financed health services that occur in non-VA settings.

    Analyses were restricted to veterans with at least 1 inpatient admission to a VAMC with a principal discharge diagnosis of HF, pneumonia, and sepsis or respiratory failure (RF) with a secondary diagnosis of pneumonia between January 2009 and September 2015. Although patients with a principal diagnosis of sepsis/RF and secondary diagnosis of pneumonia were excluded from calculations for pneumonia mortality during the years of the study, CMS changed the specifications for the pneumonia mortality measure to include patients with a primary diagnosis of sepsis in 2016. Therefore, we defined the pneumonia cohort by including patients with a primary diagnosis of sepsis or RF and a secondary diagnosis of pneumonia (n = 27 272), consistent with a prior study.10 The specific International Classification of Diseases, Ninth Revision (ICD-9) codes used to establish these diagnoses are in eTable 1 in the Supplement. Consistent with CMS and VA measure specifications, we excluded veterans with less than 12 months of continuous VA enrollment in the year prior to the selected admission, discharged alive within 24 hours of an HF or pneumonia admission, who left against medical advice, and with hospice utilization 1 year prior to admission. Because the unit of analysis was admission, we included more than 1 admission for the same veteran. In sensitivity analyses, we only considered a veteran’s first admission, as well as a random admission within the study period.17,18 The eligible study population consisted of 146 924 admissions for HF and 131 325 admissions for pneumonia across 132 VAMCs (eFigures 1 and 2 in the Supplement).

    The Providence VA's institutional review board approved the study protocol and waived the need for informed consent because the research because the research involved no more than minimal tangible or intangible risk to study participants.

    Variables

    The outcome variable was an indicator for death from any cause in the 30 days after hospital admission that was derived from the VA vital status file, which is comparable to the National Death Index with respect to completeness and accuracy.19 The outcome was attributed to the VAMC to which the veteran was initially admitted, regardless of postadmission transfers to another VAMC or a non-VA hospital.

    The primary independent variable was the quarter in which an admission occurred. Each admission corresponded to 1 of the 27 quarters between January 2009 and September 2015.

    The adjustment variables were in 3 categories: demographic (age and sex), claims-based, and clinical. The full list of claims-based and clinical variables considered for HF and pneumonia are included in the first column of Table 1.

    The claims-based variables were obtained using VA administrative data from the index admission and outpatient and inpatient encounter data for the 12 months preceding admission. The claims-based variables were identical to those included in CMS risk-adjustment models for hospital profiling, which includes different comorbidities for HF and pneumonia. We did not include comorbid conditions from Medicare claims, consistent with VA’s current methodology and a prior study reporting that adding Medicare diagnostic claims for dual users (of VA and Medicare) had little impact on hospital-level risk-standardized mortality rates.19 Furthermore, we excluded such claims to avoid the influence of coding practices in Medicare affecting measured comorbidities in the study population.

    The clinical variables were extracted from medical record data in CDW and chosen based on prior literature, clinical face validity, availability, and feasibility of extraction.20,21 They included systolic and diastolic blood pressure, heart rate, respiratory rate, oxygen saturation, body mass index (BMI, calculated as weight in kilograms divided by height in meters squared), sodium, potassium, blood urea nitrogen, creatinine, hematocrit, serum glucose, an indicator for inpatient admission 30 days prior to the index admission, B-type natriuretic peptide, and left ventricular ejection fraction. Natural language processing methods were used to extract ejection fraction for patients with HF from text sources in the VA's computerized personal record system.22 Each clinical covariate was derived during the period of 2 days prior to admission until 1 day after admission using an approach identifying the value closest to the admission time for those with multiple values. The percentage of missing values for each of the clinical variables is presented in eTable 2 in the Supplement; none have a frequency of missing exceeding 10% except for B-type natriuretic peptide (12.7%), BMI (15.0%), and ejection fraction (33.4%) in the HF cohort. For missing data, we used multiple imputation methods to generate 5 completed data sets for analysis.23

    We additionally reported the racial/ethnic composition of the HF and pneumonia admissions, as well as the average area deprivation index (ADI), a composite measure of neighborhood disadvantage, but did not include these demographic variables as risk-adjusters, which is consistent with CMS methodology.24-26

    Statistical Analyses

    We computed unadjusted mortality rates for each quarter by condition and conducted time series analyses on unadjusted mortality rates using a Bayesian structural time series model to account for seasonal patterns in mortality prior to examining the trend.27 To account for potential temporal differences in the case-mix of veterans that comprised each of the 27 quarters, we estimated directly standardized mortality rates using 3 sets of adjustment variables (1) age and sex, (2) the claims-based model, which includes age, sex, and the set of claims-based variables used by CMS/VA for hospital profiling, and (3) a model based on age, sex, and the clinical variables.

    For all 3 sets of adjustment variables, we first fitted a patient-level logistic regression model that included indicator variables for VAMCs specifying 30-day mortality as the outcome and admission quarter and the relevant set of adjustor variables as predictors.2 We recorded the area under the receiver operating characteristic curve (AUC) statistic and the Hosmer-Lemeshow goodness of fit statistic. The AUC statistic provides a measure of how well the model distinguishes between patients who experienced the outcome of interest vs those who did not, with higher values of the AUC statistic indicating better performance of the model. The DeLong Method was used to determine if differences in the AUCs for claims-based and clinical models were statistically significant. The Hosmer-Lemeshow goodness of fit statistic provides a measure of how well predictions from the claims-based and clinical models match observed mortality. Second, using each model's estimates, the probability of death for each individual in the reference population was estimated using all of their observed values for the relevant adjustor variables fixed at the first quarter. The reference population for HF (pneumonia) was the full set of all 146 924 (131 325) admissions. Finally, the standardized mortality rate for this set of variables in this specific quarter was determined by averaging these probabilities of death. This process was repeated for the other 26 quarters.

    These 27 standardized mortality rates can be directly compared across time because they were estimated using a common reference population. We compared quarterly adjusted mortality rates across time by regressing them on time. The slope represents the estimated change in mortality rates per quarter. For models including clinical data, we performed statistical analyses on the 5 imputed data sets and derived combined estimates using multiple imputation rules.23

    We also computed the estimated risk per quarter using the 3 sets of adjustment variables. For this, we fit the same patient-level logistic regression model (excluding admission quarter as a predictor) and averaged the estimated probability of death for each quarter.

    Statistical analyses were performed using SAS statistical software (version 9.4; SAS Institute, Inc) and R statistical software (version 3.4.3, R Foundation).

    Results
    Characteristics of the Study Population

    Among the 146 924 HF admissions, the mean (SD) age was 71.6 (11.4) years and 144 502 (98.4%) were men. Comparing admissions for HF in the first and last years of the study (Table 1), 20 of the 22 individual claims-based comorbid conditions increased in prevalence between 2009 and 2015. The mean number of claims-based comorbidities increased significantly from 6.0 to 6.5. The largest increases in frequency were for cardiorespiratory failure and shock (8.7 percentage points) and renal failure (7.6 percentage points). In contrast, clinical measures for HF appeared to improve over time: systolic blood pressure, oxygen saturation, hematocrit, and ejection fraction increased, whereas respiratory rates decreased. Moreover, there was no evidence that the renal function of admitted patients, as measured by serum creatinine and blood urea nitrogen, declined over time.

    Among the 131 325 admissions for pneumonia, the mean (SD) age was 70.8 (12.3) years and 127 491 (97.1%) were men. Table 1 shows that the proportion of the pneumonia cohort with a primary diagnosis of sepsis/RF and a secondary diagnosis of pneumonia increased from 1958 (10.8%) in the first year to 6828 (33.2%) in the last year (eFigure 3 in the Supplement). Between the first and last years, the prevalence for 27 of 30 comorbid conditions increased, with the largest increases observed for cardiorespiratory failure and shock (8.5 percentage points) and renal failure (8.2 percentage points). The average number of claims-based comorbidities rose from 5.0 to 5.5 in the pneumonia cohort. Signs of lower clinical severity increased over time—systolic blood pressure, oxygen saturation, and hematocrit increased, whereas respiratory rate declined. Serum creatinine and blood urea nitrogen levels remained stable. We observed similar findings in stratified analyses for the pneumonia cohort by primary diagnosis (pneumonia vs sepsis/RF) (eTables 3 and 4 in the Supplement).

    The supplementary material presents standardized differences in patient characteristics comparing the first and last years (eTables 5 and 6 in the Supplement) as well as the fraction of patients with abnormal clinical values (eTables 7 and 8 in the Supplement). eTable 9 in the Supplement shows that the racial, ethnic, and socioeconomic characteristics of patients hospitalized for HF and pneumonia were stable over time.

    Mortality Trends

    The overall 30-day mortality rates were 6.45% for HF and 11.22% for pneumonia. From the first year to the last year, the unadjusted 30-day mortality rate decreased from 6.64% to 6.29% for HF and from 11.09% to 10.26% for pneumonia (eFigure 4 in the Supplement). Mortality for admissions with a primary diagnosis of sepsis/RF and a secondary diagnosis of pneumonia decreased from 33.15% in the first year to 18.29% in the last (eFigure 5 in the Supplement). Bayesian structural time series analyses confirmed that unadjusted mortality rates for all 3 conditions declined (eFigure 6 in the Supplement).

    As shown in Table 2, adjusted mortality rates following hospitalization for HF declined from 6.90% in the first year to 6.01% in the last year in the claims-based risk-adjustment model and from 6.48% to 6.39% in the clinical model. For the pneumonia cohort, the claims-based risk-adjustment model showed a decline in adjusted mortality rates from 11.85% to 9.67%; the clinical risk-adjustment model showed a smaller decline from 11.19% to 10.46%. Figure 1 graphs quarterly trends in standardized 30-day mortality rates. The adjusted quarterly change in mortality following hospitalization for HF was −0.051 percentage points (95% CI, −0.074 to −0.027) in the claims-based model and −0.017 percentage points (95% CI, −0.039 to 0.006) in the clinical model. The adjusted quarterly change in the 30-day mortality rate following hospitalization for pneumonia was −0.084 percentage points (95% CI, −0.111 to −0.056) in the claims-based model and −0.026 percentage points (95% CI, −0.052 to 0.001) in the clinical model. In stratified analyses of the pneumonia cohort by primary diagnosis, clinical models attenuated estimated declines in risk-adjusted mortality compared with claims-based models (eTable 10, eFigure 7 in the Supplement).

    Claims-based models had AUC values of 0.689 (HF) and 0.709 (pneumonia); clinical models had AUCs of 0.777 (HF) and 0.750 (pneumonia) (Table 2). Regardless of whether the indicator variables for VAMC were included, adjustment models that included clinical variables had higher AUC statistics than those based on claims-based variables for each condition, demonstrating that clinical models had an improved ability to distinguish between patients who died within 30 days and those who did not (eTable 11 in the Supplement). Hosmer-Lemeshow goodness-of-fit statistics showed that predictions from clinical models more closely matched observed mortality status than claims-based models.

    Sensitivity analyses for multiple admissions showed similar results (eTables 12-13 in the Supplement).

    Predicted Mortality Risk

    Claims-based models showed an increase in the predicted risk of 30-day mortality over time (Figure 2). For HF, the estimated increase in risk per quarter was 0.019 percentage points (95% CI, 0.015-0.023); for pneumonia, the estimated increase in risk per quarter was 0.053 percentage points (95% CI, 0.043-0.063).

    In contrast, models adjusting for clinical variables showed a decline or no significant change in risk over time (Figure 2). The estimated decrease in risk per quarter was 0.014 percentage points for HF (95% CI, −0.020 to −0.008) and 0.004 percentage points for pneumonia (95% CI, −0.017 to 0.008). Similar patterns were identified for the predicted mortality risks for pneumonia (excluding sepsis/RF) and sepsis/RF (eFigure 8 in the Supplement).

    Discussion

    In this study of temporal trends in 30-day mortality for HF and pneumonia admissions between January 2009 and September 2015, we observed substantial declines in 30-day mortality in the VA using a claims-based risk-adjustment approach. Risk adjustment using clinical variables attenuated or eliminated these apparent decreases in hospital mortality. Compared with claims-based models, those that adjust for clinical variables more accurately distinguished between patients who died within 30 days and those who did not and generated predictions of mortality that more closely matched observed mortality rates.

    This study identified 2 mechanisms by which claims-based risk adjustment may overstate temporal declines in hospital mortality. First, hospitals appear to have increased the number and type of coded comorbid conditions over time. For instance, we found that the proportion of patients with a coded diagnosis of renal failure or shock/RF increased temporally across all cohorts, but clinical data measured at or near the time of admission showed no evidence of worsening renal function, decreased oxygen saturation, or increased respiratory rates over time. Whereas the predicted risk of mortality increased over time in claims-based models, clinical variable-based models indicated that the predicted risk of mortality declined or remained constant. A time trend analysis by Heidenreich et al9 for HF in the VA also identified increases in comorbid disease coding and improvements in most laboratory values between 2002 and 2006. The present study preceded the implementation of ICD-10; the adoption of this expanded set of diagnostic codes may further increase opportunities to report comorbid conditions.

    Second, hospitals may assign high-risk patients to alternative principal diagnoses, which would remove such patients from calculations of mortality. For instance, we found that the number of patients with coded sepsis/RF and secondary diagnoses of pneumonia more than tripled between the first and last year of the study. Furthermore, the predicted risk of mortality among these patients (as derived from the clinical model) declined markedly over time. Our findings are broadly consistent with analyses of Medicare data that have reported increases in the number of patients with a principal diagnostic code of sepsis/RF and raised concerns about hospitals’ increased labeling of pneumonia patients with a principal diagnosis of sepsis or RF.10,28

    Our study focused on hospitalized veterans and was strengthened by the availability of clinical data in the VA’s health information systems. However, it is likely that the findings of this study extend beyond the VA. Studies of hospitalized Medicare beneficiaries have also identified substantial increases in coded comorbidities over time.4,29,30 Most private-sector hospitals in the United States are subject to payment penalties or increased reimbursements on the basis of risk-adjusted outcomes. Thus, private-sector hospitals face strong incentives to comprehensively document comorbid conditions under Medicare’s value-based payments. Our findings therefore suggest that assessments of temporal trends in hospital mortality using claims-based risk adjustment should be interpreted with caution within and beyond the VA. In addition, the higher performance of clinical risk-adjustment models, as measured by AUC and goodness-of-fit statistics, compared with those using claims alone implies that the VA should continue allocating resources toward the collection of clinical data, even as it transitions to an alternative electronic medical record. Other health care systems and payers (including Medicare) should consider collecting and reporting clinical data in a standardized format and using these data for risk adjustment.

    Limitations

    This study has limitations. Risk adjustment using vital signs and laboratory values may not capture important measures of severity, such as altered mental status, substance abuse, or frailty. Some of the clinical variables in this study were missing. However, most of these had small percentages of missing data, and we used multiple imputation to account for their absence. Our results may not generalize to other conditions beyond HF and pneumonia. Nevertheless, both CMS and the VA have prioritized these conditions in public reporting of hospital mortality rates. We did not examine the effect of including Medicare claims on coded comorbidities among the subset of Veterans eligible for Medicare. Given prior studies, the inclusion of this data would likely produce further increases in the number and type of claims-based conditions in the study population over time.5,6,9 Finally, other observers have noted that the VA likely undercodes comorbid conditions compared with Medicare-financed hospitals; if true, then the discordance between claims-based and clinical models is likely even greater in non-VA hospitals.31

    Conclusions

    Using the CMS claims-based risk-adjustment methodology, 30-day mortality rates for veterans hospitalized with HF and pneumonia appeared to substantially decline between 2009 and 2015. However, adjustment for objective clinical covariates nearly eliminated these apparent improvements in patient outcomes. Assessments of temporal trends in mortality using claims-based risk adjustment should be interpreted with caution, and health systems should invest in the collection of more objective clinical information for risk adjustment.

    Back to top
    Article Information

    Corresponding Author: Amal N. Trivedi, MD, MPH, 121 S Main St, 7th Floor, Providence, RI 02903 (amal_trivedi@brown.edu).

    Accepted for Publication: October 13, 2019.

    Published Online: December 20, 2019. doi:10.1001/jamainternmed.2019.5970

    Author Contributions: Ms Silva had full access to all data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Study concept and design: Silva, Wu, Mor, Fine, Trivedi.

    Acquisition, analysis, or interpretation of data: Silva, Jiang, Gutman, Mor, Fine, Kressin, Trivedi.

    Drafting of the manuscript: Silva, Trivedi.

    Critical revision of the manuscript for important intellectual content: Silva, Jiang, Gutman, Wu, Mor, Fine, Kressin.

    Statistical analysis: Silva, Jiang, Gutman.

    Obtained funding: Wu, Mor, Trivedi.

    Administrative, technical, or material support: Mor.

    Study supervision: Mor, Fine, Trivedi.

    Conflict of Interest Disclosures: Ms Silva reported grants from Department of Veterans Affairs during the conduct of the study. Dr Gutman reported grants from VA during the conduct of the study. Dr Mor reported personal fees from naviHealth, Inc, outside the submitted work. Dr Trivedi reported grants from Department of Veterans Affairs during the conduct of the study. No other disclosures were reported.

    Funding/Support: Research reported in this publication was supported by the Department of Veterans Affairs Health Services Research and Development Service under Grant Number I01 HX002104. Dr Kressin was supported by a Senior Department of Veterans Affairs Health Services Research and Development Research Career Scientist award (RCS 02-066).

    Role of the Funder/Sponsor: The Department of Veterans Affairs had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

    References
    1.
    Kizer  KW, Dudley  RA.  Extreme makeover: Transformation of the veterans health care system.  Annu Rev Public Health. 2009;30(1):313-339. doi:10.1146/annurev.publhealth.29.020907.090940PubMedGoogle ScholarCrossref
    2.
    Joynt  KE, Orav  EJ, Zheng  J, Jha  AK.  Public Reporting of Mortality Rates for Hospitalized Medicare Patients and Trends in Mortality for Reported Conditions.  Ann Intern Med. 2016;165(3):153-160. doi:10.7326/M15-1462PubMedGoogle ScholarCrossref
    3.
    US Department of Veterans Affairs. Strategic Analytics for Improvement and Learning (SAIL). https://www.va.gov/QUALITYOFCARE/measure-up/Strategic_Analytics_for_Improvement_and_Learning_SAIL.asp. Accessed November 16, 2019.
    4.
    Lee  JS, Nsa  W, Hausmann  LRM,  et al.  Quality of care for elderly patients hospitalized for pneumonia in the United States, 2006 to 2010.  JAMA Intern Med. 2014;174(11):1806-1814. doi:10.1001/jamainternmed.2014.4501PubMedGoogle ScholarCrossref
    5.
    Chen  J, Normand  S-LT, Wang  Y, Krumholz  HM.  National and regional trends in heart failure hospitalization and mortality rates for Medicare beneficiaries, 1998-2008.  JAMA. 2011;306(15):1669-1678. doi:10.1001/jama.2011.1474PubMedGoogle ScholarCrossref
    6.
    Kaboli  PJ, Go  JT, Hockenberry  J,  et al.  Associations between reduced hospital length of stay and 30-day readmission rate and mortality: 14-year experience in 129 Veterans Affairs hospitals.  Ann Intern Med. 2012;157(12):837-845. doi:10.7326/0003-4819-157-12-201212180-00003PubMedGoogle ScholarCrossref
    7.
    Trivedi  AN, Grebla  RC.  Quality and equity of care in the veterans affairs health-care system and in medicare advantage health plans.  Med Care. 2011;49(6):560-568. doi:10.1097/MLR.0b013e31820fb0f6PubMedGoogle ScholarCrossref
    8.
    Shah  RU, Tsai  V, Klein  L, Heidenreich  PA.  Characteristics and outcomes of very elderly patients after first hospitalization for heart failure.  Circ Heart Fail. 2011;4(3):301-307. doi:10.1161/CIRCHEARTFAILURE.110.959114PubMedGoogle ScholarCrossref
    9.
    Heidenreich  PA, Sahay  A, Kapoor  JR, Pham  MX, Massie  B.  Divergent trends in survival and readmission following a hospitalization for heart failure in the Veterans Affairs health care system 2002 to 2006.  J Am Coll Cardiol. 2010;56(5):362-368. doi:10.1016/j.jacc.2010.02.053PubMedGoogle ScholarCrossref
    10.
    Lindenauer  PK, Lagu  T, Shieh  MS, Pekow  PS, Rothberg  MB.  Association of diagnostic coding with trends in hospitalizations and mortality of patients with pneumonia, 2003-2009.  JAMA. 2012;307(13):1405-1413. doi:10.1001/jama.2012.384PubMedGoogle ScholarCrossref
    11.
    Ibrahim  AM, Dimick  JB, Sinha  SS, Hollingsworth  JM, Nuliyalu  U, Ryan  AM.  Association of coded severity with readmission reduction after the Hospital Readmissions Reduction Program.  JAMA Intern Med. 2018;178(2):290-292. doi:10.1001/jamainternmed.2017.6148PubMedGoogle ScholarCrossref
    12.
    Sukul  D, Hoffman  GJ, Nuliyalu  U,  et al.  Association between Medicare policy reforms and changes in hospitalized Medicare beneficiaries’ severity of illness.  JAMA Netw Open. 2019;2(5):e193290. doi:10.1001/jamanetworkopen.2019.3290PubMedGoogle Scholar
    13.
    Shahian  DM, Silverstein  T, Lovett  AF, Wolf  RE, Normand  S-LT.  Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards.  Circulation. 2007;115(12):1518-1527. doi:10.1161/CIRCULATIONAHA.106.633008PubMedGoogle ScholarCrossref
    14.
    Krumholz  HM, Wang  Y, Mattera  JA,  et al.  An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure.  Circulation. 2006;113(13):1693-1701. doi:10.1161/CIRCULATIONAHA.105.611194PubMedGoogle ScholarCrossref
    15.
    Fonarow  GC, Pan  W, Saver  JL,  et al.  Comparison of 30-day mortality models for profiling hospital performance in acute ischemic stroke with vs without adjustment for stroke severity.  JAMA. 2012;308(3):257-264. doi:10.1001/jama.2012.7870PubMedGoogle ScholarCrossref
    16.
    Iezzoni  L. Risk Adjustment for Measuring Healthcare Outcomes, Fourth Edition. Health Administration Press; Chicago, IL. 2012.
    17.
    Volpp  KG, Rosen  AK, Rosenbaum  PR,  et al.  Mortality among patients in VA hospitals in the first 2 years following ACGME resident duty hour reform.  JAMA. 2007;298(9):984-992.PubMedGoogle ScholarCrossref
    18.
    Ross  JS, Maynard  C, Krumholz  HM,  et al.  Use of administrative claims models to assess 30-day mortality among Veterans Health Administration hospitals.  Med Care. 2010;48(7):652-658. doi:10.1097/MLR.0b013e3181dbe35dPubMedGoogle ScholarCrossref
    19.
    Sohn  M-W, Arnold  N, Maynard  C, Hynes  DM.  Accuracy and completeness of mortality data in the Department of Veterans Affairs.  Popul Health Metr. 2006;4:2. doi:10.1186/1478-7954-4-2PubMedGoogle ScholarCrossref
    20.
    Fine  MJ, Auble  TE, Yealy  DM,  et al.  A prediction rule to identify low-risk patients with community-acquired pneumonia.  N Engl J Med. 1997;336(4):243-250. doi:10.1056/NEJM199701233360402PubMedGoogle ScholarCrossref
    21.
    Rahimi  K, Bennett  D, Conrad  N,  et al.  Risk prediction in patients with heart failure: a systematic review and analysis.  JACC Heart Fail. 2014;2(5):440-446. doi:10.1016/j.jchf.2014.04.008PubMedGoogle ScholarCrossref
    22.
    Garvin  JH, DuVall  SL, South  BR,  et al.  Automated extraction of ejection fraction for quality measurement using regular expressions in unstructured information management architecture (UIMA) for heart failure.  J Am Med Inform Assoc. 2012;19(5):859-866. doi:10.1136/amiajnl-2011-000535PubMedGoogle ScholarCrossref
    23.
    Rubin  DB, ed.  Multiple Imputation for Nonresponse in Surveys. Wiley; Hoboken, New Jersey. 1987. doi:10.1002/9780470316696
    24.
    Kind  AJH, Buckingham  WR.  Making neighborhood-disadvantage metrics accessible - the Neighborhood Atlas.  N Engl J Med. 2018;378(26):2456-2458. doi:10.1056/NEJMp1802313PubMedGoogle ScholarCrossref
    25.
    Kind  AJH, Jencks  S, Brock  J,  et al.  Neighborhood socioeconomic disadvantage and 30-day rehospitalization: a retrospective cohort study.  Ann Intern Med. 2014;161(11):765-774. doi:10.7326/M13-2946PubMedGoogle ScholarCrossref
    26.
    Singh  GK.  Area deprivation and widening inequalities in US mortality, 1969-1998.  Am J Public Health. 2003;93(7):1137-1143. doi:10.2105/AJPH.93.7.1137PubMedGoogle ScholarCrossref
    27.
    Scott  SL, Varian  HR.  Predicting the present with Bayesian structural time series.  IJMMNO. 2014;5(1/2):4. doi:10.1504/IJMMNO.2014.059942Google ScholarCrossref
    28.
    Sjoding  MW, Iwashyna  TJ, Dimick  JB, Cooke  CR.  Gaming hospital-level pneumonia 30-day mortality and readmission measures by legitimate changes to diagnostic coding.  Crit Care Med. 2015;43(5):989-995. doi:10.1097/CCM.0000000000000862PubMedGoogle ScholarCrossref
    29.
    Trivedi  AN, Nsa  W, Hausmann  LRM,  et al.  Quality and equity of care in U.S. hospitals.  N Engl J Med. 2014;371(24):2298-2308. doi:10.1056/NEJMsa1405003PubMedGoogle ScholarCrossref
    30.
    Ody  C, Msall  L, Dafny  LS, Grabowski  DC, Cutler  DM.  Decreases in readmissions credited to Medicare’s program to reduce hospital readmissions have been overstated.  Health Aff (Millwood). 2019;38(1):36-43. doi:10.1377/hlthaff.2018.05178PubMedGoogle ScholarCrossref
    31.
    Radomski  TR, Fine  MJ, Gellad  WF.  Outcome after admission at Veterans Affairs vs non-Veterans Affairs hospitals.  JAMA. 2016;316(3):345-346. doi:10.1001/jama.2016.5391PubMedGoogle ScholarCrossref
    ×