Likelihood of receiving indicated care for 10 National Quality Forum measures of human immunodeficiency virus (HIV)/AIDS care based on patient and medical resource factors. For each measure, patients without a CD4+ lymphocyte count were included in the multivariate analysis, but their data are not displayed. A, Medical visit measure (147 patients without a CD4+ lymphocyte count). B, Hepatitis B virus (HBV) screening (63 patients without a CD4+ lymphocyte count). C, Hepatitis C virus (HCV) screening (63 patients without a CD4+ lymphocyte count). D, HBV vaccination (62 patients without a CD4+ lymphocyte count). E, Tuberculosis (TB) screening (63 patients without a CD4+ lymphocyte count). F, Syphilis screening (63 patients without a CD4+ lymphocyte count. G, CD4+ lymphocyte count (63 patients without a CD4+ lymphocyte count). H, Potent antiretroviral therapy (ART) (11 patients without a CD4+ lymphocyte count). I, HIV RNA control (33 patients without a CD4+ lymphocyte count). J, Pneumocystis pneumonia (PCP) prophylaxis (no patients without a CD4+ lymphocyte count were included). The reference groups for each variable for panels A to I are sex (male), age (50-59 years), race/ethnicity (white), mental health diagnosis (no), hard drug use (no), nadir CD4+ lymphocyte count (≥350 cells/μL, primary care/infectious diseases (PC/ID) outpatient visits (2-6 visits), duration of Veterans Affairs (VA) HIV care (0-2 years), facility HIV caseload (≥300 patients), region (South), and local Clinical Case Registry (CCR) use (<100 reports/y). For panel J, the references are the same with the exception of nadir CD4+ lymphocyte count (50-199 cells/μL). To convert lymphocytes to cells × 109 per liter, multiply by 0.001.
Comparison of observed and expected binomial distribution of number of times individual large and very large facilities performed in top quarter of 10 National Quality Forum measures.
Backus LI, Boothroyd DB, Phillips BR, Belperio PS, Halloran JP, Valdiserri RO, Mole LA. National Quality Forum Performance Measures for HIV/AIDS CareThe Department of Veterans Affairs' Experience. Arch Intern Med. 2010;170(14):1239–1246. doi:10.1001/archinternmed.2010.234
Information technology promises to improve health care through reporting of standardized quality-of-care measures. In 2008, the National Quality Forum (NQF) first endorsed performance measures for human immunodeficiency virus (HIV)/AIDS care. Little is known about performance on these measures in routine medical practice. We assessed performance using available electronic data for the large, diverse population with HIV in the Department of Veterans Affairs (VA) and evaluated the influence of patient and resource factors.
In a retrospective analysis of observational data for 21 564 patients with HIV receiving VA medical care in 2008, we determined performance rates for 10 NQF measures for HIV/AIDS care for the VA nationwide and for 73 facilities with caseloads of 100 or more patients with HIV.
National rates for 6 measures were greater than 80%; the remaining measures and their rates were as follows: annual syphilis screening (54%), tuberculosis screening (65%), Pneumocystis pneumonia prophylaxis (72%), and HIV RNA control (73%). For all measures, rates varied across facilities. In multivariate logistic regression models, African Americans and hard drug users were less likely to access care and less likely to receive HIV-specific care but more likely to receive indicated general medical care. Resource factors (number of primary care/infectious disease outpatient visits, duration of care, and larger facility caseload) were associated with increased likelihood of receipt of indicated general and HIV-specific care.
National performance rates were generally high, but variation in rates across facilities revealed room for improvement. Both patient and resource factors had an impact on the likelihood of receipt of indicated care.
Information technology promises to improve health care quality through population-level reporting with continuous quality improvement.1 The US Department of Veterans Affairs (VA) has developed and refined a sophisticated electronic medical record (EMR) that includes both patient-level and population-level tools.2 Over the past 5 years, the VA implemented a population-level registry system, the Clinical Case Registry (CCR), which uses the EMR to identify cohorts of veterans with targeted conditions.3 The CCR allows clinicians to generate reports for their local population of patients with human immunodefiency virus (HIV) on a variety of topics. A national CCR with data from all VA facilities supports assessment using standardized quality measures.
The National Quality Forum (NQF) is a not-for-profit, public benefit corporation established to develop and implement a national strategy for health care quality measurement and reporting.4 In July 2008, the NQF endorsed 13 measures for evaluating HIV care quality.5 The measures are a subset of those developed by the National Committee for Quality Assurance, the American Medical Association–Physician Consortium for Performance Improvement, the Infectious Disease Society of America/HIV Medicine Association, and the Health Resources and Services Administration.6 Little is known about performance on these measures in routine medical practice.
The VA cares for over 20 000 veterans with HIV infection annually.7 This large population and the national CCR provided a unique opportunity to use NQF measures to assess performance and to evaluate the influence of patient and medical resource factors.
We used the national CCR, which includes information on diagnosis codes (International Classification of Diseases, Ninth Revision [ICD-9]), laboratory results, medications, and outpatient visits and procedures (Current Procedure Terminology [CPT]).3 Data were available from at least 1996. To be included in the cohort, a veteran had to be in care at any VA medical facility in 2008 and have laboratory confirmation of HIV infection in the CCR. Because the medical visit measure, which is a minimum requirement for all NQF measures, requires patients to have at least 1 outpatient visit in each half of the year, we excluded patients who died in the first half of the year or who did not enter VA HIV care until the second half of the year.
For each patient, we used CCR data to determine whether he or she was in the eligible population and whether the NQF measure criteria had been met. Details regarding CCR data definitions for each measure are available from the authors. Three measures could not be reliably addressed with CCR data and were excluded: (1) screening for high-risk behavior, (2) screening for injection drug use that was not captured as ICD-9 or CPT codes, and (3) Chlamydia and gonorrhea screening for which wide local variation in reporting format precludes consistent national data extraction.
Patient demographic variables included age, sex, and race/ethnicity. Data on race/ethnicity are entered at VA registration, with options of American Indian or Alaska Native, Asian, black or African American, Hispanic, Native Hawaiian or other Pacific Islander, white, declined to answer, and unknown. A history of a mental health diagnosis (anxiety disorder, bipolar disorder, major depression, posttraumatic stress disorder, or schizophrenia) or hard drug use (opiates, cocaine, or amphetamines) was identified if the patient had 2 outpatient instances of a pertinent ICD-9 code. Nadir CD4+ lymphocyte count was the lowest VA value up to and including the first value in 2008.
Patients seen at multiple VA facilities in 2008 were assigned to the facility where each patient had the most primary care/infectious disease (PC/ID) outpatient visits. Duration of VA HIV care was based on the earliest CCR date of an HIV ICD-9 code. The facility HIV caseload in 2008 was a priori categorized as less than 25 patients (small), 25-99 (medium), 100-299 (large), and 300 or more (very large). The geographic region was categorized as Northeast, South, Midwest, and West.8 Facilities that generated at least 100 local CCR reports in 2008 were categorized as substantial local CCR users.
Multivariate logistic regression analyses were used to examine factors associated with patient receipt of indicated care. A priori we hypothesized that receipt of indicated care would vary based on patient factors (age, sex, race/ethnicity, mental health diagnosis, hard drug use, and nadir CD4+ lymphocyte count) and resource factors (number of PC/ID outpatient visits, duration of VA HIV care, facility caseload, region, and local CCR use).9 Accordingly, these variables were included in the multivariate models.
Among the 73 facilities with an HIV caseload of 100 to 299 (large and very large facilities), we examined the degree to which high facility performance on 1 NQF measure was related to high performance on the other 9 measures. We excluded facilities with small HIV caseloads to ensure robust performance determination. For each measure we identified “high performers” as the top 18 of these 73 sites (roughly one-quarter). We compared the distribution of the total number of times a facility was a high performer across the 10 measures with a binomial distribution for 10 independent trials with a probability of success (being a high performer) of 18 of 73 using a χ2 test. If facilities' performances are correlated across measures, we would expect more facilities at the extremes than predicted by the binomial distribution. If the 2 distributions are similar, however, this suggests that a facility's performance on 1 measure is independent of its performance on others. Data were analyzed using SAS statistical software (version 9.1; SAS Institute Inc, Cary, North Carolina) and R software (version 2.8.0; R Foundation for Statistical Computing, Vienna, Austria).
The protocol was approved by the Stanford University institutional review board and the VA Palo Alto Health Care System research and development committee.
We identified 22 411 veterans in the CCR in VA care in 2008, with VA laboratory confirmation of HIV infection. We excluded 370 patients who died in the first half of the year and 477 patients who did not enter VA HIV care until the second half of the year. The final cohort of 21 564 was overwhelmingly male (97%) but included 569 women (Table 1). The median age was 52.5 years (interquartile range [IQR], 46.3-58.8 years). Nearly half of the cohort was African American. The cohort had substantial rates of mental health diagnoses (54%) and a history of advanced HIV (median nadir CD4+ lymphocyte count, 193 cells/μL [IQR, 70-333]) (to convert lymphocytes to cells × 109 per liter, multiply by 0.001). The patients had a median of 13 outpatient visits in 2008 (IQR, 7-25 visits) with 7 visits to PC/ID clinics (IQR, 4-11 visits) and a median duration in VA HIV care of 8.6 years (IQR, 4.7-12.2 years). The cohort was concentrated in the South and received care predominantly at facilities with an HIV caseload of 300 patients or more.
National rates for the 10 NQF measures we considered appear in Table 2 with minimum and maximum rates for facilities with large HIV caseloads. Of the cohort, 17 904 (83%) met the medical visit measure criteria. National rates for 6 measures were greater than 80%. The 4 measures with lower rates were annual syphilis screening (55%), tuberculosis (TB) screening (67%), Pneumocystis pneumonia (PCP) prophylaxis (72%), and HIV RNA control (73%). Among the 73 facilities with large/very large HIV caseloads, these 4 measures also had the lowest minimum and maximum facility rates and the largest differences between the minimum and maximum rates.
Among the 17 904 patients eligible for the annual syphilis screening measure in 2008, 16 827 (94%) had syphilis screening at some time during their VA HIV care; those eligible received a VA syphilis test a median of every 1.9 years with a median of 4 tests while in VA HIV care (IQR, 2-7).
While 72% of the 2709 patients eligible for PCP prophylaxis received indicated medications within 90 days of their CD4+ lymphocyte count of less than 200 cells/μL, 89% of them received PCP prophylaxis in 2008.
At most, 24 patients eligible for the HIV RNA control measure received care at facilities that performed only HIV RNA tests with a lower limit of detection of 400 copies/mL (vs 10-75 copies/mL). Defining HIV RNA control as less than 400 copies/mL, the national rate increased to 84% and the minimum and maximum large/very large facility rates increased to 62% and 92%, respectively.
Performance rates for each measure for groups defined by patient and resource factors are available by request. The small number of patients with no available CD4+ lymphocyte count data have strikingly lower performance rates than all other groups and likely represent a group that receives HIV care outside of the VA or not at all. Aside from this group, the largest differences in performance rates occurred among groups defined by the number of PC/ID outpatient visits, duration of VA HIV care, and the facility's HIV caseload. For example, the performance rate on the syphilis measure ranged from 13% to 59% depending on the facility's HIV caseload.
In regression analysis, we found that women, patients with a mental health diagnosis, and patients with nadir CD4+ lymphocyte counts lower than 350 cells/μL were significantly more likely to access care as defined by the medical visit criterion (Figure 1A) (see eTable 1 for P values; http://www.archinternmed.com). Compared with patients in their fifties, younger patients were less likely to access care and older patients more likely. African Americans, hard drug users, and patients seen at facilities with smaller HIV caseloads and at facilities in the Northeast were also significantly less likely to access care.
For 5 general medical care measures (HBV screening, HCV screening, HBV vaccination, TB screening, and syphilis screening), the impact of patient factors varied depending on the measure (Figure 1B-F). Only a few effects were consistent: African Americans and hard drug users were independently significantly more likely to receive the indicated care for all 5 measures (eTable 1).
Resource factors had marked and more consistent effects. For all 5 general medical care measures, the likelihood of receiving indicated care increased as the number of PC/ID visits increased. This likelihood also increased with duration of VA HIV care for the 4 measures of 1-time processes of care (HBV screening, HCV screening, HBV vaccination, and TB screening), whereas the likelihood of receiving annual syphilis screening—the only re peated process of care measure—decreased as the duration of VA HIV care increased. The likelihood of receiving indicated care increased with the facility's HIV caseload. Patients at facilities with substantial local CCR use were more likely to receive indicated care for 3 of 5 measures.
For the 4 measures concerning HIV-specific care (CD4+ lymphocyte count, potent antiretroviral therapy [ART], HIV RNA control, and PCP prophylaxis), patient factors generally had inconsistent effects (Figure 1G-J) (eTable 2). Exceptions were that African Americans and hard drug users were less likely to have recent CD4+ lymphocyte counts, less likely to receive ART, and less likely to have HIV RNA control. Patients with lower nadir CD4+ lymphocyte counts were more likely to have recent CD4+ lymphocyte counts and to receive ART but were less likely to have HIV RNA control. Lower nadir CD4+ lymphocyte count also increased the likelihood of PCP prophylaxis.
Some resource factors affected the likelihood of receiving indicated HIV-specific care. For the CD4+ lymphocyte count and potent ART measures, the likelihood of receiving indicated care increased with the number of PC/ID visits. For the potent ART and HIV RNA control measures, this likelihood increased with the duration of care. Patients at facilities with smaller HIV caseloads were less likely to have recent CD4+ lymphocyte counts and HIV RNA control. Patients at facilities with substantial local CCR use were more likely to receive indicated ART.
Limiting the identification of mental health diagnosis or hard drug use to codes in 2007 and 2008 changed the multivariate results only minimally. Patients with a recent mental health diagnosis were not more likely to receive HCV screening and were less likely to have HIV RNA control. The pattern of odds ratios for patients with recent hard drug use did not change.
The total number of times that a facility with a large/very large HIV caseload was a high performer across the 10 measures ranged from none (never in the top quarter) to 9. Figure 2 shows the actual and binomial distributions of high performers. The 2 distributions are significantly different (P < .001); the actual distribution shows more facilities at the extremes—high performance on no measure and high performance on 5 or more measures—than predicted by the binomial distribution.
To our knowledge, this is the first report of performance rates for a national health care system on most of the NQF measures for HIV/AIDS care. The performance in this large and geographically diverse population of patients with HIV provides guidance on expected rates and goal rates. Overall, the VA rates were high—over 80% on 6 of 10 measures. The maximum rates among facilities with large HIV caseloads, which were greater than 90% for all 10 measures, demonstrate that extremely high performance can be achieved in routine medical practice. The variation in rates across facilities demonstrates that these measures can identify facilities where quality initiatives might improve HIV care and facilities that may provide models for such initiatives.
The national VA rates are generally higher than those reported on individual measures from routine clinical practice. For example, the HIV Outpatient Study reported that 77% of 2662 patients in care in 2007 had HCV screening.10 The Evaluation of Quality Improvement for HIV care (EQHIV) collaborative of 69 clinics with 5035 patients in care in 2001 reported rates for HCV screening of 90%, TB screening of 52%, ART of 79%, HIV RNA control (defined as <400) of 51% and PCP prophylaxis (with no specified initiation time requirement) of 74%.11
The low VA performance rates on 4 measures (syphilis screening, TB screening, PCP prophylaxis, and HIV RNA control) likely reflect 4 different issues: (1) clinician determination about the applicability of indicated care, (2) incomplete data capture, (3) stringent new time requirements, and (4) medical treatment limitations.
The syphilis screening measure requires annual syphilis testing regardless of sexual activity. Our results indicate that VA health care providers (eg, physicians) recognize that syphilis is an issue in the population of patients with HIV (94% of eligible patients had a syphilis test during VA HIV care with repeated tests a median of every 1.9 years); however, the annual syphilis screening rate was only 55%. In practice, some patients and their health care providers may forego annual syphilis screening owing to a perceived lack of high-risk sexual activity. We cannot validate this assessment because sexual activity is not systematically recorded in the EMR. Multivariate models indicate that groups at increased epidemiologic risk for syphilis (eg, men, African Americans, and younger patients)12,13 were more likely to be screened. Because sexual risk behavior likely is not related to facility HIV caseload, however, our finding that screening rates declined as facility HIV caseload declined suggests some underperformance driven by factors other than appropriate risk assessment. The large difference in the minimum and maximum large/very large facility rates (8%-97%) also strongly suggests room for improvement.
The low rate for TB screening may reflect both problems with CCR data capture from the EMR and underperformance. Tuberculosis screening was identified from 10 different EMR data elements included in the CCR. Such variation suggests that TB screening is recorded in additional data fields not captured in the CCR. Because of this variation in documentation, there is no local CCR report on TB screening, which may explain why TB screening is one measure on which substantial local CCR use did not predict improved performance.
The low rate for PCP prophylaxis likely reflects the measure's relatively stringent time criterion for the initiation of prophylaxis. Only 72% of patients with an indication received PCP prophylaxis within the criterion of 90 days, but 89% did so within the year. Such a stringent time criterion is not included in the current Centers for Disease Control and Prevention guidelines, which state that patients with a CD4+ lymphocyte count of less than 200 cells/μL should receive PCP prophylaxis but include no time requirement for initiation.14
While all aspects of care addressed in the NQF measures are important, the HIV RNA control measure probably is the most important for reducing patient morbidity and mortality from HIV/AIDS and is the only outcome measure.15- 17 It is difficult to set a target rate for HIV RNA control and thus to assess the VA rate of 73%. In practice, this rate is constrained by the limitations of available medical therapy. Health care providers cannot prevent unacceptable adverse effects that require discontinuation of a medication, nor can they directly alter the ability of the virus to develop resistance. Shared toxicity and resistance patterns among antiretrovirals mean that fewer combinations are available after the first, second, or third regimen. Seventy-two percent of our cohort had been in VA HIV care for at least 5 years, likely indicating a treatment-experienced population. Other limitations, such as suboptimal adherence and perseverance, are only partially susceptible to provider remediation. A recent review18 of adherence in routine clinical practice pooled 31 North American studies with 17 573 patients and found an adherence rate of only 55%.
Comparing the VA HIV RNA control rate with published reports is also problematic owing to differences in population and methods. Most published HIV RNA control rates are from research studies that impose eligibility restrictions to ensure a more homogeneous population—usually antiretroviral naïve—than that encountered in routine medical care. Also, these research studies usually assess HIV RNA suppression rates at a set interval after the start of ART (eg, 6 months,19 48 weeks,20 or 1 year21) or assess the time to HIV RNA suppression.22,23 In contrast, the NQF measures are cross-sectional and assess all patients in care during a specified time period. The maximum large/very large facility rate of 91% of patients with HIV with RNA levels below 10 to 75 copies/mL (92% < 400 copies/mL) may be the highest achievable for a group of patients in routine medical practice concurrently including both ART-naïve and ART-experienced patients. Nonetheless, the range of rates across large facilities (28%-91%) indicates the need to better understand why HIV RNA control is not achieved and to develop interventions to address this issue.
The differences in performance rates across categories of patient and resource factors and the results from the multivariate models for the 8 process NQF measures have implications for quality improvement. African Americans and hard drug users consistently had a higher likelihood of receipt of indicated general medical care but a lower likelihood of receipt of indicated HIV-specific care. Given the large sample size, however, these statistically significant differences may not reflect clinically significant differences. For example, in multivariate models, African Americans had an odds ratio of 0.62 for receipt of indicated ART compared with white individuals; however, the absolute rates ranged only from 89% to 93%, respectively. The largest absolute differences in rates generally occurred across groups defined by resource factors rather than patient factors. In multivariate models, resource factors had more consistent effects across multiple measures than did patient factors.
The difference between the actual and binomial distribution of high-performing large VA facilities indicates that a facility's performance on 1 measure is likely related to its performance on other measures. Our results are in contrast to those of the EQHIV collaborative,24 which found no more high-performing facilities than were predicted by chance. Quality improvement efforts can work to identify practices at these apparently high-performing facilities that could be adopted by other facilities.
Substantial local CCR use generally predicted an increased likelihood of receipt of the indicated care for most measures. The local population reports may have led to improved care on those measures addressed by local CCR reports. Alternatively, CCR use may identify facilities already invested in local population management that would have outperformed other facilities even without the CCR reports. However, it seems unlikely that health care providers would spend time to run CCR reports if they did not feel that the reports benefited their practice.
There are several limitations to the present analysis. First, we were unable to assess 3 of the 13 NQF measures from available data fields in the CCR. Second, several of the data elements that we used to identify TB screening indicate that a skin test was placed but not whether the patient returned for test reading. Third, while a hard drug diagnosis may provide a marker for the HIV risk factor of intravenous drug use, we do not have information about other HIV risk factors, particularly sexual practices, which may have an impact on receipt of indicated care. Fourth, we were unable to account for individual provider characteristics (eg, specialty training and HIV experience). Finally, actual performance rates may be even higher than those we reported. Indicated care that is documented only in the text of a note is not captured in the CCR. Moreover, we have no means to measure the extent veterans use non-VA sources for health care.
In summary, national VA rates for many of the NQF measures for HIV/AIDS care are generally high. However, on all measures there is facility-level variation even within groups defined by HIV caseload, which indicates room for improvement. There also were sites that performed well on multiple measures. The ability to measure performance should aid quality managers, health care providers, and administrators in identifying best practices from high-performing facilities and in assisting low-performing facilities to improve care. Following quality initiatives, another analysis of NQF measures can be undertaken to assess improvement.
Correspondence: Lisa I. Backus, MD, PhD, Center for Quality Management in Public Health, Veterans Affairs Palo Alto Health Care System, 3801 Miranda Ave, Mail Code 132, Palo Alto, CA 94304 (firstname.lastname@example.org).
Accepted for Publication: December 23, 2009.
Author Contributions: Dr Backus had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Backus, Phillips, Belperio, Halloran, and Mole. Acquisition of data: Mole. Analysis and interpretation of data: Backus, Boothroyd, Phillips, Belperio, Halloran, Valdiserri, and Mole. Drafting of the manuscript: Backus. Critical revision of the manuscript for important intellectual content: Boothroyd, Phillips, Belperio, Halloran, Valdiserri, and Mole. Statistical analysis: Boothroyd. Administrative, technical, and material support: Valdiserri and Mole. Study supervision: Valdiserri and Mole.
Financial Disclosure: None reported.
Additional Contributions: We thank the local CCR coordinators who make this information possible.