Measuring Underuse of Necessary Care Among Elderly Medicare Beneficiaries Using Inpatient and Outpatient Claims

Context Continuing changes in the health care delivery system make it essential to monitor underuse of needed care, even for relatively well-insured populations. Traditional approaches to measuring underuse have relied on patient surveys and chart reviews, which are expensive, or simple single-condition claims-based indicators, which are not clinically convincing.

Objective To develop a comprehensive, low-cost system for measuring underuse of necessary care among elderly patients using inpatient and outpatient Medicare claims.

Design A 7-member, multispecialty expert physician panel was assembled and used a modified Delphi method to develop clinically detailed underuse indicators likely to be associated with avoidable poor outcomes for 15 common acute and chronic medical and surgical conditions. An automated system was developed to calculate the indicators using administrative data.

Setting and Subjects A total of 345,253 randomly selected elderly US Medicare beneficiaries in 1994-1996.

Main Outcome Measures Proportion of beneficiaries receiving care, stratified by indicators of necessary care (n = 40, including 3 for preventive care), and avoidable outcomes (n = 6).

Results For 16 of 40 necessary care indicators (including preventive care indicators), beneficiaries received the indicated care less than two thirds of the time. Of all indicators, African Americans scored significantly worse than whites on 16 and better on 2; residents of poverty areas scored significantly lower than nonresidents on 17 and higher on 1; residents of federally defined Health Professional Shortage Areas scored significantly lower than nonresidents on 16 and higher on none (P<.05 for all).

Conclusions This claims-based method detected substantial underuse problems likely to result in negative outcomes in elderly populations. Significantly more underuse problems were detected in populations known to receive less-than-average medical care. The method can serve as a reliable, valid tool for monitoring trends in underuse of needed care for older patients and for comparing care across health care plans and geographic areas based on claims data.

Monitoring access to needed medical care has become increasingly important in today's rapidly changing medical marketplace. Traditionally, measurement efforts have focused on vulnerable (eg, poor, uninsured) populations, who have a higher risk of being sick and whose access to care is below average. Numerous studies have shown that these vulnerable populations underuse needed services. They have a lower likelihood of seeing a physician, higher emergency department use, a greater likelihood of delaying care, and lower use of preventive care services.^1-4 Not surprisingly, these populations also have poorer health outcomes and higher mortality rates.^4-6

The recent push for cost containment in health care has generated additional interest in the underuse issue beyond the traditional concern for vulnerable populations.⁷ Access to and quality of care have become important issues in assessing all health insurance programs, including Medicare, which has been the focus of a federally mandated tracking effort since 1989.^4,8 In addition, managed care, fee restraints, utilization review, and other cost containment mechanisms have introduced the possibility that even fairly mainstream insured populations may encounter barriers to use of needed services.

Measuring underuse of needed care has proven to be a difficult methodological problem. Earlier studies often focused on use of services, such as the emergency department, but could not determine whether patients who underused these services actually received less evidence-based clinical care, as measured by specific criteria for indicated clinical processes. Criticism of the relatively few assessments of underuse that were based on specific criteria centered on 4 concerns. First, sample identification often depended on the very use that was being evaluated. Second, the survey and chart review data techniques used were expensive to collect and, thus, difficult to apply on a routine basis, although targeted techniques were successful in reducing costs in some studies.⁹ Third, most methods have focused on only a few medical conditions, even though development of a system that encompasses multiple medical conditions and both inpatient and outpatient care is essential for broad-based quality monitoring.^10-13 Fourth, these methods were rarely validated in terms of their ability to detect differences in care across patient populations.

Several recent efforts to develop and evaluate clinical performance indicators of quality have attempted to address these concerns. The Health Care Cost and Utilization Project evaluated a system of 33 mostly surgical indicators of avoidable complications and outcomes based on hospital discharge data.¹³ As part of its Comparing Hospital Performance Indicators demonstration project,¹⁴ the Joint Commission on Accreditation of Healthcare Organizations evaluated medical record–based quality measurement systems for use by its accredited organizations. The Health Care Financing Administration (HCFA) has developed quality indicators to monitor Medicare quality of care for several conditions, but only 2 of them (breast cancer and diabetes) rely solely on administrative data.^15,16 Likewise, for a project that compares Medicare risk plans and the Medicare fee-for-service sector using performance measures from the Health Plan Employer Data Information Set (HEDIS) 3.0, HCFA developed only 3 measures based solely on administrative data (diabetic retinal eye examinations, mental health hospitalization follow-up care, and mammography).¹⁷

A number of studies have evaluated rates of procedure use across populations but, with few exceptions, have done so without applying precise indicators of need.^4,18-20 Professional societies, such as the American Medical Association, and private evaluation and accreditation bodies, such as the National Committee for Quality Assurance, have also developed multicondition performance measurement systems that are based on chart and administrative data and include underuse evaluations, but these have not been fully evaluated for validity based on expert judgment of criteria or ability to distinguish levels of underuse among patient populations.²¹

In an effort to monitor underuse among Medicare beneficiaries, we sought to build on prior work to develop and validate an inexpensive measurement system that relies solely on administrative data. We wanted the system to scan a large enough set of important conditions that it would be useful for identifying target areas for quality improvement within a health care delivery system. In addition, we intended to develop a method that would eventually apply to the entire Medicare population for both outpatient and inpatient care. While outpatient data for those who are enrolled in Medicare managed care plans will begin to be collected in 2000-2001, outpatient data on other Medicare beneficiaries is currently available. We used rigorous expert panel methods to ensure that the measure was clinically valid and evidence based. We further validated the measure's ability to detect clinically and statistically significant differences by applying it to populations expected to have problems with underuse based on prior studies, such as those in underserved areas, and comparing results for these areas to those with greater medical resources. If successful, development of a clinically valid, comprehensive, inpatient and outpatient claims–based measurement tool capable of detecting differences in underuse would provide health care organizations with the high-yield, low-cost screening tool they need.²²

Using information from the published literature and expert opinion, we developed clinical indicators of underuse for the elderly Medicare population and applied them to administrative data. We tested and validated the system on Medicare claims from 1992-1993 and applied it to claims from 1994-1996. In defining the indicators, we incorporated clinical performance standards related to necessary care. Necessary care denotes care for which (1) the benefits of the care outweigh the risks (ie, the care is appropriate), (2) the benefits to the patient are likely and substantial, and (3) physicians have judged that not recommending the care would be improper. We asked the expert panel to identify services that met this minimum quality standard for the average patient visiting the average physician.²³

This research entailed selecting medical conditions, developing clinical indicators for each condition, using an expert panel to rate the indicators, selecting a final set of indicators, developing computer algorithms to calculate the indicators from administrative data, applying the indicators to Medicare claims data, and evaluating the indicators for vulnerable populations.

We started with several lists of conditions thought to be amenable to quality improvement efforts for elderly and other populations.^24-26 We narrowed this list by selecting conditions (1) with a high prevalence or incidence among the elderly population, (2) for which effective medical treatment is available, and (3) that are identifiable from diagnoses coded on claims data.

We developed 2 types of indicators: those reflecting minimum standards of acceptable care (necessary care indicators) and those representing potentially avoidable outcomes (avoidable outcome indicators). Necessary care indicators assess whether patients with a specified condition receive certain procedures. For example, patients with diabetes should have an annual eye examination. Avoidable outcome indicators are diagnoses that should appear less frequently in the claims records of patients who have adequate access to needed care, and, thus, reflect antecedent underuse. An example is a diagnosis of ruptured appendix.

We based the necessary care indicators on both inpatient and outpatient care. For each condition, we attempted to identify indicators for each stage of care: initial evaluation, diagnostic tests, therapeutic interventions, hospitalization follow-up, monitoring of routine care, and avoidable outcomes. We developed the initial set of proposed underuse indicators from available sources, including randomized controlled trials and meta-analyses of such trials (whenever possible), review articles, practice guidelines, observational studies, medical textbooks, consensus reports, opinions of individual experts, and our own judgment. For each indicator, project staff summarized the supporting evidence with references.²⁷ Randomized controlled trial data on the relationship between indicated care and outcomes were not available from the literature for all indicators. For a subset of the indicators (post–myocardial infarction visit, gastrointestinal tract workup for iron deficiency anemia, carotid imaging for carotid territory stroke, eye examination for diabetic patients, and mammogram for women aged 65-74 years), we were able to link the necessary care specified in the indicator to an improvement in outcomes using simple decision analysis trees anchoring panelists' ratings. For avoidable outcome indicators, we relied on previous work related to sentinel events,²⁸ ambulatory-sensitive conditions,²⁹ and related work.³⁰

We assembled a 7-member, multispecialty panel of physicians that convened twice to discuss and rate the indicators using a modified Delphi method. Nominated by relevant specialty societies, the panelists practiced family medicine, general internal medicine, geriatrics, cardiology, nephrology, endocrinology, and general surgery in academic, fee-for-service, and managed care settings. In addition to preventive care, the expert panel evaluated indicators for the following 15 conditions: acute myocardial infarction, anemia, angina, breast cancer, cerebrovascular accident, cholelithiasis, chronic obstructive pulmonary disease, congestive heart failure, depression, diabetes, gastrointestinal bleeding, hip fracture, hypertension, pneumonia, and transient ischemic attack.

Panel members rated the indicators 4 times before and during the 2 meetings. They rated necessary care indicators on 4 dimensions (outcome improvement, necessity, feasibility, and suitability) and avoidable outcome indicators on 3 dimensions (preventability, feasibility, and suitability). Panelists recorded each rating on a 9-point scale (for outcome improvement, 1 = unlikely and 9 = very likely; for necessity, 1 = clearly not necessary and 9 = clearly necessary; for feasibility, 1 = not feasible and 9 = definitely feasible; for suitability, 1 = not suitable and 9 = suitable; and for preventability, 1 = never preventable and 9 = always preventable). After each round of rating, median ratings summarizing the scores of the 7 panelists were calculated. After round 1, indicators with a median rating of 7 or more on the necessity (or preventability) scale and 4 or more on all other scales were retained. We chose 4 as the threshold because we planned to do feasibility testing and wanted to be inclusive. After rounds 2 and 4, indicators were retained based on a median suitability rating of 7 or higher with no statistical disagreement.²³

We proposed 136 indicators to the panelists: 105 necessary care indicators and 31 avoidable outcome indicators. Prior to the first meeting, panelists received summaries of the evidence supporting each indicator. Before the second meeting, we gave panel members numeric tables showing how the indicators performed on claims data for a 1% sample of Medicare beneficiaries. For each round of ratings, indicators were added or deleted based on the panelists' ratings: in round 1, 11 indicators were added; in round 2, 17 were added and 18 were deleted; in round 3, 1 was added and 83 were deleted; and in round 4, 3 were added and 31 were deleted.

Ultimately, almost two thirds of the initial set of indicators were deleted, resulting in a total of 47 indicators (41 necessary care indicators and 6 avoidable outcome indicators; Table 1). All of these indicators received a median suitability rating of 7 or higher without statistical disagreement among the panelists,²³ and none had a necessity, preventability, or feasibility rating of less than 6. Although the panelists' ratings varied (mean ratings among the 7 panelists ranged from 6.8 to 7.4), there was substantial agreement. As in previous panel projects, specialist panelists rated indicators in their area of specialty somewhat more highly³¹ but ultimately tended to agree with the overall final panel disposition (κ = 0.80). No indicator receiving a median suitability rating of 7 or higher was disqualified because of panel disagreement. One necessary care indicator, having a lipid profile in the first year after initial diagnosis of angina, was dropped during subsequent analysis because of implementation and programming difficulties, leaving 46 reported here. Of these, 5 were based on randomized controlled trial evidence, 7 on observational trials, and the remainder on expert opinion.

We developed an automated system for calculating the indicators from administrative data. The system for scoring the indicators uses inpatient and outpatient utilization data from Medicare part A and B claims. All Medicare beneficiaries younger than 65 years were excluded from the sample to ensure generalizability since all people aged 65 years or older are eligible for Medicare.

We analyzed data on a randomly selected 1% sample of Medicare beneficiaries aged 65 years or older from 2 periods: January 1, 1992, through December 31, 1993, and July 1, 1994, through June 30, 1996. We used the results of the 1992-1993 analysis to develop the computer algorithms, which we then applied to the 1994-1996 data reported here.

The 2 samples were restricted to beneficiaries aged 65 years or older who were enrolled in traditional fee-for-service Medicare for at least 1 month during 1 of the study periods. We excluded months during which beneficiaries were enrolled in a managed care program because these utilization data were not available for the study periods. Hospital inpatient, hospital outpatient, and physician claims for services incurred from July 1, 1994, through June 30, 1996, were obtained from Medicare Standard Analytic files. Beneficiary characteristics were merged from records in the denominator file. A total of 345,253 individuals met our inclusion criteria for at least 1 indicator.

We identified beneficiaries with the target medical conditions and treatments using diagnoses and procedures recorded on Medicare claims. We constructed algorithms for each indicator specifying the qualifying diagnosis (International Classification of Diseases, Ninth Revision [ICD-9] codes), the time frame required to observe whether care was provided, and the necessary care (ICD-9 and Current Procedural Terminology, Fourth Edition procedure codes).

For each indicator, we included only beneficiaries with the relevant diagnoses who were enrolled in Medicare fee-for-service for an adequate number of months to have received the necessary care. We then calculated the proportion of these beneficiaries who had received the necessary care, as well as the proportion of several vulnerable populations within this sample who had received the necessary care. For each indicator, the age-sex distribution of the entire eligible population was used for direct standardization of each vulnerable population's rate. We conducted 3 paired t tests for each indicator, comparing underuse for (1) African American and white beneficiaries; (2) beneficiaries residing in a federally defined Health Professional Shortage Area (HPSA)³² and those residing outside an HPSA; and (3) beneficiaries residing in a poverty ZIP code (in which more than 30% of the population lives under the federally defined poverty line) and those residing in a nonpoverty ZIP code. For indicators that focus on care following hospitalization, we excluded beneficiaries discharged to other hospital facilities or to home health agencies.

Table 1 shows results for the final 46 indicators for all beneficiaries based on the Medicare claims data analysis for 1994-1996. For each indicator, we present 2 types of data: the proportion of beneficiaries who received the indicated care and the number of beneficiaries who were eligible to receive the care (ie, a diagnosis or treatment code indicated that they had the condition). All beneficiaries were eligible for at least 1 preventive care indicator and 45% of beneficiaries were eligible for at least 1 nonpreventive care indicator.

The results for all beneficiaries show that the proportion who received necessary care varied greatly by condition and treatment. For 14 of the 37 necessary care indicators, the administrative data show that less than two thirds of Medicare beneficiaries with these conditions received care that a physician panel considered to be a minimum quality standard. Five avoidable outcomes occurred infrequently among all beneficiaries; results for the sixth avoidable outcome show that more than half of all patients with chronic obstructive pulmonary disease were hospitalized for a respiratory diagnosis. Of all beneficiaries, 87% had visited a physician 1 or more times in the past year, and 50% had had an eye examination in the past 2 years. Of the female beneficiaries, almost half of those younger than 75 years had had a mammogram in the past 2 years.

Table 2 displays the statistically significant results of the paired comparisons we calculated for the 3 vulnerable populations. We show the age- and sex-adjusted proportion of vulnerable and nonvulnerable beneficiaries who received the indicated care and the number of vulnerable and nonvulnerable beneficiaries who were eligible to receive the care. Results that were statistically significant at the 5% level as well as those that were statistically significant at the 5% level with a difference in size of 10% or more are indicated.

The results indicate that vulnerable populations (African Americans, those living in HPSAs, and those living in poverty areas) were less likely than their counterparts to receive necessary care and preventive care and were more likely to have higher rates of avoidable outcomes. Figure 1 shows the comparative results for these vulnerable populations. The figure indicates the total number of indicators (necessary, avoidable, and preventive) for which each respective vulnerable group received better, worse, or the same level of care (or had higher rates of avoidable outcomes) as nonvulnerable populations.

African Americans scored significantly worse on 16 of 46 indicators, 10 of the nonpreventive necessary care indicators, all 3 of the preventive care indicators, and 3 of the 6 avoidable outcomes indicators. African Americans had lower rates of follow-up after hospitalization and fewer necessary blood tests and eye examinations. African Americans scored significantly better in only 2 indicators. Health Professional Shortage Area and poverty area residents showed similar patterns. They scored significantly worse on 12 and 11 of the necessary care indicators, 3 and 3 of the preventive care indicators, and 1 and 3 of the avoidable outcomes indicators, respectively. Poverty area residents scored significantly better on only 1 indicator. Restricting analyses of necessary care indicators to patients enrolled for at least 13 months did not change the direction or significance of any of the necessary care indicator results.

We also compared underuse in vulnerable populations according to the strength of the evidence on which the indicators were based: randomized controlled trials and observational trials vs expert opinion. For the 12 indicators based on randomized controlled trials and observational trials, African Americans scored worse on 10 of the indicators, of which 5 were statistically significant, and scored better on 2 of the indicators; HPSA residents scored worse on 8 of these indicators, 4 of which were statistically significant, and scored better on 4 of these indicators; and residents of poverty areas scored worse on 10 of these indicators, 5 of which were statistically significant, and scored better on 2 of the indicators.

Of the 34 indicators based on expert opinion, African Americans scored worse on 30 of the indicators, 11 of which were statistically significant, scored better on 3 of the indicators, of which 2 were statistically significant, and were equal to non–African Americans on 1 indicator; residents of HPSA areas scored worse on 28 of the indicators, of which 12 were statistically significant, scored better on 4 of the indicators, and were equal to non–HPSA residents on 2 indicators; and residents of poverty areas scored worse on 27 of these indicators, of which 12 were statistically significant, scored better on 6 of the indicators, of which 1 was statistically significant, and were equal to residents of nonpoverty areas on 1 indicator.

The underuse monitoring system we describe is significant in terms of its breadth, the rigor with which the indicators were selected, and its relative ease of use. The indicators span several phases of care, including prevention, initial evaluation, diagnostic tests, therapeutic interventions, follow-up, and monitoring for acute, chronic, medical, and surgical conditions. Because we included preventive care indicators, every Medicare beneficiary is eligible for at least 1 indicator (ie, yearly physician visit); moreover, 45% of the beneficiaries were eligible for at least 1 non–preventive necessary care indicator. We selected the 47 indicators using a rigorous evidence-based process of literature review, feasibility testing, and expert opinion. Moreover, because the indicators were designed to be applied to routinely collected data, we avoided the costs and resources associated with medical record review.

When we applied the system to Medicare claims data, our results suggested that underuse of necessary care is widespread for the 15 target conditions, even in the relatively well-insured Medicare population. For almost half of the indicators, less than two thirds of beneficiaries received needed care. Underuse was more likely to occur among African Americans and residents of poverty areas or those areas with a shortage of health care professionals, indicating construct validity for our measures. Although some overlap may exist among these 3 vulnerable populations, this is not a serious issue since the results are similar for all of the groups. These findings persisted regardless of the strength of the evidence on which the indicators were based (randomized controlled trials and observational trials vs expert opinion).

The clinically based indicator system presented here has several advantages.³³ First, claims data are routinely collected and relatively inexpensive to analyze. Moreover, measures based on claims data can be calculated in a timely fashion, thus facilitating the repeated evaluations crucial to identifying trends and analyzing programmatic success. Claims data also allow for easy identification of geographical and ethnic subgroups with particular access problems. This system combines inpatient and outpatient claims data, thereby providing a more complete assessment of underuse, although the lack of pharmacy data limited indicator selection. Finally, the indicators were subject to rigorous expert review, though the experts often lacked randomized controlled trials to support their decisions.

By 2001, Medicare health maintenance organizations (Medicare+Choice organizations) will be required to submit encounter data to HCFA for care other than inpatient hospital stays.³⁴ In 2004, these data, including physician office visits and hospital outpatient department visits, will be incorporated into a new risk adjustment method that HCFA will use to make monthly capitation payments to Medicare+Choice organizations.³⁴ Thus, with access to inpatient and outpatient data for Medicare beneficiaries enrolled in both traditional Medicare and Medicare+Choice, the system could be applied routinely, uniformly, and inexpensively to the entire Medicare population. Medicare's current survey- and HEDIS-based systems, which compare Medicare+Choice and traditional Medicare, are costly and may be too expensive to be applied to all beneficiaries and may not translate well between the 2 settings. Beyond Medicare, the Institute of Medicine and the Presidents' Advisory Commission have called for a national quality report card, and a system such as this might serve as its basis in the short term.⁷

However, claims data–based systems have several limitations, as well. First, administrative data lack the clinical detail found in the medical record. Hannan et al³⁵ found that complications and comorbidities were more difficult to distinguish using administrative data than using medical records. Iezzoni et al³⁶ found that their administrative-based screening tool for hospital quality had a sensitivity of 0.92 and a specificity of 0.62 compared with charts. While prospective electronic clinical data collection would combine the detail of medical records with the ease of collection of administrative data,⁸ such systems are not yet available.

Validation studies of claims-based systems have shown them to be reasonably accurate but far from perfect. In 1 study, Medicare claims identified 95.6% of cataract surgeries.³⁷ However, other studies have found underreporting of chronic conditions in inpatient claims, although some of the conditions included in our system (diabetes, congestive heart failure, unstable angina, and malignancies) were better represented.^38-40 Similarly, Roos et al⁴¹ found that administrative data were quite specific (0.88-0.98) in identifying respiratory and cardiovascular surgical comorbidities when compared with anesthesiologists' assessments, but the data were not very sensitive. Adding outpatient claims to the system, as we did, may improve chart-claims agreement.⁴²

Another potential limitation involves tracking medical services obtained outside of the Medicare program. While Medicare is the primary payer and, thus, will receive the vast majority of claims even for services that other programs cover secondarily, there are some service categories that may be underrepresented. The design of the indicator system takes this into account; for example, there are no medication indicators because Medicare lacks an outpatient pharmaceutical benefit. However, some indicated services (eg, mammograms) may be obtained outside the Medicare program. Therefore, our results can be used to evaluate underuse only within Medicare.

It is also difficult to adjust our measures for risk. Since established risk adjustment systems predict the outcomes but not the receipt of indicated care, their applicability to the necessary care indicators presented here is questionable.⁴³ Adjusting for comorbidities using claims data is also problematic since it is sometimes difficult to distinguish complications (results of poor care) from comorbidities (predisposition to poor outcomes).³⁵ By limiting eligibility to a reasonably well-defined and homogeneous set of clinical circumstances, we implicitly risk-adjusted our results (eg, diabetic patients with and without comorbidities both need eye examinations). However, residual unaccountable risk may explain some of the observed variations. In assessing performance of individual hospitals or physicians, including comorbidities would be reasonable, especially for the avoidable outcomes indicators. For comparison of population-based measures, the age- and sex-adjusted measures reported here are an important first step.

Additionally, it should be noted that the mammogram indicator was calculated only for women younger than 75 years and does not meet the current clinical standard for yearly mammograms. When the study was conducted, Medicare would pay for a mammogram only once every 2 years for women who were not at high risk for breast cancer, which was the clinical standard at that time. Following the Balanced Budget Act in 1997, Medicare changed its policy and now covers yearly mammograms for all women aged 65 years or older.

In our calculations, we did not expect to obtain a value of 1.0 for any of the indicators, even among people with excellent access, because claims data cannot capture all clinical characteristics that comprise the indications and contraindications for specific services. In addition, billing and coding problems could affect the claims data, and we had no mechanism to track services that patients obtained outside of Medicare. This raises the issue of what the calculated values for each indicator mean. A benchmark approach could be used to solve this problem: by estimating the rates for white populations living in nonpoverty and non-HPSA areas, we could establish a benchmark with which the other values could be compared.

Despite these limitations, claims-based systems may be used in a variety of ways to inexpensively measure underuse. Screening administrative data to determine areas of a health care system in need of further investigation is the first step in a continuous quality improvement framework,^22,44,45 allowing identification of individual facilities or medical groups at risk. Moreover, many of the system's inherent biases will be reduced if the measures are used to track claims data for an extended period. Therefore, this system may be used to guide internal quality improvement efforts for large medical groups or plans, as well as purchasers' or regulators' evaluations. Future research, using chart reviews and patient surveys, is needed to directly validate the indicator system. However, the results of our initial application indicate substantial underuse, particularly among traditionally vulnerable populations.