How Far Have State Medicaid Agencies Advanced in Performance Measurement for Children?

Background  While children represent the largest population group enrolled in Medicaid managed care, little is known about the pediatric performance measures used by state Medicaid agencies.

Objective  To identify Medicaid managed care requirements for using Health Plan Employer Data and Information Set and other performance measures for children (defined as those aged 0-21 years in this study).

Design  A structured telephone survey of pediatric performance measures.

Participants  Survey respondents were state Medicaid officials responsible for managed care quality oversight in 39 states.

Main Outcome Measures  Percentage of states in 1998 with effectiveness-of-care measures on health promotion and disease prevention, early detection and screening, and acute and chronic illness; with use measures on preventive care, ambulatory care, pharmacy, inpatient hospital care, and mental health and chemical dependency services; and with access measures on primary care, low-birth-weight neonates delivered at appropriate facilities, and dental care.

Results  In 1998, state Medicaid agencies placed most of their emphasis on monitoring preventive care for children, with immunization rates being the primary focus. Far less attention was directed at assessing the treatment of acute illness. Although more than half of states monitored the treatment of chronic childhood conditions, they focused exclusively on asthma and selected mental health diagnoses.

Conclusions  States are still in the initial phases of designing and implementing quality oversight systems for Medicaid-insured children. Additional quality reporting requirements are clearly needed to assess the treatment of acute and chronic illness among children along with more age-specific reporting requirements.

THE SUBJECT of quality of care for Medicaid-insured children has received increasing attention during the past few years in response to the growing enrollment of children into managed care organizations and the release of Medicaid's Health Plan Employer Data and Information Set (HEDIS)¹ and its subsequent updates. Despite the growing interest in monitoring quality of care for children, little is known about what performance measures state Medicaid agencies use for children. Information is also not available regarding states' future directions for improving pediatric performance measures.

In 1999, only one published study² has examined state Medicaid quality-of-care requirements. That study of 30 Medicaid agencies, which included measures for all age groups and was conducted between 1995 and 1996, found that childhood immunizations, use of prenatal care in the first trimester, and satisfaction with care were the quality performance requirements most commonly adopted by states. At the time of that study, the Medicaid version of HEDIS had not been released. States did report, however, that they planned to collect more quality performance measures in the future.²

The lack of literature on quality monitoring for Medicaid-insured children is problematic for several reasons. Children enrolled in Medicaid are at greater risk of health problems than non–Medicaid-insured children.³ They also represent the largest and fastest-growing population served by Medicaid. More than half of all Medicaid recipients are children,⁴ and the proportion of child Medicaid recipients is likely to increase in the next few years as expanded outreach efforts bring in previously unenrolled eligible children and uninsured children at higher income levels who qualify for Medicaid as a result of the Children's Health Insurance Program.⁵ In addition, children are more likely to be served through managed care organizations than adults. Eight of the largest state Medicaid agencies reported in 1995 that between 63% and 73% of their managed care enrollees were children younger than 20 years.¹ Since that time, enrollment of children into capitated arrangements has con tinued to increase.^6,7 Unfortunately, no national data exist on the exact number of Medicaid-insured children enrolled in capitated managed care arrangements.

This article on pediatric performance measurement requirements under Medicaid managed care addresses the following 4 questions: (1) What HEDIS measures are states using for children in the areas of effectiveness of care, use of services, and access and availability of care? (2) What non-HEDIS measures are being used to adapt or substitute for HEDIS measures? (3) How do states rank overall on the breadth of their quality performance measures for children? (4) What are states' expected future directions in measuring quality of care for children? (In this article, children are defined as those aged 0-21 years.) Responses to these questions can help guide policy makers, managed care organizations, and clinicians in assessing the strengths and limitations of state Medicaid performance requirements for children. They can also provide guidance about potential opportunities for improving pediatric performance requirements.

Information presented in this article is based on a telephone survey of 39 state Medicaid agencies conducted in the late winter and spring of 1998. The states included in the sample contracted with managed care organizations at the end of 1996.⁶ We elected not to use the larger sample of 45 states that contracted with managed care organizations at the end of 1997⁷ because of the limited experience of these states with managed care contracting and quality monitoring.

Before conducting the telephone survey, we analyzed each state's model managed care contract and identified all performance measures pertinent to children. This information was used as background information for the survey. Originally, we intended to analyze information derived from the contracts; however, contracts were often vague with respect to quality reporting requirements. In addition, when attempting to verify contract information, we discovered significant disparities between the performance measures defined in the contracts and those that the state Medicaid agency staff reported were in place during the telephone survey. Most states had fewer contract specifications, while a few states had more. Possible reasons for this discrepancy are that many states do not update their model contracts annually and that states often develop plan-specific contracts that differ from their model health maintenance organization contracts. To provide an accurate picture of state activities, this article is based only on the information collected from the telephone surveys.

Survey respondents for each state Medicaid agency were typically the staff member(s) responsible for managed care quality oversight. In some states, where quality responsibilities were shared, 2 or 3 additional Medicaid staff members were interviewed, often including those responsible for Early and Periodic Screening, Diagnosis, and Treatment (EPSDT) and mental health services. The survey instrument was designed by us (M.A.M., R.R.G., H.B.F., C.M.M., C.E.I.) and pretested with a few states. Each interview lasted approximately 60 minutes. A 100% response rate was achieved through multiple callbacks.

State Medicaid respondents were asked a series of 50 closed-ended questions about their use of performance measures in 3 areas: effectiveness of care, use of services, and access and availability of care. Respondents from each state received a summary of the questions before the survey to allow them sufficient time to research their state's policies.

Because of the widespread use and acceptance of HEDIS by managed care organizations and state Medicaid agencies, survey questions were designed to elicit states' use of child-specific measures from HEDIS. (Measures were obtained from HEDIS draft version 3.0, which was widely available at the time this survey was conducted.⁸ In the final version of HEDIS 3.0, only 1 pediatric measure was eliminated—use of appropriate medications for people with asthma.) Survey questions were also designed to elicit the use of state-developed or state-modified HEDIS measures. Often, states modified HEDIS measures by changing the age group or the period for required Medicaid enrollment. In some instances, states used a combination of HEDIS and non-HEDIS measures.

For this study, we collected information pertaining to capitated services, including those delivered by behavioral health and dental plans. Only quality performance requirements specific to children aged 0 to 21 years were examined. Except for measures relating to low-birth-weight neonates, no pregnancy-related measures were included in this study. Quality monitoring activities that did not include an objective data set or method of measurement were also excluded from our analysis. In addition, we did not ask state Medicaid agencies how each set of measures was actually being used. Thus, what is reported in this article is only the scope of pediatric performance measures that Medicaid managed care organizations were required to report on in 1998.

Survey questions on effectiveness of care covered all aspects of health care—from health maintenance and disease prevention to early detection and screening to treatment of acute and chronic illness. Questions on utilization covered a range of services, including preventive care, ambulatory care, inpatient hospital care, mental health and chemical dependency treatment, pharmacy, and other services. Questions regarding states' methods for monitoring EPSDT were also included. Questions on access and availability of care addressed children's access to primary care providers, low-birth-weight neonates delivered at appropriate facilities, annual dental care visits, and access to family planning services. General access measures applicable to all populations, such as appointment waiting times for emergent, urgent, and routine care, were not examined. We did, however, include appointment waiting times if they were specific to children (eg, waiting times for EPSDT screens for new members).

As part of our analysis, we ranked states according to the number and type of pediatric quality reporting requirements used. Using HEDIS 3.0, we identified a core set of 10 child health measures. For effectiveness of care, we identified 4 pediatric measurement areas—immunization rates, low-birth-weight rates, treatment of acute conditions, and treatment of chronic physical or behavioral health conditions. For use of services, we also selected 4 measurement areas—well-child visits, ambulatory care, pharmacy services, and inpatient hospital care. For access and availability of care, we identified 2 pediatric measurement areas—access to primary care providers and low-birth-weight neonates delivered at appropriate facilities. According to this ranking system, a state with measures in all of these 10 areas would have the most comprehensive set of quality performance measures for children. We eliminated from our list of pediatric measures behavioral health and dental care use measures because as many as a quarter of states paid for all of these services on a fee-for-service basis outside of managed care.

We analyzed the association between the number of core pediatric measures and certain managed care variables, expecting to find a positive association between each of the variables and the number of pediatric performance measures. The managed care variables included the following: (1) state experience with managed care, which was defined as the length of time that a state Medicaid agency enrolled children into capitated arrangements on either a voluntary or a mandatory basis; (2) state managed care penetration, which was defined as the proportion of a state's Medicaid recipients enrolled in health maintenance organizations, health-insuring organizations, prepaid health plans, and other capitated arrangements; and (3) state policy variance from federal Medicaid managed care requirements, which was defined as the implementation of a section 1115 demonstration waiver program.^9,10 (Under section 1115 of the Social Security Act, a state is able to mandate enrollment into managed care, contract with plans that do not meet federal requirements, and lock enrollees into a particular plan for as long as 12 months.)

State Medicaid agencies reported that the most commonly used measure of effectiveness of care for children was the administration of childhood immunizations, as shown in Table 1. All but 2 of the 39 states in our sample required managed care plans to report on immunization rates. The HEDIS specifications for vaccinations among 2-year-old children were used in about three quarters of states. Less than half of states, however, used the adolescent immunization HEDIS measure.

Immunization is one of the few quality reporting areas for which states instituted incentives or penalties to achieve a certain level of compliance. Five states reported offering managed care plan incentives in the form of payment bonuses or additional auto-assigned enrollees. Another 5 states applied penalties in the form of payment withholds, refunds, or reductions in auto-assigned enrollees.

The early detection and screening measure most often required by states was the low-birth-weight rate. Of the 26 states requiring this, most followed HEDIS specifications for rates of low- and very low-birth-weight neonates. Non-HEDIS measures on early detection, used in 8 states, were for lead poisoning or cervical cancer screening in adolescents.

Reporting of pediatric acute illness treatment measures was seldom part of state Medicaid agency requirements for health maintenance organizations. Of the 7 states with an acute illness measure, the HEDIS otitis media measure for children younger than 5 years was used most frequently.

Reporting of chronic illness treatment measures was more often required by state Medicaid agencies than measures for the treatment of acute illness. As many as 24 states monitored asthma treatment, although fewer than a third used the HEDIS specifications for children between the ages of 5 and 20 years. Most of the states instead conducted focused studies on asthma treatment or analyzed emergency department or hospital use rates among children with asthma. The only other chronic care measure was for diabetes; this non-HEDIS measure was used by 1 state. No other chronic physical condition measures were adopted by states, although many states expressed an interest in adding measures in this area in the future.

States were less likely to monitor treatment of mental health conditions than treatment of chronic physical conditions. Seventeen of the 28 states that capitated all or most of these services had mental health measures, and almost two thirds of these used the HEDIS measure, which specifies ambulatory care follow-up for children older than 6 years who are hospitalized for certain mental health disorders. Interestingly, we found that states using managed care plans to deliver both physical and mental health services were twice as likely to have quality-reporting requirements in this area compared with states separately contracting with behavioral health plans.

The reporting changes that most states expect to introduce in the future are likely to coincide with future modifications of HEDIS. However, some states reported efforts to develop their own performance measures based on the encounter data received from plans.

Although state Medicaid agencies differed in the number and type of requirements they imposed on plans regarding service use reporting, most states required plans to submit inpatient, ambulatory care, and preventive care use data for children.

The service most often monitored for children was inpatient hospital care, as shown in Table 2. More than three fourths of the 31 states with an inpatient hospital use measure followed HEDIS specifications for the hospitalization of children aged younger than 1, 1 to 9, and 10 to 19 years.

Ambulatory care service use rates were monitored in 29 states. Most of these states included HEDIS reporting requirements for emergency department visits and outpatient hospital visits, based on the same age breakdowns used for hospital care. Ambulatory surgery or outpatient procedures were much less likely to be required measures. Only 4 of the 13 states with ambulatory surgery or procedure measures monitored myringotomy or tonsillectomy rates.

About the same number of states monitored the periodicity of well-child visits, primarily following the HEDIS specifications for infants (aged 0-15 months), young children (aged 3-6 years), and adolescents (aged 12-21 years). All states required some type of EPSDT reporting, however. A third of states required plans to submit EPSDT data directly from the Health Care Financing Administration's (HCFA's) Annual EPSDT Participation Report Form 416, while the remaining two thirds allowed plans to submit encounter or claims data, which the state used to complete Form 416. States cited numerous difficulties in obtaining accurate EPSDT reports from providers using Form 416, including confusion about when to use EPSDT vs Current Procedural Terminology codes, lack of information on children's period of enrollment and type of eligibility, and an inability to compile the total number of screening services, including vision and hearing tests, received by children. Concern about duplicative reporting for the HEDIS preventive care measure was also mentioned. Nevertheless, many states reported using incentives or penalties to encourage a higher level of EPSDT compliance. Twelve states imposed penalties and 7 states used incentives similar to those used to assure immunization compliance.

Only 19 of the 28 states that capitated all or most mental health and chemical dependency services for children monitored their use. About two thirds of these states followed at least some of the HEDIS specifications for children aged 0 through 12 and 13 through 17 years. Interestingly, as many as 15 states used non-HEDIS measures, most often addressing additional services, such as residential treatment or emergency department care, or hospital readmission rates within fewer days than those specified by HEDIS (30 vs 90 days). Again, our survey results revealed that states that capitated their mental health services through managed care plans were twice as likely to require reporting on mental health service use than states that contracted separately with behavioral health plans.

Only 14 states monitored the use of pharmacy services for children. Most states with pharmacy measures used the HEDIS specifications, which measure the total and average number of prescriptions per month for children between the ages of 0 and 9 and 10 and 19 years. Another 4 states adopted non-HEDIS pharmacy measures, which were typically based on the use of specific types of medications.

When asked about other pediatric health service use reporting requirements, several additional services were mentioned. Specifically, home health visit rates were monitored in 12 states, physical therapy visit rates in 9 states, and occupational therapy and speech therapy visit rates each in 7 states. Less than half of these states monitored home health and therapy services.

The changes that states anticipate making in the future for service use monitoring pertain to EPSDT reporting and to new HEDIS measures as they evolve. Two states, however, mentioned their interest in monitoring services used by children with chronic conditions.

Compared with effectiveness of care and service use, few measures for access and availability relating specifically to children were included in states' quality-of-care programs, as shown in Table 3. Rather, states used appointment availability, distance, and telephone access measures that are applicable to child and adult Medicaid recipients. They also used general member satisfaction surveys to evaluate access and availability of care.

The pediatric access performance measure most often used by state Medicaid agencies was access to primary care providers, which was required in 26 states. Two thirds of these states used the HEDIS specifications and monitored primary care visits for children between the ages of 1 and 11 years. Another 15 states used non-HEDIS measures—either primary care provider–child population ratios or compliance rates for visits following enrollment. The provider ratios, used in 9 states, varied from 1 pediatric practitioner per 1000 children to 1 per 2000 children. Waiting times for pediatric visits, used in 6 states, stipulated that plans must offer new child members a visit or an EPSDT screen within periods that ranged between 21 and 90 days after enrollment.

Two thirds of the 28 states that capitated dental services had an access measure on dental care. Of these 19 states, about a third used the HEDIS specification for annual dental visits for children aged 4 to 21 years. Most of the 12 states that used non-HEDIS measures specified annual dental visits for children and adults.

Access to appropriate hospital facilities for high-risk infants was monitored in only 9 states. Most states with this measure used the HEDIS specifications.

When asked about future changes in this area of quality monitoring, most respondents noted that their state would consider new HEDIS modifications, if developed. A few also mentioned an interest in monitoring timely access to EPSDT visits for new managed care enrollees.

Based on our survey of 39 state Medicaid agencies, we found that, on average, states monitored 6.2 of the 10 core pediatric quality-of-care measures. While 33 of 39 states had at least 1 measure in each of the 3 performance measurement domains (effectiveness of care, service use, and access and availability), states varied with respect to the total number and type of measures for which plans were required to report. State Medicaid agencies frequently had at least 3 pediatric measures in the effectiveness-of-care and the service use domains. In contrast, they usually had only 1 access or availability measure specifically for children; in several cases, they had none. Overall, when states were ranked according to the number of potential pediatric measures, one fifth (n=8) were classified as having a high ranking (8-10 measures), more than two thirds (n=27) were classified as having a middle ranking (4-7 measures), and the remaining one tenth (n=4) were classified as having a low ranking (0-3 measures).

States implementing section 1115 demonstration waivers were somewhat more likely to have more core pediatric measures, probably because they are federally required to submit their encounter data for external review and to conduct specific quality monitoring activities. Experience in contracting with capitated managed care and managed care penetration rates were not, however, associated with more pediatric performance measures.

State Medicaid agencies placed most of their quality performance monitoring emphasis for children on preventive care services, requiring managed care organizations to report on immunization rates, well-child visit rates, and EPSDT use rates. However, according to most state Medicaid agency staff, the EPSDT data collection method, based on HCFA's Form 416, is nearly impossible to implement. Relatively little emphasis was placed on acute care measures. By comparison, chronic care measures for children, based on the treatment of asthma and selected mental health diagnoses, are somewhat more common. Still, more than a third of states have no measures for the treatment of asthma or any other chronic physical condition, and the same proportion have no measures for the treatment of mental health conditions. Our survey revealed that many states are interested in adding performance requirements for childhood chronic conditions but are struggling with the difficulties associated with monitoring low-incidence conditions with variable functional and severity levels. For monitoring mental health and chemical dependency among children and adolescents, several states noted specific challenges, including the use of separate behavioral health plans, greater reliance on non-HEDIS measures, and oversight responsibilities delegated to state mental health and substance abuse agencies. Thus, in most states, it appears that behavioral health services are not being held to the same level of scrutiny as physical health services.

Pediatric quality reporting requirements used by state Medicaid agencies primarily address infants and young children. They address adolescents to a somewhat lesser extent. Children between the ages of 6 and 10 years, however, are seldom specifically identified in any of the effectiveness-of-care, service use, or access measures required by state Medicaid agencies. States' ability to monitor plan performance for this middle age group is limited by the fact that almost all of the HEDIS use measures specify broad age categories (eg, 1-9 and 10-19 years) and, therefore, mask important distinctions among children of different ages. Health Plan Employer Data and Information Set effectiveness-of-care and access measures, including HEDIS 1999, also fail to separate the middle childhood age group. In addition to gaps in age-specific performance measures, our survey revealed that few states require plans to submit their data by child eligibility group served. As a result, states have a limited ability to compare plan performance in serving children at higher risk of adverse health outcomes.

Our survey confirmed that states are relying on HEDIS specifications as the basis for their own performance measures. Moreover, future changes in state Medicaid quality reporting requirements will likely be based on HEDIS modifications. Many states, though, are attempting to create their own encounter-based reporting systems, which they anticipate will ultimately allow them greater flexibility in evaluating more types of health services, smaller age groupings, and additional conditions. However, significant operational issues must be overcome before the successful use of these data sets, which were never originally designed for quality reporting purposes. States that receive claims and encounter data from health plans repeatedly commented about the inaccuracy of the data and the level of effort necessary to audit the data sets. This issue is likely to continue and perhaps amplify as more commercial plans leave the Medicaid market and smaller physician, clinic, and hospital-based organizations remain.

Despite the fact that children are the main population group insured by Medicaid, states are still at an early stage in implementing a comprehensive set of performance measures. With the exception of immunization rates, no other pediatric quality-of-care measure is used consistently across all states. This makes cross-state comparisons difficult, if not impossible. While state Medicaid agencies are relying on HEDIS measures to guide their quality oversight programs, at this time they are only using a selected set of measures—often because plan data are unavailable or unreliable. Our analysis of reporting requirements for 10 pediatric measures further underscores the few required measures and the variations across states, despite the fact that all states are providing essentially the same set of health care services to children. Whether the number or type of pediatric measures used by states will result in child health improvement is unknown. Yet, it is evident that most states are opting for a small number of child-specific reporting requirements.

Several reasons may account for the fact that a common set of HEDIS pediatric performance measures has not been uniformly adopted. The HCFA has never mandated the use of HEDIS or any specific subset of HEDIS measures. Moreover, states have already invested in their own unique quality management systems. Finally, the costs of collecting uniform, reliable data are high and the burden on managed care organizations and pediatric providers for measuring performance has been much higher than anticipated.

These findings suggest the need for the federal government to establish a minimum uniform set of pediatric quality reporting requirements that addresses not only preventive care but also acute and chronic care for children insured by Medicaid, the Children's Health Insurance Program, and other public and private mechanisms. A broader set of measurement specifications that focuses on well children, children with acute illnesses, and children with chronic illnesses (including those with mental health conditions) would represent major improvements in Medicaid quality reporting.

Our study revealed that most states accept selected HEDIS measures without revision. They simply do not have the expertise or the money to develop their own quality measures. Measurement development is more appropriately the responsibility of national agencies and academic institutions, whereas states, as they reported to us, must concentrate on assuring participation and collaboration with managed care plans.

Efforts under way by the Foundation for Accountability¹¹ to develop recommendations for a new set of child and adolescent measures for preventive, acute, and chronic care are well-timed and could be instrumental in advancing states' capacity to evaluate all aspects of pediatric care—whether financed through Medicaid, the Children's Health Insurance Program, or privately funded sources. It will be important for the HCFA and for other federal agencies and national organizations to define and fund specific priorities for pediatric measurement development. Our findings indicate that the first focus should be on measurement development for acute and chronic care for children (including those with behavioral health disorders) since states have so few measures in place to monitor these services.

Consistent with the new quality requirements established under the Balanced Budget Act of 1997,¹² states are required to have a quality assessment and performance improvement strategy that will be assessed at least every 3 years. States are also mandated to incorporate this quality strategy into managed care contracts. As a result, discrepancies between contract language and actual quality requirements should be eliminated. It is certainly possible, even likely, that in the future states will significantly improve their pediatric quality monitoring for Medicaid, and possibly also for the Children's Health Insurance Program, by meeting the requirements of the Balanced Budget Act of 1997 and adopting the new pediatric measures recommended by the Foundation for Accountability. If the new pediatric version of the Consumer Assessment of Health Plans is used by all state Medicaid agencies, this would represent a major advance in assessing satisfaction with managed care for children. Moreover, if new methods under development by the Foundation for Accountability were used in the future to screen children with special health care needs, it would be possible to evaluate satisfaction with care among children with and without chronic conditions.

Despite the improvements that are likely to take place in pediatric measurement development in the near future, states repeatedly mentioned the need for more technical support to implement existing HEDIS requirements, including improved state and plan data systems. They also cited a need for guidance on the use of encounter data. The HCFA and its regional offices, in collaboration with the National Committee for Quality Assurance, the Foundation for Accountability, and other quality experts, should consider offering more intensive and ongoing continuing education and technical support services to state Medicaid agencies, managed care organizations, and pediatric practices. Without this investment, it is unclear how effectively states can monitor that children enrolled in managed care organizations receive the highest possible quality of care.

Accepted for publication January 5, 2000.

This study is a joint product of the 3 centers funded by the federal Maternal & Child Health Bureau, Rockville, Md: grant MCU-06MCP1 from the Maternal & Child Health Policy Research Center on Managed Care and Children With Special Needs, Washington, DC; grant MCU-069384 from the Policy Information and Analysis Center on Childhood and Adolescents, University of California, San Francisco; and grant MCU-069385 from the National Policy Center for Infancy and Early Childhood Research, University of California, Los Angeles.

We thank the state Medicaid agency staffs responsible for quality of care who participated in our survey; Tara Murphy, BA, and Christine Chen, BA, of the Maternal & Child Health Policy Research Center for assisting in data abstraction; and Robert Pantell, MD, Claire Brindis, PhD, Courtney Cart, MSW, MPH, Paul Newacheck, DrPH, and Neal Halfon, MD, for their helpful comments.

Corresponding author: Margaret A. McManus, MHS, Maternal & Child Health Policy Research Center, 750 17th St NW, Suite 1025, Washington, DC 20006-4607 (e-mail: mcmanus@wizard.net).