[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
Purchase Options:
[Skip to Content Landing]
July 5, 2000

Ethical Considerations in the Public Policy Laboratory

JAMA. 2000;284(1):85-87. doi:10.1001/jama.284.1.85

Standards for evaluating clinical innovations are very different from standards for evaluating policy innovations. In the case of the former, the research community appropriately insists on exquisite attention to fair participant selection, a favorable risk-benefit ratio of the intervention, informed consent, and respect for enrolled subjects.1 Conversely, innovations in public policy are initiated by legislative action or agency dictate; evaluation (if it occurs) is often retrospective and conducted without the oversight of an institutional review board (IRB) or without informed consent. In this issue of THE JOURNAL, Kerpelman et al2 describe an admittedly controversial program that used the threat of sanctions of welfare benefits to encourage compliance with early childhood immunizations. Their evaluation highlights some of the ethical problems involved in policy evaluations.

In 1993, the state of Georgia began to require that all families receiving Aid to Families with Dependent Children (AFDC) provide proof of up-to-date immunization status for all children aged 6 years and younger. According to the policy, families that did not provide this proof would be warned about sanctions if they failed to obtain immunization documentation for a child and would ultimately lose benefits for persistently underimmunized children. To evaluate this initiative (called the Preschool Immunization Project [PIP]), 1500 families enrolled in AFDC in Muscogee County, Georgia, were randomly chosen as the intervention group and received the "incentive" of the threatened loss of their AFDC benefits. For comparison, 1000 AFDC-beneficiary families from the same county were exempted from the new statewide policy. Though the latter group of families were encouraged to immunize their children, they received no information regarding sanctions. Over a 4-year period of follow-up, Kerpelman et al found that immunization rates were significantly higher (6%-7% for single immunizations; 12% for combined series completion at age 24 months) among children whose parents knew that their AFDC benefits were at risk. Positive effects of the intervention were most apparent during the first 2 years and were generally maintained subsequently.

On the surface, the PIP improved immunization rates among preschool-aged children. But was the threat of sanctions ethical, as a policy directed toward improving children's health? Negative incentives, although their effectiveness may be debated, are common in public policy: eg, financial penalties for late filing of taxes, loss of licensure for severe traffic offenses, or restricting school attendance for underimmunized grade school–aged children. The ethical stumbling block in a program like PIP rests in the fact that financial penalties for delayed immunization were threatened only to individuals already at high risk for economic deprivation: low-income, predominantly minority (black) families with a high proportion of single parents and young children3 who were dependent on welfare support. Research involving racial minorities and among vulnerable populations in general raises particular ethical issues that have been the focus of concern and reform over the last half century. While reducing persistent health disparities among minority populations (including significant inequalities in childhood immunization rates) is now a national research priority,4 the ethical imperative in research efforts is to decrease—not potentially increase—the overall risk borne by individuals more likely to experience poor health. Programs that link AFDC sanctions to immunizations have prompted concern about placing parents and children at heightened risk for deprivation, and critics have called for a thorough investigation of the untoward effects of such programs.5,6

How much is known about the untoward effects of the PIP? As Kerpelman and colleagues2 report, only 17 warnings were given, and very few families (11 of 1500; <1%) ever received actual sanctions. Of these 11 sanctioned families, 8 reapplied for AFDC and their benefits were reinstated within a period of 1 to 6 months. The authors suggest that the sanction intervention worked by providing the threat of sanctions, but that, because the threat was seldom carried out, no harm was done. However, other than tracking changes in immunization status and surveying 40 families (<3%) about the additional burden of requiring immunization documentation, the investigators did not evaluate more fully the impact of the threat of sanctions on parents' behavior. For example, significantly more families dropped out of the intervention group than the control group (38.3% vs 34.5% at 4 years). The authors present no data on whether these rates are attributable to differences in follow-up AFDC eligibility, family relocation outside the county, or families' responses to the punitive nature of the intervention. Information on whether parents left AFDC simply because of the threat of monetary sanctions, whether sanctions were applied or not, or how threats of sanctions changed parents' perceptions of the process of obtaining preventive health care for their children would help to characterize more completely the possible negative consequences of this Georgia program and judge whether the increase in immunization rates justified the overall burden imposed on families.

Over the past decade, several studies have established the effectiveness of less controversial programs that improve immunization rates among low-income children. Provider reminder-recall systems, provider feedback interventions, and multicomponent programs that include educational and motivational elements as positive incentives for parents have improved immunization rates in multiple studies (as reviewed by Shefer et al7). In addition, a few studies have promoted adherence to immunization for families enrolled in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC),8-10 in which program beneficiaries were required to attend WIC sites more often than usual if their children were not up-to-date on immunizations (ie, negative incentives were used). In contrast to AFDC beneficiaries in the PIP, however, WIC beneficiaries in these studies did not receive the threat of reduction in their overall benefits. Rather, more frequent visits ensured that benefits would be provided, despite noncompliance with immunization recommendations. For example, Birkhead et al8 conducted an intervention trial in 6 WIC sites in New York City between April and September 1991. Parents of children whose immunizations were up-to-date were issued the usual 2-month supply of food vouchers; parents of children not up-to-date with immunizations received a 1-month supply of vouchers. Children in the intervention group were 2.9 times more likely than controls to have received measles immunization overall and to have been immunized against measles more rapidly (26 days vs 45 days) after an index WIC visit.

When the Georgia legislature enacted the PIP initiative in 1992, the standard of care for improving immunization rates among poor children was not as clear as it is today. As states attempted to address inadequate immunization rates among preschool-aged children 8 to 10 years ago, many state governments opted not to use punitive measures but rather decided to strengthen the links between welfare and health programs. Several states, such as Alabama, Connecticut, Delaware, Hawaii, Nebraska, and South Dakota, established a broad range of interventions, including home visits to families of participants in job training programs, on-site immunizations at job-training offices, referrals for underimmunized children from welfare agencies to health care providers, and links to immunization clinics for beneficiaries of AFDC, Medicaid, and food stamps.5 Such programs targeted systemic barriers to access that affected poor families and sought to ameliorate them. In contrast, programs with AFDC sanctions exposed common barriers to health care access for poor families (eg, lack of a regular source of care,11 lack of child health insurance,12 or both in the era before the State Children's Health Insurance Programs) and then threatened to sanction the families for failing to resolve those barriers.5

The PIP also was controversial because, unlike a clinical investigation, the evaluation did not seek IRB approval or the prior informed consent of study participants. Instead, because the PIP was a federally sanctioned, state-level welfare innovation, the US Department of Health and Human Services (HHS) waived the need for IRB review or informed consent (although all parents included in the analysis gave written informed consent for the release of their children's immunization records). Under Section 1115 of the Social Security Act,13 the secretary of HHS may waive compliance with core AFDC legislation (namely, the requirements for states to provide welfare benefits for certain individuals) so that states may carry out projects "judged likely to promote the objectives of AFDC."3 During the past 7 years, such AFDC waivers were approved for more than 40 states,3 as a means of encouraging states to serve as "laboratories" for welfare policy innovation.14 Although the waiver legislation calls for evaluations of demonstration projects by objective investigators, with random assignment if possible, little mention is made of participant protections, except in the case of child support initiatives.13

The Office for Human Research Protections, which is expected to replace the Office for Protection from Research Risks by July 2000, is responsible for ensuring the safety and welfare of individuals who participate in HHS-sponsored investigations.15 As did the Office for Protection from Research Risks, the Office for Human Research Protections works under the auspices of HHS to oversee the work of local IRBs to maximize the benefits of clinical research while minimizing risks to subjects.15 Accordingly, the goals of Section 1115 and the Office for Human Research Protections should be consistent under the common oversight of HHS. An assistant secretary of HHS who recently testified before Congress said, "[HHS] must never lose sight of this department's responsibility to families who depend on our programs for assistance . . . and who deserve to be protected when they become subject to demonstrations."14 This perspective, expressed in 1999, unites the goals of welfare reform and responsible research, but it is not clear whether the same degree of attention to subjects' rights was part and parcel of welfare demonstration projects at the time the PIP was initiated. Of note, a similar, contemporaneous program with AFDC sanctions implemented in Maryland (which failed to demonstrate improvement in immunization status) also had an IRB waiver but indicated in their published evaluation that their protocol met local institutional requirements for clinical studies.16

Although the study by Kerpelman et al can be criticized for not examining extensively enough the potential negative impacts of the PIP's threatened sanctions, the ultimate responsibility for Georgia's AFDC beneficiaries lies squarely with the state's Department of Human Resources and also with HHS. In granting waivers for demonstration projects that permit program evaluations to proceed without IRB oversight or without obtaining prior informed consent, state and federal agencies are obliged to take responsibility for protecting participants and ensuring that evaluators attempt to measure not only positive outcomes of interest but also outcomes that adversely affect families' well-being. While not every program granted an HHS waiver will succeed, the process for evaluating waiver programs should accurately identify innovations that are safe. Lawmakers and their constituents should recognize that evaluations that fail to characterize fully the negative consequences of welfare policy innovations may overestimate eventual programmatic benefit and may underestimate worsening health status among eligible program beneficiaries.

The image of states as local laboratories of public policy innovation has certain appeal. However, when the public policy in question is welfare support for impoverished children and their families, investigators and citizens share a collective responsibility to make certain that each state's public policy laboratories meet the highest ethical standards. Evaluations of welfare programs are research and should be held to the same ethical standard as clinical evaluations.

Emanuel EJ, Wendler D, Grady C. What makes clinical research ethical?  JAMA.2000;283:2701-2711.Google Scholar
Kerpelman LC, Connell DB, Gunn WJ. Effect of a monetary sanction on immunization rates of recipients of Aid to Families with Dependent Children.  JAMA.2000;284:53-59.Google Scholar
US Department of Health and Human Services.  Green Book Overview of Entitlement ProgramsWashington, DC: US Dept of Health and Human Services; 1998.
 Hearings Before the Subcommittee on Labor, Health and Human Services, Education, and Related Agencies of the Senate Committee on Appropriations.  105th Cong, 2nd Sess (1998) (testimony of Donna Shalala, secretary of the US Dept of Health and Human Services).
George E. Needling the System: Welfare Agency Approaches to Preschool ImmunizationWashington, DC: Center for Law and Social Policy; 1993.
Wood D, Halfon N. Reconfiguring child health services in the inner city.  JAMA.1998;280:1182-1183.Google Scholar
Shefer A, Briss P, Rodewald L.  et al.  Improving immunization coverage rates: an evidence-based review of the literature.  Epidemiol Rev.1999;21:96-141.Google Scholar
Birkhead GS, LeBaron CW, Parsons P.  et al.  The immunization of children enrolled in the Special Supplemental Food Program for Women, Infants, and Children (WIC): the impact of different strategies.  JAMA.1995;274:312-316.Google Scholar
Hoekstra EJ, LeBaron CW, Megaloeconomou Y.  et al.  Impact of a large-scale immunization initiative in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).  JAMA.1998;280:1143-1147.Google Scholar
Hutchins SS, Rosenthal J, Eason P.  et al.  Effectiveness and cost-effectiveness of linking the Special Supplemental Program for Women, Infants, and Children (WIC) and immunization activities.  J Public Health Policy.1999;20:408-426.Google Scholar
Brennan N, Rajan S. National Survey of America's Families, 1997: children and nonelderly adults with no usual source of health care. Washington, DC: The Urban Institute. Available at: http://newfederalism.urban.org/nsaf/health_b4.html. Accessibility verified June 9, 2000.
Brennan N, Holahan J, Kenney G. National Survey of America's Families, 1997: health insurance coverage of children. Washington, DC: The Urban Institute. Available at: http://newfederalism.urban.org/nsaf/health_b1.html. Accessibility verified June 9, 2000.
 Social Security Act, 42 USC 1315 §1115 (1965). Available at: http://www.ssa.gov/OP_Home/ssact/title11/1115.htm. Accessibility verified June 9, 2000.
 Hearings Before the Subcommittee on Government Management, Information and Technology and Subcommittee on National Economic Growth, Natural Resources and Regulatory Affairs of the House Committee on Government Reform . 106th Cong, 1st Sess (1999) (testimony of John Callahan, assistant secretary for management and budget of the US Dept of Health and Human Services).
 New Office for Human Research Protections created, Dr Greg Koski named director [fact sheet]. Washington, DC: US Dept of Health and Human Services; June 6, 2000. Available at: http://www.hhs.gov/news/press/2000pres/20000606.html. Accessibility verified June 13, 2000.
Minkovitz C, Holt E, Hughart N.  et al.  The effect of parental monetary sanctions on the vaccination status of young children: an evaluation of welfare reform in Maryland.  Arch Pediatr Adolesc Med.1999;153:1242-1247.Google Scholar