Evidence, Anecdotes, and Health Policy

It is often hard to comprehend the human toll of public health disasters. How can anyone possibly understand 1 million lives lost to COVID-19 in the US? Sometimes hearing the story of one life lost can be a window into the tragedy of countless others. The same is true of so many health policy decisions that affect millions, but may seem abstract; the stories of individuals can drive home the real importance of those decisions.

But there is a risk to drawing policy conclusions from stories, no matter how powerful. Most policies have complicated effects that are different for different people. Stories that are true can still be quite misleading because any individual’s experience may not be representative of the effect of the policy for most people. It is the best part of human nature to be moved by a fellow human being’s story, but it is a dangerous way to make policy.

Consider the effects of expanding health insurance. Having insurance can affect the health care that people use across many settings (such as office visits, screening tests, prescription drugs, and emergency department [ED] visits) and thus spending on health care. It can also affect financial security (such as debt and overdue bills) and health outcomes (such as mental health, management of chronic physical conditions, mortality, and overall well-being). Assessing the effect of insurance expansion thus requires incorporating these multifaceted outcomes. But those effects might also be quite different for different people.

For example, does health insurance increase or decrease ED use? One can imagine 2 equally plausible answers. On the one hand, by reducing the cost of an ED visit for patients, insurance might lead to more visits for those with insurance than those without insurance, all else being equal. On the other hand, by increasing access to preventive and primary care and potentially improving health, health insurance might lead to fewer ED visits for those with insurance than those without insurance. As with many questions in health policy, we need rigorous data and analyses to generate valid answers.

Getting a good answer first requires a way to separate out the effect of insurance itself from all of the other confounding factors that affect insurance status, health care use, and health outcomes. For example, jobs that offer health insurance may also tend to come with higher wages, and people in those jobs may also have better access to safe environments, housing, social supports, and high-quality health care. Separating those factors out requires thoughtful statistical analysis and the right setting (such as a randomized clinical trial¹ or a quasi-experimental approach^2,3). Within these careful designs, data analysis can yield insights about both overall effects and how those effects vary across people. It is worth noting that data do not have to be quantitative. Important lessons can be learned from systematic analysis of qualitative data, but that analysis does need to be systematic.

In the Oregon Health Insurance Experiment—a study on which I collaborated with Amy Finkelstein, PhD, Heidi Allen, PhD, MSW, Bill Wright, PhD, and others—we collected qualitative as well as quantitative information in the context of a lottery used to allocate limited slots in Oregon’s Medicaid program—a de facto randomized clinical trial. The stories we heard from participants about their experiences and their health were compelling and also illustrated how different those experiences could be.⁴

Consistent with the idea that covering uninsured individuals could reduce ED use, one uninsured respondent told us, “When I was uninsured, unless something happened where I had to go to the hospital, then I’d just go to the emergency room and deal with it. Emergency rooms, from what I understand, they can never turn you away.” But consistent with the idea that insurance would increase the use of the ED, an insured respondent told us, “Without coverage I wouldn’t have gone to ER those nights I was in crisis because I was already in crisis, and [the bills] just would have been too much for me to take on mentally or financially.”

Both of these stories convey important lived experiences. But which story is representative of the experience more people are likely to have?

Our analysis showed that insured persons went to the ED 40% more than uninsured individuals.⁵ This finding surprised many people. Was it because Medicaid did not actually enable access to high-quality primary care? Some patient stories suggested this might be the case. One respondent on Medicaid reported that “They [Medicaid] have no doctors that are actually seeing new patients,” and another reported that the clinic that the new enrollee went to “said they would contact me and then they never did.” But others reported that Medicaid gave them new access to the care they needed. One noted, “I love my doctor that I see with [Medicaid]. He calls me all the time to check up on me.” Another said, “I thought maybe [with state insurance] you just get the minimum care, they’re not really going to go further, but I was totally wrong…I felt like I got the full care that I needed.” The net result was that Medicaid increased physician office visits and ED visits; insurance did not lead to substitution of office visits for ED care.^6,7

This information is crucial for policy makers to have. Many have argued that expanding insurance will save money because previously uninsured individuals will get preventive and primary care that will stave off more expensive ED visits. We did not find that to be the case. This finding does not mean that insurance should not be expanded. Rather, it helps paint a realistic picture of the costs to be weighed against the benefits.

This is just one illustration. In fact, Amy Finkelstein and I created a quiz that people can try for themselves (although a couple of the questions are already answered above). This quiz has a series of paired stories relaying very different experiences. Absent systematic analysis, each seems potentially persuasive and could be marshaled as evidence in support of a particular policy choice, making such human stories both extremely valuable and potentially dangerously misleading. Someone in favor of expanding Medicaid can find many true stories of people for whom the insurance policy kept them out of the ED and reduced the overall spending on their health care, yet the overall effect of the Medicaid expansion we studied was to increase ED use.

We ought not to ignore the personal stories that lend meaning and generate empathy and understanding in the name of evidence-based policy. Putting a human face on policy choices is crucial to internalizing the implications of these high-impact decisions. But policy makers need rigorous analysis to gauge the effect of policies that can have enormous consequences for so many.

Published: June 16, 2022. doi:10.1001/jamahealthforum.2022.2427

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2022 Baicker K. JAMA Health Forum.

Corresponding Author: Katherine Baicker, PhD, University of Chicago, Harris School of Public Policy, 1307 E 60th St, Chicago, IL 60637 (kbaicker@uchicago.edu).

Conflict of Interest Disclosures: Dr Baicker reported receiving grants from the National Institute on Aging and serving on the board of directors for Eli Lilly and Mayo Clinic.