The Price of Success—How to Evaluate COVID-19 Vaccines When They’re Available Outside of Clinical Trials | Vaccination | JAMA | JAMA Network
[Skip to Navigation]
Sign In
Medical News & Perspectives
February 18, 2021

The Price of Success—How to Evaluate COVID-19 Vaccines When They’re Available Outside of Clinical Trials

JAMA. 2021;325(10):918-921. doi:10.1001/jama.2021.0641

In science, as in other fields of endeavor, success can have unexpected consequences.

Health care workers at the University of Miami Miller School of Medicine verify which participants receive vaccine or placebo during a phase 3 coronavirus disease 2019 vaccine clinical trial being conducted there in September 2020.

Taimy Alvarez/AP Images

Take the debut of 2 highly efficacious coronavirus disease 2019 (COVID-19) vaccines more quickly than anticipated.

The situation has spurred debate about the ethics, let alone the feasibility, of continuing or launching blinded, placebo-controlled trials—generally considered the gold standard of medical research—of investigational COVID-19 vaccines.

“This is unprecedented and took almost everybody by surprise,” bioethicist David Wendler, PhD, of the National Institutes of Health Clinical Center, said in an interview.

In regularly consulting with Operation Warp Speed—the federal government’s COVID-19 vaccine accelerator—Wendler said, he couldn’t recall anyone suggesting that 1 vaccine, let alone 2, would demonstrate 95% efficacy against symptomatic disease by December 2020.

“People don’t know what to do about it,” he said.

Trial participants are hearing mixed messages, Wendler said. Public health leaders are urging the public to get vaccinated, but researchers are suggesting that trial participants remain blinded and forgo vaccination for at least a few more months, he said.

Phase 3 trials of the 2 vaccines demonstrated efficacy at warp speed because US COVID-19 cases surged in the fall. “These are case-driven trials,” Greg Glenn, MD, president of research and development at Maryland-based Novavax, which began enrolling patients in the US and Mexico in its phase 3 trial in late December, explained in an interview.

In other words, as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) circulated more widely, trial participants were more likely to be exposed to it. So COVID-19 cases among those who received placebo shots accumulated rapidly, enabling trials to show a clear efficacy difference between the control and active immunization groups.

For example, COVID-19’s grip on the US enabled Moderna to conduct its primary analysis 5 months sooner than expected, Senior Vice President Jacqueline Miller, MD, said at a December US Food and Drug Administration (FDA) advisory committee meeting. Based on the committee’s advice, the FDA on the next day authorized emergency use of the vaccine, developed in partnership with Operation Warp Speed and the National Institute of Allergy and Infectious Diseases (NIAID). Pfizer-BioNTech’s COVID-19 vaccine, reviewed by an FDA advisory committee a week before Moderna’s was, also quickly received Emergency Use Authorization (EUA).

But the speed with which 2 COVID-19 vaccines received EUAs throws a wrench into ongoing and future placebo-controlled vaccine trials: Why would anyone want to remain or enroll in such studies when they could get vaccinated outside of them?

“It’s a nice problem to have,” Dean Follmann, PhD, chief of the NIAID’s Biostatistics Research Branch and a member of the Moderna trial’s protocol team, noted in an interview.

The FDA invited Stanford Medicine’s Steven Goodman, MD, PhD, whose own research has focused on the proper measurement, conceptualization, and synthesis of evidence, to discuss the dilemma at the advisory committee meetings for both the Pfizer-BioNTech and Moderna vaccines.

“The issues we’ll be considering here are not just relevant for this vaccine but for many clinical trials going on [now] and in the future,” Goodman told the panel at the December 17, 2020, Moderna meeting.

“That Death Weighs Very Heavily”

By the day of the advisory committee meeting, 1 of the 185 placebo recipients in Moderna’s phase 3 trial who developed symptomatic, laboratory-confirmed COVID-19 had died.

“That death weighs very heavily on me,” Miller said at the meeting (neither Moderna nor Pfizer would provide representatives to comment for this story). “Additional severe cases and death [among placebo recipients] is not a question of ‘if’ but of ‘when,’” she continued. “That also weighs heavily on me.”

The Moderna representatives told the advisory panel that they had thousands of extra doses set aside for the trial that would expire in late January. Instead of letting them go to waste, the company planned to offer leftover doses to every trial participant who’d received placebo shots. But in most states, only health care workers, nursing home residents and staff, and people aged 65 years or older were eligible for vaccine by late January. That meant young, healthy trial participants who weren’t frontline health care workers would be jumping the queue.

That’s not to say participants in phase 3 trials enrolled so they could be first in line for vaccination, the FDA’s Philip Krause, MD, said in an interview. “Placebo-controlled trials in general do depend to some degree on the altruism of the people in the study.”

When participants in Moderna’s phase 3 trial originally consented, they were told, “You will not get the vaccine when this is over,” Goodman said in an interview. However, Moderna’s switch to unblinding and vaccinating all takers makes it seem that personal gain is the main reason to enroll in a COVID-19 vaccine trial, he said.

Daniel Freedman, DO, a 35-year-old pediatric neurologist at UT Health Austin, participated in Haemophilus influenzae type B and hepatitis A vaccine trials as an infant. “I’m a huge vaccine advocate, so I kind of jumped at the opportunity” to enroll in the Pfizer-BioNTech phase 3 vaccine trial, Freedman, a pediatrician’s son, said in an interview.

After receiving his shots in August and September, Freedman, like some trial participants, suspected they were the real deal because of brief muscle aches and chills afterward. He sought to verify his suspicion only because his hospital was beginning to vaccinate health care workers. While he did indeed get the vaccine in the trial, Freedman said, he’ll continue returning for scheduled follow-up visits. He doesn’t understand why trial participants further down the line for vaccination outside the trial can’t wait to be unblinded until the FDA licenses the vaccine. “I want the studies to be done properly,” he said.

Pfizer representatives told the FDA advisory panel that they planned to unblind trial participants and offer vaccine to the placebo group when it was their turn outside the trial. But the next day, after the vaccine received an EUA, Pfizer told participants that it had set aside vaccine for trial participants who had received placebo shots and planned to offer the first dose of vaccine to all of them by March 1, which means that some will likely be immunized weeks or even months before their peers in the community.

“People can make the argument: I volunteered to be in the trial…I should be able to get the vaccine right away even if I’m young and healthy,” Susan Ellenberg, PhD, a University of Pennsylvania professor of biostatistics, medical ethics, and health policy, said in an interview. “I think that’s a reasonable argument.”

At a December 23, 2020, press briefing, Moncref Slaoui, PhD, MBA, Operation Warp Speed chief scientific advisor in the Trump administration and currently a consultant to the accelerator, called Pfizer’s original plan to vaccinate placebo participants when it was their turn outside the trial “intellectually very elegant and appropriate.” However, he noted, “it’s a very difficult one,” given that participants would become eligible for vaccination in the community at varying times across different states.

“I think it’s very important that we unblind the trial at once and offer the placebo group vaccines,” Slaoui said at the briefing. “They should be rewarded” for trial participation, he said. (Former FDA Commissioner David Kessler, MD, Slaoui’s successor, did not respond to an interview request.)

The protocol for the phase 3 trial of Janssen Biotech’s vaccine, which has enrolled approximately 45 000 people in the US, Latin America, and South Africa, “allows for unblinding of individuals who are in a priority group for receiving an authorized vaccine,” a spokesman for the Johnson & Johnson subsidiary said in an email. “If they are unblinded, and we learn they received placebo, they will be advised to receive the authorized vaccine.”

The Janssen vaccine uses a genetically modified adenovirus to deliver the genetic code of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) spike protein to induce an immune response. The day after Novavax released its preliminary results, Janssen announced that preliminary results showed its single-dose vaccine to be 66% effective in protecting against moderate to severe COVID-19 symptoms. The FDA scheduled a February 26 advisory committee meeting to discuss Janssen’s request for an EUA for its vaccine.

“Blowing Up” Trials

Some argue that there is no ethical imperative to reward vaccine trial participants, especially considering how difficult it has been for even high-risk individuals to get vaccinated outside of trials. Instead, they argue, there is an ethical imperative to continue blinded, placebo-controlled trials to gather long-term safety and efficacy data.

By unblinding trial participants, “you lose a valid comparison group,” Goodman said. “There will be this sense, and it will be sort of true, that the study is over.”

Unlike, say, a highly effective cancer drug, “the vaccine is not literally a life-and-death issue today and tomorrow” for most trial participants, Goodman said. So, he noted, those running COVID-19 vaccine trials shouldn’t feel obligated to unblind participants and vaccinate placebo recipients right away.

Doing so implies “you can just blow up the trial” on the basis of promising preliminary results, establishing “an ethical model for future trials that we maybe don’t want to set,” Goodman said.

Follmann said his “greatest fear” about immediately unblinding participants is that those who received the vaccine will take greater risks—forgoing masks and social distancing—than those who received the placebo. Differential behavior between the 2 groups could make a vaccine appear less effective than it really is.

“Another concern is that mild subjective symptoms might be differentially dismissed/elevated in the unblinded arms,” Follmann and his coauthors wrote in a December 14, 2020, article that has not yet been peer-reviewed. The article’s coauthors included Lindsey Baden, MD, a Brigham and Women’s Hospital physician who is a principal investigator for the Moderna vaccine’s phase 3 trial, as well as scientists from Moderna, AstraZeneca, Janssen Pharmaceuticals, Novavax, and Merck, all of which were working on COVID-19 vaccines (subsequently, Merck canceled its COVID-19 vaccine program).

“I just worry [unblinding] will sort of wreck the study,” Follmann said. “We do blinded studies for a reason.” On the other hand, those conducting the phase 3 trials say they worry that large numbers of participants would be lost to follow-up if not unblinded and vaccinated, because they’d simply drop out.

What Could Be Lost

In late November 2020, the International Coalition of Medicines Regulatory Authorities (ICMRA), of which the FDA is a member, issued a statement supporting the continuation of randomized, controlled COVID-19 vaccine trials.

“Unless maintaining participants in their randomised treatment groups (vaccinated or control) after a vaccine is approved is clearly infeasible, we recommend that clinical trials should proceed as initially planned with a follow-up of at least one year or more from completion of assigned doses,” the ICMRA stated.

Although vaccines might demonstrate high short-term efficacy, only longer placebo-controlled trials can fill in the remaining information gaps, the World Health Organization’s (WHO’s) Ad Hoc Expert Group on the Next Steps for COVID-19 Vaccine Evaluation noted in a December 2 article entitled “Placebo-Controlled Trials of COVID-19 Vaccines—Why We Still Need Them.” Those gaps include a more comprehensive assessment of short-term safety, questions about whether waning vaccine-induced protection could lead to vaccine-enhanced disease in infected vaccinees, information on protection against severe COVID-19, and possible associations between age or comorbidities and level of protection.

Coauthors of the international WHO expert group’s article included Krause, deputy director of the Office of Vaccines Research and Review at the FDA’s Center for Biologics Evaluation and Research (CBER), and Ellenberg, who worked at CBER for 12 years.

The Pfizer-BioNTech goal to give every placebo participant who wants it their first vaccine dose by March 1 “strikes a reasonable balance between the likelihood that people will seek vaccination outside the trial and the likely timing that vaccine will become available,” Krause said. Both Pfizer-BioNTech and Moderna phase 3 trials began enrolling participants in late July, so some would have 6 months of follow-up by March 1, he noted.

Given that such trials have considerable social value, “it can be ethically acceptable to continue a placebo-controlled trial for a short period after the vaccine candidate has been found to be safe and efficacious, even when participants might be able to access the vaccine candidate outside the trial…,” Wendler and his coauthors suggested in a December 11 article.

In a December 22 letter, Representative Lloyd Doggett, a Texas Democrat who chairs the House Ways and Means Committee’s Subcommittee on Health, urged FDA officials to require the continuation of double-blinded randomized clinical trials as a condition for licensure—the next regulatory step—of COVID-19 vaccines that have received EUAs.

“The continuation of clinical trials is critical to our understanding of the efficacy and length of immunity the vaccines offer,” Doggett wrote to then-Commissioner Stephen Hahn, MD, and CBER Director Peter Marks, MD, PhD. “At a minimum…participants should only receive access to the vaccine when they would have otherwise received it based on [Centers for Disease Control and Prevention] and state health guidelines.”

End of Placebo-Controlled Trials?

Now that at least 2 vaccines have received EUAs, will people enroll in placebo-controlled trials of other vaccine candidates?

After all, “As far as new vaccines, there is no doubt that placebo-controlled trials are the most valuable ways to evaluate them,” Krause said.

As of February 10, 20 candidates were being tested in large-scale efficacy—or phase 3—trials, according to the New York Times’ global COVID-19 vaccine tracker. Nearly twice as many were in much smaller phase 1 trials.

One way to attract trial participants is by upping the odds that they’ll receive the investigational vaccine, not the placebo. While Pfizer-BioNTech and Moderna trial participants had a 50-50 chance of getting the vaccine, those in Novavax’s phase 3 trial have 2 to 1 odds of getting the investigational vaccine.

Unlike the Pfizer-BioNTech and Moderna vaccines, which contain genetic strands known as messenger RNA (mRNA) and require very low temperatures for long-term storage, the Operation Warp Speed–supported Novavax vaccine contains a spike protein made using recombinant nanoparticle technology and, according to the manufacturer, can be stored at least 6 months in a standard refrigerator.

“They’re beating down the door” to enroll, Glenn said in early January. “I think you’re going to see us enroll quickly and very well.” The company already completed a 15 000-person phase 3 trial in the UK and a phase 2b trial with more than 4400 participants in South Africa. On January 28, the company announced that interim results showed efficacy against symptomatic COVID-19 was 89.3% in the UK and 60% in South Africa, (preliminary sequencing showed that 92% of the COVID-19 cases in the South Africa trial were caused by virus with the B.1.351 variant that is prevalent there).

Shortly after Glenn spoke to JAMA, the Washington Post reported that people aged 65 years or older in the US were dropping out of Novavax’s phase 3 trial so they could get the Pfizer-BioNTech or Moderna vaccine. Novavax, responding to the story in a prepared statement, said 20% of the 11 000 trial participants enrolled by that point were older than 65 years, and 2% had asked to be unblinded. When fully enrolled with 30 000 participants, the trial expects that 25% of them will be at least 65 years old, according to the statement.

John Zaia, MD, director of the Center for Gene Therapy at the City of Hope, is enrolling 117 participants in a phase 1 COVID-19 vaccine trial, planned to last a year. The vaccine uses a synthetic modified vaccinia Ankara (MVA) platform developed by City of Hope researchers. MVA is a highly attenuated poxvirus vector used in vaccines against smallpox and other infectious diseases. While Pfizer-BioNTech, Moderna, and Novavax have tested 2-dose vaccines, the City of Hope is testing a single dose as well as 2 doses of its vaccine, which is being developed in a freeze-dried form that won’t require storage at low temperatures.

Only 1 of 8 phase 1 trial participants is being randomized to receive a placebo, with the rest getting at least 1 dose of the investigational vaccine. Zaia said he discourages prospective participants from enrolling if they think they’re soon going to get vaccinated outside the trial, which has an upper age limit of 54 years. “You have to make the decision now,” he tells them.

Alternative Approaches

Although placebo-controlled trials are considered the gold standard, scientists recognize that eventually they may not be possible. “This is not really an ethical question,” Krause said. “It’s more of a practical question.”

One thing that’s clear, he said, is “the world is going to need more [COVID-19] vaccines, and there’s going to be a need to study those vaccines.”

Goodman and others at the FDA advisory committee meetings talked about an alternative to unblinding trial participants and vaccinating the placebo group. In a blinded crossover design, everyone would return for 2 more shots when it’s their turn to be immunized outside the trial. The placebo group would get the vaccine, while the vaccine group would receive saline shots.

“The logistics are something that are not trivial,” Bill Gruber, MD, Pfizer’s senior vice president of vaccine clinical research and development, told the FDA advisory committee.

Follmann described the blinded crossover approach in the recent article he coauthored with pharmaceutical company scientists. Instead of comparing the placebo group to the vaccinated group, researchers would compare the group vaccinated first to the group vaccinated second.

“It’s not been done for any other vaccine,” Follmann said of the approach. While it might have been too late for Moderna to switch its phase 3 trial to a blinded crossover, he said, “in the future I think it will be a serious design consideration.” Novavax has been discussing how and whether to do a blinded crossover, Glenn said.

Another alternative to a conventional placebo-controlled trial would be a noninferiority trial, which would randomize participants to an investigational vaccine or a vaccine whose safety and efficacy have already been demonstrated.

Thomas Lingelbach, chief executive officer of Valneva, an international pharmaceutical company based in France, told Bloomberg News on February 5 that his company was discussing with regulators whether participants in the control arm of its UK late-stage vaccine trial, which aims to enroll about 4000 people, could be given an authorized COVID-19 vaccine instead of a placebo. Now that several COVID-19 vaccines have shown efficacy, it wouldn’t be ethical to give Valneva’s control group a placebo, he said.

However, by injecting all participants with an active vaccine, noninferiority trials need to be about triple the size or length of placebo-controlled trials to accrue enough COVID-19 cases, Krause noted. Such trials “certainly can be performed,” he said, but the problem is they can only identify vaccines with comparable efficacy.

When the active comparator vaccine’s efficacy level is 90% or more, as with the mRNA vaccines, noninferiority trials might not be powered sufficiently enough to confirm as worthwhile an experimental vaccine having a 60% to 70% efficacy level, Krause and his coauthors noted in a recent article. And yet, a vaccine with a 60% to 70% efficacy level, especially one that could be easily stored and administered in a single dose, would be a valuable addition to the COVID-19 vaccine arsenal, Krause said.

Another option would be to conduct placebo-controlled trials of new vaccines in countries that have none. Although concerns were raised in the past about the ethics of HIV/AIDS research conducted by high-income countries in low-income countries, Goodman said the situation with COVID-19 vaccines is different. People in the US were the first to sign up for clinical trials, he explained, and individuals living in countries that don’t have access to a vaccine might be eager to enroll in a trial in which they have at least a 50% chance of getting one.

Instead of comparing COVID-19 cases in placebo recipients vs vaccine recipients to determine efficacy and durability—a task that will be increasingly daunting as more individuals are immunized with authorized vaccines and cases drop—researchers could look for correlates of immunity, once those are determined. “If they know what immune responses are associated with protection, they won’t have to do these large field trials,” Ellenberg said. “Vaccines have been approved that way for years.”

Meanwhile, Krause is a pragmatist. As COVID-19 vaccines become more widely available and the pandemic eventually wanes, conducting placebo-controlled trials to assess safety and efficacy will become progressively more difficult. Instead, he said, “there’s no doubt that in the long run, we’re going to have to rely increasingly on observational data.”

Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words