We Could Know Soon How Effective the Potential COVID Vaccines Are

When will a COVID-19 vaccine be approved? That is the foremost question globally as the COVID-19 pandemic—entering its 10th month—shows no sign of easing on its own. To answer that question, we first need to know when sufficient evidence of the safety and effectiveness of the candidate vaccines will be available. The good news is that convincing signals of efficacy (or, alternatively, ineffectiveness) may come relatively soon.

COVID-19 vaccine efficacy assessments will be based on criteria released by the U.S. Food and Drug Administration (FDA) in June and the World Health Organization (WHO) in May. The FDA’s review will govern which vaccines are approved for use in the U.S.; the WHO’s guidance will inform the regulatory processes in many other countries around the world.

There are currently five vaccines targeted by the federal government’s Operation Warp Speed (OWS) program in, or nearing, Phase III clinical trials (see Figure 1). This the final step in the regulatory approval process and involves administering the vaccine, or a placebo, to tens of thousands of trial volunteers.

President Trump had signaled that he wanted one or more of the vaccines approved before the election in November. That has been ruled out by recent supplemental FDA guidance on the specific criteria needed to justify an emergency use authorization (EUA). To provide more time for potential adverse events to emerge, FDA is telling the sponsors of the vaccine candidates that it will not consider EUA applications until two months after half the trial participants have received the last dose of the candidate vaccine. In all cases, this will push EUA reviews until after November 3.

Slowing the review to improve confidence in the safety of potential vaccines does not mean we will have to wait that long to have an idea of how effective they are in slowing the pandemic. Indeed, it is possible that the vaccine sponsors with trials underway will know soon—perhaps within weeks—if their candidates work in preventing people exposed to the virus from developing COVID-19.

Setting the Bar for a Minimum Level of Efficacy

Phase III trials for COVID-19 vaccines must follow the protocols of a high-quality randomized and blinded test. Volunteers will be divided into two basic groups: those getting the vaccine and those getting a placebo. Neither the test subjects nor those administering the vaccines will know which volunteers are in which group. The process of assignment will be random to ensure a sound basis for statistical analysis.

The trials are run to see if the candidate vaccines are safe enough to administer to millions, and potentially billions, of people, and to see how well they work in preventing people from getting sick from the virus. Vaccine efficacy will be stated as a percentage reduction in COVID-19 cases among those receiving the vaccine versus those receiving the placebo. For instance, if during the course of a Phase III trial, 100 cases of COVID-19 emerge among those who received the placebo, and just 20 among those receiving the candidate vaccine, the estimated efficacy of that vaccine would be 80 percent.

The true efficacy rate of the vaccines—that is, how well they will end up working when administered to millions of people—is not known in advance of the trials. Because the trials are based on outcomes from a limited number of volunteer participants, they offer only uncertain estimates of true efficacy. However, using statistical probabilities, the trials will produce ranges of expected vaccine efficacy within which it is highly likely that the vaccines’ true efficacy rates will fall.

While vaccine trials require tens of thousands of participants, efficacy determinations will be based on a relatively small share of the overall trial enrollment. The crucial variable is the number of volunteers who develop COVID-19. For many diseases for which vaccines are desirable, the likelihood of becoming ill is low—perhaps 1 percent or less. This is true even during the current COVID-19 pandemic. The FDA’s goal in slowing the review of emergency approval applications is about increasing confidence in the safety of the vaccines, but we may know more quickly about efficacy because of the likelihood that sufficient numbers of people will develop symptoms of the disease before the established waiting period for safety reviews has been satisfied.

A statistical test determines whether there is sufficient evidence of vaccine efficacy. If the number of COVID-19 cases among those who received the vaccine is substantially lower than the number of cases among those who received the placebo, then the vaccine will be deemed efficacious.

Both the FDA and the WHO stipulate that vaccine efficacy of 50 percent, as observed in Phase III trials, is the minimum required for regulatory approval. In addition, based on the observed trial data, there should be no more than a small chance that the vaccine’s actual, or true, efficacy (when administered to millions of people) is below a 30 percent reduction in disease incidence.

The WHO proposes that the trial’s outcome should imply no more than a 2.5 percent chance that a vaccine with true efficacy of less than 30 percent will be approved. In other words, the 95 percent confidence interval for the true efficacy rate must exclude 30 percent.

The Timing of Efficacy Signals

There are four main factors that will drive the timing of COVID-19 vaccine efficacy determinations.

• Attack rate. This is the frequency with which non-vaccinated individuals become sick based on the presence of the virus in a community. Higher attack rates mean that participants contract COVID-19 more quickly, and thus decrease the time required to observe a given number of confirmed cases.

• Sample size. More participants in a trial mean that, for a given disease attack rate and fixed amount of time, there will be more observed cases of disease. Thus, the trial might be able to conclude sooner, send a clearer signal of vaccine efficacy, or both. The trials now underway are based on substantially different enrollment numbers. Pfizer is aiming for 44,000 volunteers in its trial, and Johnson and Johnson is shooting for 60,000. Moderna’s trial, which is now fully enrolled, is based on a sample size of 30,000 (15,000 receiving the vaccine, and 15,000 the placebo).

• Number of COVID-19 cases. There is a trade-off between establishing sufficient confidence in the efficacy rate of candidate vaccines and the speed with which the required data for making a decision will accumulate. Higher numbers of observed cases of COVID-19 translate into higher statistical confidence that the observed difference between cases in the vaccine and placebo arms is a reflection of the true efficacy of the vaccine. For instance, assessments that take place with only 50 observed cases will have wider confidence intervals than those with 100 observed cases, and thus more uncertainty about how effective the vaccine would be when widely deployed. However, it will take less time to accumulate 50 positive cases of COVID-19 than 100, and that matters in a pandemic. The sponsors of the vaccines can specify in their trial protocols the case count intervals they will use to look at efficacy rates. However, it is the FDA that will make final decisions on regulatory approval, and it seems clear that the agency is leaning toward higher levels of confidence as opposed to speed.

• True vaccine efficacy. The true vaccine efficacy rate interacts with the sample size, required number of positive cases, and attack rate to determine how long the trial lasts and the probability that the trial succeeds. Unlike the sample size, which is fixed as a part of the trial protocol, and the attack rate, which can be estimated from existing data on disease rates, the true efficacy rate is unknown to the trial sponsor. But beliefs about vaccine efficacy are a major consideration when determining how many positive cases must be observed before evaluating the vaccine’s effectiveness. For instance, vaccines that are believed to be very effective will require fewer positive cases to achieve sufficient confidence that the Phase III trial demonstrates efficacy because the sponsors believe there is only a small chance that the observed data will lead to the disapproval of an effective vaccine. Vaccines with efficacy rates that are believed to be only marginally greater than 50 percent, in contrast, risk unlucky outcomes—that is a trial that, by chance, has observed efficacy under 50 percent—if the number of cases used to trigger an efficacy review is too few.

Modeling Phase III Efficacy Trials

Mathematical simulations of Phase III trial efficacy assessments, using statistical modeling software, can provide useful insights into the interactions of the factors affecting the trial’s length, the required observed efficacy rate, and the probability that a candidate vaccine satisfies the efficacy criteria.

Figure 2 reports the confidence intervals produced in such simulations when the data are assessed after various levels of confirmed cases and with varying rates of observed vaccine efficacy. (The confidence intervals depicted in red would fail the test, while those in green would pass it.)

Confidence intervals show the lower and upper bounds within which the expected efficacy of the vaccines would fall. For instance, as shown, if a candidate vaccine has an observed efficacy of 70 percent and the trial concludes after 50 positive cases, there would be 95 percent confidence that the vaccine has true efficacy of between 44 percent and 86 percent. Because the lower bound of the confidence interval exceeds 30 percent, it would satisfy the WHO criteria. In contrast, if a vaccine has an observed efficacy rate of 60 percent, 50 cases would not be enough to have high confidence that its actual efficacy was above 30 percent (the lower bound in the chart is 28 percent, below). Finally, for a vaccine with a 50 percent observed efficacy rate, 150 observed cases are sufficient to have 97.5 percent confidence that the true efficacy rate exceeds 30 percent. These results indicate that, so long as the trial concludes after 150 confirmed cases and the observed efficacy rate exceeds 50 percent, both statistical criteria will be satisfied. Finally, an observed efficacy rate of 40 percent will always fail the statistical test, due to the first statistical criterion, which requires that the trial results imply a true efficacy rate that is greater than 50 percent.

This analysis of the confidence intervals associated with various trial results demonstrates that the power of Phase III trials to identify effective versus ineffective vaccines increases with the number of confirmed COVID-19 cases emerging during the test period.

Additional analyses can provide detail on the trade-off between the power of the trial to identify effective vaccines and the number of observed cases before evaluating the candidate vaccine’s efficacy. At 150 confirmed cases, a vaccine with a true efficacy rate of 60 percent has a 9 in 10 chance of satisfying the statistical criteria. At only 100 cases, the probability of exceeding the required thresholds declines to 3 in 4. At just 50 cases, the probability is closer to 1 in 2.

Early Signals

One way to balance the trade-off between the quality of the signal and the duration of the trial is to evaluate vaccine efficacy at a series of intermediate case counts. The WHO, for instance, recommends that the trial be evaluated after 50, 100, and 150 confirmed cases, with required vaccine efficacy thresholds of 76 percent, 59 percent, and 50 percent, respectively, at each stage.

The progressively weaker efficacy requirements across the three evaluation stages are consistent with the desire to identify an effective vaccine as quickly as possible while retaining the power of the statistical test. If, instead, we applied the same criteria at each intermediate stage as the criteria that we apply to the final analysis, we would approve vaccines with efficacy rates lower than 50 percent too frequently because each evaluation stage would offer another opportunity for an insufficiently effective vaccine to satisfy the statistical criteria by random chance.

Figure 3 provides a summary of the intermediate evaluation points selected by the sponsors of the four ongoing Phase III trials of vaccine candidates in the OWS portfolio. (Novavax’s trial has yet to commence.) As shown, Moderna’s Phase III trial includes intermediate evaluations at 53 and 106 cases, before a final evaluation at 151 cases, and requires an observed efficacy rate of 74 percent to demonstrate efficacy after 53 cases (Moderna 2020). Pfizer’s trial includes four intermediate evaluation stages—the first after just 32 confirmed cases of COVID-19 among trial participants—and a final evaluation at 164 cases. To demonstrate efficacy after 32 cases, however, Pfizer’s observed efficacy rate would need to exceed 77 percent (Pfizer 2020). This methodology ensures that only highly effective vaccines satisfy the regulatory criteria after just 50 positive cases, whereas vaccines with efficacy rates closer to 50 percent are more likely to require at least 150 observed cases before receiving approval.

The effect of permitting multistage evaluation on the expected duration of the trial, however, depends largely on the vaccine’s true effectiveness. A trial for a vaccine with a true efficacy rate of 55 percent that evaluates the vaccine at 50, 100, and 150 positive cases using the WHO-recommended criteria will, on average, require 130 positive cases. More than 6 times in 10, the trial would continue to its 150-case conclusion.

Summing Up

Multiple COVID-19 vaccine candidates have shown promise in early-stage clinical trials and now have entered into the final step of the regulatory approval process. Because of the urgency of the moment, there is a worry that governments may be too quick to approve vaccines before they have fully demonstrated their worth. That is certainly an argument for not cutting corners on safety evaluations.

In terms of efficacy, however, valid statistical determinations likely can be made rather quickly, taking just a few months once a trial is at full enrollment rather than half a year or longer. That is due mainly to the prevalence of COVID-19, which increases the likelihood that trial participants will be exposed to the virus in their communities. If a candidate vaccine is truly effective at halting progression to COVID-19, that fact will be known soon enough by observing the presence of the disease primarily among those receiving the placebo in the on-going Phase III trials.

James C. Capretta is a resident fellow and holds the Milton Friedman Chair at the American Enterprise Institute. Scott Ganz is a research fellow in economic policy studies at AEI and an assistant professor at Georgia Tech’s School of Public Policy. They are the authors of “Awaiting the Signal: Assessing the Efficacy of COVID-19 Vaccines,” published by AEI.

Photograph by Chandan Khanna/AFP/Getty Images.