Observational studies are emerging as fundamental sources of information about vaccine effectiveness outside the controlled environment of randomized trials, and they are being used to generate evidence of effectiveness against outcomes that are underpowered in trials, such as hospitalization or intensive care unit (ICU) admission, or for narrow subgroups.1 These studies can monitor the waning of vaccine effectiveness or measure the performance of vaccines against novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) variants when large randomized, controlled trials are not feasible.2

Thompson et al.3 now describe in the Journal the application of a retrospective test-negative design to estimate coronavirus disease 2019 (Covid-19) vaccine effectiveness in adults 50 years of age or older. A multisite network contributed data on 41,552 admissions to 187 hospitals and 21,522 visits to 221 emergency departments or urgent care clinics. These data were derived from patients who had accessed medical care for Covid-19–like illness and had had molecular testing for SARS-CoV-2. In the test-negative design, the case patients were those who tested positive for SARS-CoV-2, and the control patients were those who tested negative. Vaccine effectiveness was estimated by comparing the odds of vaccination between cases and controls. Table 1 shows how data from such studies may be used to calculate vaccine effectiveness.

The test-negative design may be used to estimate vaccine effectiveness against medically attended, laboratory-confirmed SARS-CoV-2 infection among patients who would seek — and have access to — medical care.4 Thompson et al. estimated effectiveness with respect to three distinct outcomes: an emergency department or urgent care visit, hospitalization, and admission to an ICU. Each effectiveness measure reflected the combined benefit of vaccines to prevent infection with SARS-CoV-2 and reduce subsequent progression to medically attended disease.

The test-negative design has been routinely used to estimate vaccine effectiveness against seasonal influenza,5 but its application in studies of Covid-19, although increasingly common, is new. Readers are likely to wonder how to interpret critically the effectiveness estimates resulting from such a design. We identified four important points to consider.

First, are there unmeasured differences between vaccinated and unvaccinated persons that may influence the occurrence of Covid-19? Confounding by both measured and unmeasured variables is a concern in all observational studies. In this context, the confounders are variables that influence both the receipt of vaccine and the occurrence of medically attended Covid-19; these variables include exposure to the virus, the risk of severe disease associated with infection, and access to or uptake of care. Thompson et al. used case weighting6 and logistic regression to adjust for several measured confounders, including demographic and clinical variables and calendar time. A key feature of the test-negative design is that restriction to a population with access to and uptake of medical care reduces unmeasured confounding due to health care–seeking behavior, whereby persons who are more likely to be vaccinated are more likely to seek care when ill.

Second, are cases and controls sampled without bias? An expected consequence of this design is that case patients and control patients will enter the study with similar disease manifestations. In a typical prospective test-negative design, study inclusion is decided before the test result is obtained,7 so that selection bias associated with knowledge of the infection status is avoided. Designs with retrospective ascertainment of infection status (such as that of the study by Thompson et al.) are prone to selection bias if, for example, patients who are highly motivated to be tested and vaccinated also are more likely to access health care services than those who are not highly motivated. In this instance, vaccine effectiveness could be underestimated because vaccinated persons with positive SARS-CoV-2 test results would be overrepresented.

Third, is the patient’s SARS-CoV-2 infection status or vaccination status misclassified? Such misclassification is another potential source of bias. The direction of bias depends on the underlying relationships among reasons for test misclassification, vaccination status, and timing of tests.7,8 Thompson et al. address the timing issue by broadening the testing period to include tests to detect infection that occurred within 14 days before to less than 72 hours after a hospital admission or an emergency department or urgent care clinic visit. They investigated the potential effect of test misclassification by simulating and analyzing synthetic cohorts (described in Section S4 in the Supplementary Appendix, available with the full text of the article at NEJM.org) and found that misclassification bias would cause underestimation of vaccine effectiveness in the main analyses.

Finally, are the results generalizable to populations that have different access to medical care or different health care–seeking behaviors? Studies with test-negative designs are restricted to the inclusion of persons who access health care services. Although generalizability beyond that population cannot be assessed with the study data alone, severe medical outcomes (e.g., hospitalization and ICU admission) are considered to be less sensitive to differences in care seeking.8 To the extent that health care–seeking behaviors, thresholds for admission, and general accessibility vary across sites, the consistent effectiveness of full vaccination across different network sites in the study conducted by Thompson et al. suggests a substantial effect. It is nevertheless essential to consider that unemployed persons, those who have limited insurance, and undocumented workers will have higher thresholds for seeking health care and will generally be at higher risk for serious illness than other persons, regardless of vaccination status; this limits the generalizability of the findings to the disadvantaged groups who are not represented in this study.

Owing to their applicability to large electronic health records and their logistic simplicity relative to large prospective cohorts, test-negative designs can be expected to play an important role in monitoring the effectiveness of Covid-19 vaccines in the United States and elsewhere. Methods to analyze data from studies with test-negative designs are the focus of considerable ongoing research.5,7-10 A clear understanding of the assumptions underlying the design, the reasons for using it in practice, and its relative strengths and limitations is essential for readers to critically assess, interpret, and apply the findings in a principled fashion. Researchers who use test-negative designs to investigate Covid-19 vaccine effectiveness can look to the article by Thompson et al. for examples of how to report primary findings and assess the sensitivity of these findings to potential biases that are specific to the test-negative design.