Real-World Effectiveness and Noninferiority Evaluation and Comparison of Messenger RNA–Based and Protein-Based COVID-19 Vaccines: Protocol for the BEEHIVE Randomized Study With a Hybrid Effectiveness Design

Introduction

Background

Early in the COVID-19 pandemic, there was vigilant surveillance by regulatory authorities to monitor SARS-CoV-2 cases and assess the impact of COVID-19 vaccines on controlling viral spread. When the public health emergency status was not renewed in May 2023, surveillance mechanisms greatly diminished or stopped entirely [1]. With the continued evolution of immune-evasive variants, COVID-19 remains a public health concern [1]. Maintaining surveillance of the vaccine effectiveness (VE) of updated COVID-19 vaccines targeting new strains is informative for ongoing vaccination campaigns and for the development of new vaccine formulations [2].

There are 3 COVID-19 vaccines approved for use by the US Food and Drug Administration (FDA) [3-5]. Of these, 2 are messenger RNA (mRNA)–based vaccines (Pfizer–BioNTech COVID-19 vaccine and Moderna COVID-19 vaccine) and 1 is an adjuvanted recombinant spike protein–based vaccine (Novavax COVID-19 vaccine). A key difference among these formulations is the use of mRNA encoding the SARS-CoV-2 spike protein in the Pfizer–BioNTech and Moderna vaccines [3,4], compared with the use of recombinant spike protein with Matrix-M adjuvant in the protein-based Novavax vaccine [5-7]. Prototype monovalent COVID-19 vaccines of each of these formulations, targeting the ancestral Wuhan strain, have undergone a series of updates since their initial authorization (ie, bivalent Wuhan and BA.4/5, monovalent XBB.1.5, and monovalent JN.1 lineage) to target circulating lineages of relevance at the time [8].

There are limited clinical and real-world data on the VE of XBB.1.5–based vaccines [9,10]. Notably, existing reports focus on assessments of symptomatic infection over a limited period. A better understanding is needed of the protection these vaccine formulations may provide against both symptomatic and asymptomatic infection for several months after vaccination.

Objectives

The Booster Epidemiological Evaluation of Health, Illness and Vaccine Efficacy (BEEHIVE) study (NCT06065176) was developed as a randomized trial with a hybrid design that includes a randomized comparison between active vaccine groups and an observational control group that did not receive a dose of the study vaccine. The goal was to assess the VE of the 2023-2024 formulations (against XBB.1.5) for the Pfizer–BioNTech and Novavax COVID-19 vaccines in a real-world setting [11]. Importantly, this study was designed with a comprehensive surveillance structure to identify SARS-CoV-2 infections, regardless of symptom presence, over a 6-month period following study vaccination. The study protocol and end points are described here, along with the novel recruitment methods and strategies used to maintain participant engagement.

Methods

Study Design

The BEEHIVE study was designed to enroll approximately 1500 participants. Participants were aged ≥18 years; resided in Salt Lake City, Utah (including a 60-mile radius of surrounding areas); had previously received ≥2 doses of an mRNA-based COVID-19 vaccine authorized by the US FDA; and met the eligibility criteria described in Textbox 1.

Study groups comprised 2 randomized, blinded groups who received the 2023-2024 formulation of either the Novavax COVID-19 vaccine or the Pfizer–BioNTech COVID-19 vaccine and a nonrandomized, observational control group of volunteers who participated in the study but did not elect to receive a study vaccine or any 2023-2024 COVID-19 vaccine formulation during the surveillance period (Figure 1A).

‎

Participant Recruitment

Several unique mechanisms were used to facilitate participant recruitment. Radio and television advertisements were deployed in the Salt Lake City area, and flyers for the BEEHIVE study were distributed to patient groups representative of populations that could meet eligibility criteria (Multimedia Appendix 1). Participants from previous COVID-19 studies (ie, RECOVER [Research on the Epidemiology of SARS-CoV-2 in Essential Response Personnel] and PROTECT [Pediatric Research Observing Trends and Exposures in COVID-19 Timelines]) [12,13] or an influenza vaccine study (ie, RAIVEN [Randomized Assessment of Influenza Vaccine Efficacy Network]) [14] who consented to be contacted for future studies were sent recruitment invitations.

Additional outreach efforts included joining occupational health vaccination campaigns and other existing mass-vaccination events, advertising through social media and local news outlets, staffing booths at community events, and deploying mobile vaccination stations. Collaborations were established with local community groups, clinics, businesses, universities, older adult centers, and fire and police departments. Potential participants received a link to the online eligibility survey via email and were informed of their eligibility status upon survey completion. After completion of consent and Health Insurance Portability and Accountability Act (HIPAA) authorization forms, eligible participants were invited to create an online portal account, complete the enrollment survey, and schedule an in-person enrollment visit online (Multimedia Appendix 2).

Vaccination History

For participants who provided consent to having their vaccine records obtained by the study staff, the state immunization information system or registry was accessed to obtain COVID-19 and influenza vaccination status and history. Participants were also asked to self-report their COVID-19 and influenza vaccination history and to provide vaccine documentation to the study.

Study Objectives

There were 2 aims of the BEEHIVE study. The primary aim was to compare VE between study-vaccinated participants and a control group who declined receipt of an updated COVID‑19 vaccine (2023-2024 formula) in preventing symptomatic SARS-CoV-2 infection. The secondary aim was to determine the relative VE (rVE) of the 2023-2024 formulation of the Pfizer–BioNTech mRNA and Novavax COVID-19 vaccines in preventing symptomatic SARS-CoV-2 infection.

Secondary objectives included evaluating the effect of the number of previous COVID-19 vaccinations on the rVE of the Novavax COVID-19 vaccine. Additional secondary objectives were to assess predictors of asymptomatic versus symptomatic SARS-CoV-2 infection; clinical characteristics and outcomes associated with COVID-19, including illness duration and severity; sociodemographic and health characteristics associated with prolonged or severe illness; the impact of COVID-19 on indicators of functioning (eg, missed work, ability to complete normal work and home activities, and working while ill); the proportion of COVID-19 illnesses that were medically attended; and factors associated with seeking medical care and treatment. Illness characteristics and duration for primary infection versus reinfection will be compared, and the effect modification of VE by sociodemographic characteristics, occupation, health status, and other risk factors will be evaluated.

The goals of additional outcomes analyses are to assess whether the 2023-2024 vaccine formulation modifies illness severity and duration among participants with breakthrough infection. Data related to the incidence of post–COVID-19 condition (PCCs) were collected, along with identification of factors associated with PCC symptoms; assessment of whether these symptoms differed between randomized and control study groups; evaluation of whether PCC symptoms were modified by the number of previous vaccine doses; and assessment of the impact of PCC symptoms on overall functioning and mental health. PCCs were defined as any self-reported condition (continuous and relapsing or remitting) occurring after COVID-19 and persisting for at least 1 month after the initial illness, including fatigue, memory problems, dyspnea, sleep problems, joint pain, and/or newly developed medical conditions (eg, myocarditis, heart failure, cardiac dysrhythmias, stroke, chronic kidney diseases, neurologic conditions, and diabetes) [15]. PCCs were grouped by symptom category (eg, cardiovascular, respiratory, and neurological) and by number of symptoms (≥1 or ≥2). Finally, the knowledge, attitudes, and practices (KAPs) of participants related to 2023-2024 vaccines are to be characterized, and associations between KAPs and subsequent vaccination behaviors (including vaccine refusal, hesitancy, or incomplete adherence to vaccination recommendations) will be examined.

Ethical Oversight

Before study implementation, the protocol, informed consent form, participant education and recruitment materials, data collection instruments, and other protocol-related documents were approved by the University of Utah and Westat institutional review boards (I00172738 and 8084.01, respectively). In-person signatures were obtained for informed consent and HIPAA authorization forms. Participants were able to access copies of the consent forms through their portal accounts and received a signed hard copy. All participants provided consent and were allowed to withdraw at any time.

The research data platform and study database were maintained by Westat, Inc. To protect confidentiality, we used assigned identification numbers rather than personal information on study forms. We stored data in locked files and/or secured computers. For information from this study presented publicly or published in a medical journal, individuals will not be identified by name or by any other personally identifiable information. Participants received compensation for study activities (refer to the Remuneration and Engagement section for additional details).

Randomization and Blinding

A computerized random-number generator produced the randomized list for the active vaccinated arms before study enrollment. Randomly permuted blocks (designated A-F) of size 6, with 3 assignments to each vaccine type per block, were generated to achieve 1:1 allocation. Randomization was stratified by sex and age (18-49 years and ≥50 years). Eligible participants who provided consent indicating agreement to receive an updated 2023-2024 vaccine were able to schedule their first study visit and vaccination appointment.

Participants were randomized to either the Novavax COVID-19 vaccine group or the Pfizer–BioNTech COVID-19 vaccine group; however, based on appointment selection and attendance, some participants who consented may not have completed randomization or received a study vaccine. Enrolled participants who provided consent to the study but declined receipt of the 2023-2024 formulation were included in the nonrandomized control group until enrollment was full. Participants who changed their minds at the time of their first study visit were able to switch groups (ie, initially agreed to be randomized to receive a vaccine but then switched to the nonrandomized control group, or vice versa) at the time of their appointment.

Randomized participants, study investigators, and study personnel were blinded to the type of vaccine received during the study, and multiple procedures were performed to maintain blinding. Specifically, vaccine packaging, administration, and documentation practices were standardized to avoid any distinguishable characteristics between the 2 vaccines. Additionally, all study staff, except for licensed health care providers who administered the vaccines, were blinded and received training on the importance of consistent data collection and maintaining blinding. Vaccines were prepared in a designated space that was obscured from participants and study personnel.

Study personnel who administered the vaccines were unblinded and underwent specialized training to ensure that vaccine type was not inadvertently disclosed. These personnel were not involved in data collection or analysis and interacted with the participants only to confirm their identity and administer the vaccine. The Westat randomization coordinator and limited data management staff were also unblinded to facilitate development and maintenance of the randomization list and systems. These staff were not involved in data collection, data analysis, or direct interaction with participants. Participants and study investigators were unblinded at the end of data collection and at the conclusion of the study, respectively. All study-related documents and samples contain a unique identifier for each participant to preserve blinding.

Interventions

Randomized participants were to receive a single dose of the study vaccine (0.3 ml for Pfizer–BioNTech or 0.5 ml for Novavax) administered in the deltoid muscle and were to be monitored for 15 minutes after vaccination for any allergic reactions. An investigational new drug application was submitted to and approved by the US FDA for use of the Novavax COVID-19 vaccine in this study, as it was authorized under an Emergency Use Authorization at the time of the study [5].

Surveillance: Surveys

Surveys were developed based on a pre-existing framework used in prospective clinical trials evaluating influenza and COVID-19 vaccines [13,14,16]. In the enrollment survey, the participants self-reported their sociodemographic characteristics, health status and behaviors, occupational status and employment history, chronic medical conditions, COVID-19 and influenza vaccination history, and previous SARS-CoV-2 infections.

Postvaccination surveys were shared 24 hours, 48 hours, and 6 days after study vaccination to obtain information on potential vaccine side effects. Active surveillance of outcomes occurred for participants on the same day each week, beginning the week following the enrollment and study vaccination visit, and continued for 24 weeks (Figure 1B).

Symptoms were assessed weekly through surveillance and illness messaging surveys for 24 weeks after enrollment and study vaccination. Surveillance included weekly subjective assessment of COVID-19–like illness (CLI), which was defined as experiencing at least 1 of the following symptoms: chills, malaise, fatigue, headache, cough, shortness of breath, sore throat, runny nose or nasal congestion, nausea or vomiting, diarrhea, muscle or body aches, or change in smell or taste.

Follow-up surveys were deployed at designated time points and allowed participants to provide sociodemographic and illness updates, assess KAPs, and collect information on PCCs (Textbox 2).

During the surveillance period, if a participant indicated a CLI symptom on a weekly survey or reported an acute illness between their designated weekly surveillance contacts, they were directed to the Acute Illness Survey (Figure 1B). In this survey, they were asked to identify qualifying symptoms from a detailed checklist, confirm the date of symptom onset, and indicate whether the symptoms were ongoing. If the participant did not confirm symptoms of CLI, they returned to weekly surveillance. If the symptoms were ongoing, participants were prompted to collect a specimen and submit the results according to study procedures. Illness Update and Illness Recovery Surveys were deployed to monitor symptoms until a participant reported ≥90% recovery (Figure 1B). Survey data were collected electronically via a text messaging interface and online surveys (Multimedia Appendix 2).

Surveillance: Testing

All participants received OHC COVID-19 Antigen Self-Test kits (Osang Healthcare) for weekly use, along with instructions on how to use the kit, interpret results, and upload results to the study portal (Multimedia Appendix 1). Assay sensitivity is 91% and relative specificity is 99%, as described by the manufacturing company. Anterior nasal swab specimens were self-collected weekly on the same day (day 0), and a photograph of the results was uploaded regardless of symptom status (Figure 1B).

If a participant reported an acute illness and/or collected an illness specimen on or outside their assigned routine specimen day, they were asked to collect a second specimen on the next day (day 1) and a third specimen 2 days later (day 3). For this study, the interpretation of test results was based on the participant’s symptom status on the first day of testing (day 0) and whether they did or did not have CLI symptoms.

Safety Assessments

Safety assessments included monitoring and recording of solicited (local and systemic reactogenicity events) and unsolicited adverse events (AEs), serious AEs, and AEs of special interest (AESIs). Solicited AEs were collected through the Postvaccination Survey. Unsolicited AEs could be reported at any time from study vaccination through completion of the last study-related procedure.

The relation of each event to the study vaccine (related or not related), whether the event was expected or unexpected, whether the event was serious or not, and the intensity of AEs and serious AEs were investigator determined. Severity was categorized as mild (grade 1), moderate (grade 2), severe (grade 3), life-threatening (grade 4), or death (grade 5). Protocol-defined AESIs were myocarditis and pericarditis and were collected by participant report, with verification through medical record review and follow-up interviews by study clinicians to obtain additional details.

Participants received an informational sheet describing symptoms of myocarditis and pericarditis, a list of nearby medical facilities for care seeking, and a letter for their evaluating physician, which included recommendations for clinical management. Probable and confirmed cases of AESIs were defined as previously described [17]. Safety reporting related to the investigational new drug status of the Novavax COVID-19 vaccine included sharing suspected unexpected serious adverse reactions and relevant medication errors with the appropriate regulatory authorities.

Statistical Analyses

The primary population for VE analyses will be the modified intent-to-treat (mITT) population, which includes all randomized participants who received a study vaccine and responded to ≥1 active surveillance survey, and all participants in the control group. Any participant with a CLI-associated SARS-CoV-2 infection confirmed by COVID-19 rapid antigen test 0 to 6 days after vaccination will be excluded from the mITT population. Analyses will also be performed with the per-protocol population, which includes participants who met eligibility criteria, received a study vaccine per protocol, participated in study surveillance by responding to at least 1 surveillance contact during the SARS-CoV-2 circulation period (active surveillance), and did not receive another SARS-CoV-2 vaccine outside of the study during the study period. If participants were in the nonvaccinated control group but subsequently received a 2023-2024 updated vaccine, they will be censored at that point. Any participants withdrawing early from the study will be censored at that time and included in analyses. There will be no imputation for missing data.

Using publicly available data [18] on COVID-19 cases in Utah from October 1, 2022, to May 28, 2023, the average rate of reported cases in fully vaccinated individuals from fall 2022 through spring 2023 was 39.64 per 100,000 person-days. On the basis of previously reported survey results of clinic and laboratory testing rates for COVID-19 [19] and the use of at-home test kits in this study, the estimated case reporting was 43.1% higher. Using these values and adjustments to the observed rates of infection in Utah in fall 2022 leads to a conservative estimate of the rate of CLI-associated SARS-CoV-2 infection in the control group, confirmed by COVID-19 rapid antigen test, of 80 per 100,000 person-days. With 1500 participants enrolled (1200 randomized and 300 nonrandomized control), assuming ≤15% attrition, there would have been 80% power to detect a VE of ≥49.4% for the primary aim (ie, VE between randomized and control participant groups), examined with a 2-sided alpha 0.05 superiority test.

For the secondary aim, noninferiority of the Novavax COVID-19 vaccine versus the Pfizer–BioNTech COVID-19 vaccine will be declared if the lower one-sided 95% CI for rVE is >50%. With 600 participants per vaccine group and an assumed infection rate of 60 per 100,000 person-days in the Pfizer–BioNTech COVID-19 vaccine arm and a rVE of 10% for the Novavax COVID-19 vaccine versus the Pfizer–BioNTech COVID-19 vaccine (ie, Novavax>Pfizer–BioNTech) [20,21], there would be 80% power for the noninferiority objective. The positive rVE used in the sample size calculation is based on estimates from large retrospective studies [20,21], which showed that the Novavax COVID-19 vaccine provides added protection against medically attended and severe SARS-CoV-2 infection. Sample size calculations for study aims were performed in PASS 2023 (NCSS, LLC) using the Cox proportional hazards model to estimate hazard ratios. Certain secondary objectives will be underpowered if there are insufficient numbers of nonsymptomatic cases, reinfections, and/or PCCs during the study period.

Participant demographics and characteristics will be summarized descriptively in the mITT population. The Cox proportional hazards model will be used to analyze time-to-event data to evaluate VE in preventing symptomatic COVID-19 infection. The observation period started 7 days after the receipt of the updated 2023-2024 vaccine for the randomized groups and at the enrollment date for the control group. Event onset time is defined as the earlier of the date of a positive rapid antigen test or the date of CLI symptom onset reported within a week of a positive rapid antigen test. Covariates of age (continuous variable) and vaccination dose history (categorical variable) will be included in the model to control for confounding. Additional demographic variables will be considered for inclusion if they change the hazard ratio for the treatment variable by ≥10%. Subgroup analyses based on age group and immune challenge (ie, most recent COVID-19 infection or vaccination) were conducted to assess potential effect modification across age and immune-challenge subgroups. All analyses will be performed using SAS (version 9.4 or higher; SAS Institute Inc), or R (version 4.1.2 or higher; R Foundation for Statistical Computing).

Remuneration and Engagement

Participants received compensation in the form of Amazon, Inc. gift cards, with the value proportional to the number of study activities completed (Multimedia Appendix 1). Newsletters were shared throughout the study with investigators and participants, including information such as enrollment numbers, instructions on how to remain compliant and up-to-date with study protocol and surveys, and updates on COVID-19 activity in the area. Study branding was consistently presented from recruitment throughout the study.

Results

Participant Enrollment and Data Collection

Participant screening occurred from November 17, 2023, through March 14, 2024. Monitoring occurred from enrollment through the end of the study on September 9, 2024. A total of 1188 participants were enrolled: 909 in the randomized vaccine groups (Novavax COVID-19 vaccine, 452/909, 49.7%; Pfizer–BioNTech COVID-19 vaccine, n=457, 50.3%) and 279 in the nonrandomized control group. Data analyses are ongoing, and results are expected to be published in 2026.

Vaccination Timing and SARS-CoV-2 Circulation

Vaccination and data collection during the study occurred when XBB.1.5-, JN.1-, KP.2-, and KP.3-lineage SARS-CoV-2 variants were circulating in the United States and the Utah region (Figure 2 [22]). Study vaccinations were relatively evenly distributed throughout the enrollment period, with most occurring when XBB.1.5 and JN.1 represented ≥50% of SARS-CoV-2 strains circulating in Utah (January 19, 2023, and December 30, 2023, respectively).

‎

Discussion

The BEEHIVE study will provide real-world VE data for the Novavax COVID-19 vaccine when administered as an additional dose after at least a primary series of an mRNA-based COVID-19 vaccine and will compare these VE data with those of the Pfizer–BioNTech COVID-19 mRNA vaccine. Outcomes may provide information on the benefits of heterologous dosing or the use of mixed vaccine formulations, for administration beyond primary vaccination.

Access to participant cohorts from previous studies allowed recruitment of individuals who were more likely to participate and facilitated access to sociodemographic information and vaccination history. Mobile vaccine clinics, attendance at community events, and collaborations with civic groups were more effective in engaging typically underrepresented communities (eg, older adults and racial and ethnic minority individuals) [23,24] than standard clinical trial site strategies. Electronic surveillance and the use of online and text-based surveys supported efficient data collection and sustained participant engagement throughout the 24-week surveillance period.

A potential limitation of this study includes the underrepresentation of participants in the older adult categories. Additionally, the variability in SARS-CoV-2 circulation and the introduction of new variant strains during the study period would impact infection rates and symptom severity. Notably, the testing kits, which were identical for all study participants, had received Emergency Use Authorization from the US FDA for Omicron and subvariant testing. The single-site nature of the trial may restrict geographic diversity among the participants; however, the feasibility of a real-world study design was established and can be adapted for future multisite studies. Because rapid antigen tests were used as part of the real-world study design, repeat testing procedures were included in the design to mitigate concerns regarding the absence of more robust clinical assays (eg, polymerase chain reaction). Another consideration is that the control group that opted out of receiving a study vaccination may introduce selection bias. Briefly, participants who declined the recommended vaccine may have differed demographically or behaviorally from those who chose to receive a study vaccine [25].

Participant enrollment (n=1188) was lower than the target of 1500 described in sample size calculations. We acknowledge that the study is underpowered due to recruitment challenges, and the implications of the smaller sample size on effect detectability will be discussed in the final report. Additionally, assumptions of superior rVE of Novavax compared with mRNA vaccines based on medically attended COVID-19 infection may not directly translate to protection against symptomatic SARS-CoV-2 infection. Future noninferiority studies should consider sample size calculations assuming an rVE of 0 to ensure a more robust study design.

Continued surveillance of COVID-19 cases and VE data for vaccine formulations updated based on predominant SARS-CoV-2 strains is necessary to inform public health policy and management. The BEEHIVE study will provide valuable real-world VE data for populations representative of individuals who have received the 2023-2024 updated COVID-19 vaccine, including those who received a dose of the Novavax COVID-19 vaccine after receipt of ≥2 mRNA-based COVID-19 vaccines.