Background: As the COVID-19 pandemic continues to affect life in the United States, the important role of nonpharmaceutical preventive behaviors (such as wearing a face mask) in reducing the risk of infection has become clear. During the pandemic, researchers have observed the rapid proliferation of misinformed or inconsistent narratives about COVID-19. There is growing evidence that such misinformed narratives are associated with various forms of undesirable behavior (eg, burning down cell towers). Furthermore, individuals’ adherence to recommended COVID-19 preventive guidelines has been inconsistent, and such mandates have engendered opposition and controversy. Recent research suggests the possibility that trust in science and scientists may be an important thread to weave throughout these seemingly disparate components of the modern public health landscape. Thus, this paper describes the protocol for a randomized trial of a brief, digital intervention designed to increase trust in science.
Objective: The objective of this study is to examine whether exposure to a curated infographic can increase trust in science, reduce the believability of misinformed narratives, and increase the likelihood to engage in preventive behaviors.
Methods: This is a randomized, placebo-controlled, superiority trial comprising 2 parallel groups. A sample of 1000 adults aged ≥18 years who are representative of the population of the United States by gender, race and ethnicity, and age will be randomly assigned (via a 1:1 allocation) to an intervention or a placebo-control arm. The intervention will be a digital infographic with content based on principles of trust in science, developed by a health communications expert. The intervention will then be both pretested and pilot-tested to determine its viability. Study outcomes will include trust in science, a COVID-19 narrative belief latent profile membership, and the likelihood to engage in preventive behaviors, which will be controlled by 8 theoretically selected covariates.
Results: This study was funded in August 2020, approved by the Indiana University Institutional Review Board on September 15, 2020, and prospectively registered with ClinicalTrials.gov.
Conclusions: COVID-19 misinformation prophylaxis is crucial. This proposed experiment investigates the impact of a brief yet actionable intervention that can be easily disseminated to increase individuals’ trust in science, with the intention of affecting misinformation believability and, consequently, preventive behavioral intentions.
Trial Registration: ClinicalTrials.gov NCT04557241; https://clinicaltrials.gov/ct2/show/NCT04557241
International Registered Report Identifier (IRRID): PRR1-10.2196/24383
Crowdfunding campaign to support this specific research
We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.
Suggested contribution levels: $20/$50/$100
COVID-19 and Misinformation
The COVID-19 pandemic has significantly affected the United States in numerous ways, both directly and indirectly. As of July 17, 2020, the number of daily new cases per 100,000 people was sharply increasing at a rate that exceeded corresponding increases in testing numbers across various states in the country . The same day, the death toll due to COVID-19 in the United States was 133,600 [ ], but this number did not reflect the anticipated lag time between diagnosis and fatality. In addition, there has been extensive disruption of most major societal structures, including economic and educational systems during the pandemic [ ]. From July 10 to July 17, 2020, alone, the Centers for Disease Control and Prevention (CDC) reported 133 news stories about major social impact due to COVID-19 [ ].
Behavioral preventive measures continue to be an effective primary public health tool for addressing challenges during the pandemic [- ]. However, adherence to behavioral recommendations to prevent COVID-19 spread has been inconsistent, especially in the United States, with documented instances of refusal among lawmakers [ ], airline passengers [ ], general consumers [ ], and churches [ ], among others. As COVID-19 cases and related fatalities continue to rise, it is critical to identify the factors underpinning refusal to undertake basic preventive measures against disease transmission and adopt suitable measures to address them.
In addition to the spread of COVID-19, researchers have reported extensive proliferation of misinformation and conspiracies about the disease [- ]. Intensive efforts have been made to delineate the accuracy of information about COVID-19 shared on social media (ie, by using natural language processing) [ ]. However, numerous governmental and scientific organizations have simultaneously issued inconsistent and contradictory guidance about preventive measures such as face masks, further complicating the issue [ ]. Although differential but reasonable interpretations of extant data and modification of recommendations in response to new data are expected of the scientific process, public correspondence, including that via official channels, has suggested otherwise. This oversaturation—with different sources of information of varying quality being constantly disseminated—adds considerably to the complexity of COVID-19 prophylaxis and may have real-world consequences. For example, one can identify specific, problematic behavioral outcomes that can be conceptually mapped to believing particular misinformed COVID-19 narratives (eg, 5G wireless, Bill Gates vaccination, and restriction of liberty) [ ].
Research on misinformation and conspiracy theories, in general, has suggested that political orientation , religious commitment [ ], and cognitive sophistication [ ] are core factors associated with such beliefs. However, emerging research and commentary specific to COVID-19, including our own study, have indicated strong associations between trust in science and political and religious factors, as well as the belief in misinformed narratives and support for public health prevention efforts [ , - ].
COVID-19 Nonpharmaceutical Preventive Behaviors
Extant cross-sectional research investigating COVID-19 nonpharmaceutical preventive behaviors (NPBs) has frequently identified that unchangeable factors or hard-to-change factors serve as significant predictors, such as political beliefs [, ], choice of news channel [ ], age, and sex [ ]. Single studies have reported that COVID-19 conspiracies may mediate between vertical individualism (believing that people are autonomous and unequal) and social distancing [ ] and that beliefs about the efficacy of NPBs increase voluntary preventive compliance [ ]. Other studies have also indicated that self-efficacy and perceived severity might be associative factors of NPBs [ , ].
Thus far, only 2 experimental studies on COVID-19 misinformation and NPBs have been conducted. One study focused on the likelihood of sharing false narratives on social media and found that sharing misinformation is a function of inattention, but this study did not examine trust or beliefs . The other study focused on trust in science and examined political orientation as a moderating variable, using support for social distancing as the outcome, but this study did not address misinformation [ ]. To our knowledge, our proposed study will be the first experimental study to examine improving trust in science as a means of reducing the likelihood of believing scientifically implausible narratives about COVID-19 and increasing intentions to engage in COVID-19 NPBs.
Specifically, we will expand on current knowledge by assessing whether exposure to a brief informational statement (in the form of an infographic) about the scientific process can increase trust in science, reduce the likelihood of believing scientifically implausible narratives about COVID-19, and increase intentions to engage in recommended COVID-19 NPBs. Given the scale and scope of the pandemic, an intervention with a small effect size, if feasible to deploy within the cost-effective US social media infrastructure, would have the potential to save lives through increased adherence to NPBs. However, it is important not to rush the deployment of brief social media campaigns without careful research and planning to verify efficacy, given that such campaigns could have iatrogenic effects even when designed by scientists and media experts [, ]. This echoes the recent findings by Lane and Fauci [ ], who remind researchers even in the context of a pandemic, it is important to generate scientifically sound evidence.
Conceptual Framework for the Proposed Study
Our prior research on COVID-19 misinformation identified 4 different COVID-19 belief profiles . The “scientific” profile was the most common (~70% of the respondents), comprising individuals who reported high believability for a statement about the zoonotic origin of COVID-19 and low believability for misinformed narratives. The other 3 “nonscientific” profiles were not named and were instead numbered alongside conceptual descriptions. Profile 2 (~8% of the respondents) comprised individuals who reported high believability for all narratives, including the zoonotic statement and misinformed statements. Profile 3 (~12% of the respondents) comprised individuals who reported moderate believability for all narratives, and the lowest believability for the zoonotic statement of any profile (although it was still reasonably high, with a mean score of 4.59 on a scale of 1-7). Finally, Profile 4 (~10% of the respondents) was similar to Profile 2, except for a comparatively lower endorsement of the narrative that 5G networks caused the spread of COVID-19. Trust in science (a scale variable, scored from 1 to 5, and computed from a 21-item questionnaire) [ ] was substantively associated with different profile memberships, after controlling for political orientation, religious commitment, race and ethnicity, gender, age, and education level. Compared to the scientific profile, each 1-point decrease in trust in science was associated with 5 (for Profile 3) to 14.3 (for Profiles 2 and 4) times higher adjusted odds of belonging to nonscientific profiles [ ]. Therefore, we speculate that intervening at the level of trust in science will potentially nudge some individuals into the scientific profile.
However, whether trust in science will further affect individuals’ intention to engage in preventive NPBs via mediated beliefs in misinformation remains to be clarified. To the extent that some common narratives suggest that COVID-19 does not pose a serious health threat , such narratives may reduce the magnitude of “perceived severity” based on the health belief model. It is also notable that cross-sectional studies have suggested an association between perceived severity, self-efficacy, and COVID-19 NPBs in Turkey and Kenya [ , ].
Separately, a complex network analysis in the United Kingdom and Netherlands found that COVID-19 NPBs are most closely related to normative beliefs held by family and friends, along with the beliefs that preventive measures work . Given the emerging conspiratorial narrative that face masks are an attempt to exert social control and do not actually prevent COVID-19 transmission, it is plausible that such narratives also affect the perceived efficacy of common NPBs. More direct, yet anecdotal findings in support of this mediated relationship include documented incidences of this type of misinformation being explicitly stated by individuals who publicly refuse to engage in NPBs [ - ]. Finally, in the only causal finding, Koetke et al [ ] reported that trust in science exerted a direct influence on the intention to practice social distancing, but their work focused on political ideology, not misinformation, as a mediating variable. A similar associative finding by Chambon et al [ ] suggested that “trust in authorities” was moderately associated with an increase in NPBs. A depiction of the overall conceptual framework of study variables is illustrated in .
Study Aims and Hypotheses
This study will accomplish the following 3 aims:
We aim to assess the effect of a brief informational infographic about the scientific process on trust in science. We hypothesize that exposure to such an intervention will have a moderate, positive effect on trust in science.
We aim to assess the effect of a brief informational infographic about the scientific process on the likelihood of believing scientifically implausible narratives about COVID-19. We hypothesize that exposure to such an intervention will have a small, negative effect on the likelihood of believing implausible narratives, as evidenced by profile membership, and that this will be partly mediated by trust in science.
We aim to assess the effect of a brief informational infographic about the scientific process on behavioral intentions to engage in recommended COVID-19 NPBs. We hypothesize that exposure to such an intervention will have a small, positive effect on behavioral intentions to engage in recommended COVID-19 NPBs that will be partly mediated by misinformation profile membership.
Ethics Approval and Consent to Participate
This study has been approved by the Indiana University Institutional Review Board (IRB), protocol #2008571490. Participants in both the pilot test and the main study will digitally indicate their consent to participate after reviewing a study information sheet.
We propose a single-stage, randomized, superiority trial comprising 2 parallel groups allocated with a 1:1 ratio. The study design and workflow involving participants is shown in. The comparator in this study will be a control (“placebo”) infographic that is completely unrelated to science (eg, an infographic about cats) but is developed using the same communication and graphical style. The full trial, including presentation of the intervention or control condition, will be embedded within the data collection platform and will be completed in a single sitting.
Participants will be recruited using the data collection platform Prolific, which is one of two primary online crowdsourced research platforms (the other is Amazon’s Mechanical Turk, or mTurk). Evidence suggests that both platforms replicate known experimental outcomes when studies are structured correctly and that the platforms compare favorably to a university-recruited subject pool [, ]. Prolific also has the ability to collect a nationally representative sample for the United States by age, sex, and race and ethnicity, thereby improving generalizability of findings [ ]. Prolific outperforms mTurk with regard to the number of accessible participants (>40,000 vs 15,000-30,000), responsiveness, diversity, and quality of participants [ , ]. Recent studies, including studies on COVID-19 perceptions, have successfully used nationally representative samples from Prolific [ ].
For inclusion in the study, participants must be identified by Prolific as part of a nationally representative sample. Only participants aged 18 years or older and residing in the United States will be considered. Individuals who decline to digitally sign the informed consent document will be excluded from the study and replaced by other eligible individuals. Based on best practice recommendations for crowdsourced digital research, attention checks and screens for “bots” and international users with virtual private networks to mimic US internet protocol addresses will be embedded within the instruments, and failure of more than one attention check, or any bot or location check will result in subject exclusion and replacement . Attention checks will be located prior to randomization in Qualtrics, a cloud-based survey tool, so replacements will be randomized with the same allocation ratio and will be drawn in a manner that preserves the representativeness of the sample.
The primary intervention in this study will be an infographic that is designed to build trust in the scientific process. Infographics are preferable to narratives or text because they focus on visuals as part of the storytelling process and facilitate cognitive information processing, knowledge absorption, and enhanced persuasion [- ]. The infographic design used in this study will follow best practices in health communication. The message communicated will be simple and jargon free. The infographic will comprise visuals of individuals (scientists), charts, text, and numerical data [ ]. Attention will be paid to the images used, color, frames, representation, and composition (eg, how the elements in the infographic are organized to show their relationship to each other) [ ]. The design process will be completed in the following 2 stages (note that these are development stages, not trial stages).
Sample concepts and messaging will be created based on the core constructs underlying the trust in science inventory . The messages will assume divergent approaches to clarify what science is, how the scientific process operates, and how science is a self-correcting process. This content may focus, for example, on how the self-correcting elements of science serve to enhance people’s quality of life (eg, addressing items 1 and 21 from the inventory [ ]). In such a case, the accompanying visuals might demonstrate a flat earth progressing to a globe, and then to a picture of the earth captured from a space station. These messages will be informally discussed by the authors’ nonscientific social network in preparation for a formal pilot study.
In collaboration with professional graphic designers, the study team will design 5 infographics for the pilot testing. At present, we plan that each infographic will use an “internet comic” style of presentation that will be familiar to most participants to express core concepts underlying trust in science. The infographics will each be presented to 20 mTurk users (using the same procedures to screen for respondent quality as the overall study). Enrolled participants (N=1000) will first complete the trust in science inventory, and they will then be randomly assigned to view one of the infographics, following which they will be required to complete the inventory again. Participants will also be asked to qualitatively describe the meaning conveyed through the infographic (an open-ended question)  and will be asked to indicate how believable they find the infographic using a validated modification of the narrative believability scale (nbs-12) [ ]. In this manner, the best-performing infographic, based on the judgment of the study team, will be used for the intervention arm of the experiment, and will be made available as a supplemental file alongside the published results. This decision will be made based on qualitative response, believability, and any observed effect on trust in science (although in the latter case, the pilot sample is not sufficiently powered to test a hypothesis, so we will consider the data broadly within the context of the other metrics).
Participants will be required to pause for at least 60 seconds while viewing the infographic (ie, the button to proceed forward will be hidden). The control infographic will be completed by the same designers but will be a placebo (ie, it will be a summary of basic information about a neutral topic, such as cats). It will mention neither science nor scientific processes.
Primary Study Outcomes
Trust in Science
Participants’ “trust in science” will be measured both before and after they view the intervention or placebo using the 21-item scale developed by Nadelson et al . Our previous research has found that this scale is highly reliable for diverse, online, and crowdsourced respondents [ , ].
Next, participants’ “believability profiles” will be computed via a latent profile analysis of the believability measures described below, which will be administered after the infographic is viewed. The measures and approach used to conceptualize believability of misinformation were developed and first used in our recent study on COVID-19 narratives . As in the original study, we will not prespecify the existence of certain profiles, although we do expect to identify similar profiles in this study (potentially with differences introduced by the additional metrics described subsequently).
Response options for all believability measures will use well-established semantic differential responses for believability of different statements (eg, as used by Herzberg et al ), ranging from 1 (“extremely unbelievable”) to 7 (“extremely believable”). The original measures [ ] assessed the believability of 4 statements related to COVID-19 that were derived from common pieces of misinformation (as of April 2020), as identified by a team at Cornell University [ ]. For instance, one of the statements was, “The recent rollout of 5G cellphone networks caused the spread of COVID-19.” Another statement asked subjects about their believability of a particular statement reflecting scientific consensus (ie, the zoonotic origin of the virus). Our ability to use latent profile analysis to generate plausible and conceptually meaningful subgroups with a good model fit points toward a certain degree of validity of these questions. This is noteworthy especially given the existence of a latent profile that generally believed the scientific consensus explanation and no other narrative (which would be less likely with invalid questions). Therefore, in this study, we will assess participants’ believability of the following 6 statements, which include slightly modified versions of the 4 above-referenced statements, as well as 2 new statements based on scientific findings clarifying emergent, persistent misinformation about the use of face masks to prevent COVID-19 spread [ - ]:
- “The rollout of 5G cellphone networks caused the spread of COVID-19.”
- “SARS-Cov-2, the virus that causes COVID-19, likely originated in animals (like bats) and then spread to humans.”
- “Bill Gates caused (or helped cause) the spread of COVID-19 in order to expand his vaccination programs.”
- “COVID-19 was developed as a military weapon (by China, the United States, or some other country).”
- “The number of deaths from COVID-19 has been exaggerated as a way to restrict liberties in the United States.”
- “Wearing a face mask for COVID-19 prevention can cause oxygen deficiency or carbon dioxide intoxication.”
- “Face masks are probably not helpful in reducing COVID-19 spread in a community.”
Participants’ behavioral intentions will focus on the following specific recommendations current proposed by the CDC :
- Wash your hands often (or use a hand sanitizer that contains at least 60% alcohol).
- Avoid close contact (stay at least 6 feet from other people).
- Cover your mouth and nose with a mask when around others.
- Cover coughs and sneezes.
- Clean and disinfect frequently touched surfaces daily.
- Monitor your health daily.
Each recommendation will be placed into a behavioral intention questionnaire format to assess self-reported likelihood of behaviors using a guide published by Azjen who proposed the Theory of Planned Behavior . For example, “I intend to clean and disinfect frequently touched surfaces daily for the next month,” with response options ranging from 1 (“likely”) to 7 (“unlikely”). These questions will be administered after the infographic is viewed.
We will also collect the following additional variables to serve as covariates in the models based on other factors suspected to influence trust in science, believability of misinformation about COVID-19, and/or behavioral intentions regarding NPBs.
- Political orientation and religious commitment, using the scales from our previous works [ , ]
- Race and ethnicity, gender, age, and education level
- Whether the respondent has been diagnosed with, or believes they have had, COVID-19 [ ]
- Perceived severity and self-efficacy regarding COVID-19, based on the health belief model and that used by Yıldırıma & Gülerc [ ] (which was derived from similar work related to SARS)
- Normative beliefs about friends’ and family’s COVID-19 behaviors (single item from Chambon et al [ ])
We will recruit 1,000 individuals using Prolific, which is the maximum sample size permitted for a nationally representative sample via this platform. This sample size will give allow us to detect small (Cohen’s d=0.18) differences between both groups with 80% power. This effect size would be more than sufficient for both analysis types, that is, LMM (linear mixed models) and path analyses, thus accounting for power and other sample size considerations common to path analysis (eg, overspecification) .
Participants will be recruited via the Prolific platform to ensure a nationally representative sample of the US is composed with regard to age, sex, and race and ethnicity. To ensure compliance with institutional review board requirements, the questionnaires and procedures (including the study information sheet) will be hosted securely within Indiana University’s Qualtrics platform. Prolific will refer the sampled individuals to the Qualtrics system. Thereafter, we will randomly generate a unique identification for each Qualtrics user and ask the participants to enter it into Prolific to verify that they have completed the survey. We anticipate the survey will take no more than 15 minutes to complete; compensation will be set by Prolific to be equitable but noncoercive for research. The proposed methodology mirrors the protocol we successfully used with mTurk in our previous works [, ].
Allocation of Interventions
The allocation sequence will be managed using the Randomizer tool in the Qualtrics platform  to ensure a 1:1 allocation of participants to intervention (1) and control (0) arms, each comprising 500 participants. Allocation will occur after consent has been processed. Furthermore, the procedure is automated, thereby ensuring allocation concealment.
Although this is, in practice, a double-blind study since participants will be unaware that they are randomized and all study mechanics will be processed using a computer, analysts will not be blinded to the meaning of the assignment variable. However, 2 independent consultant analysts have been retained to verify all results and subsequent interpretation.
Data Collection and Management
All data will be collected and stored using the QualtricsXM (Qualtrics) digital platform. This platform enables direct exporting of data to a variety of formats (eg, Excel and SPSS). To promote data quality, respondents will be required to respond to all items on each page of the survey before proceeding to the next part of the study, which would ensure there are no missing data from completed cases. While excellent participant retention is expected since the study (including intervention) is brief and occurs contemporaneously, some participants may decide to quit prior to completion. If a participant has provided any data beyond the study information sheet, the case will be analyzed according to the study arm to which it was assigned, with missing data managed as described in Statistical Analysis (eg, intention-to-treat). However, this will not apply to participants who are excluded due to failed quality checks (see Eligibility Criteria) because these are designed to filter for cases that were not eligible but were enrolled in the study inappropriately (eg, “bots” or individuals who mask their global location with a virtual private network).
Missing Data and Data Quality
Missing data will be addressed using either full information maximum likelihood or Markov Chain Monte Carlo multiple imputation strategies . In case there is a violation of missing at random in preliminary analyses (which is unlikely), we will incorporate strategies representing the missingness. We will further explore data quality in terms of outliers, measurement error, non-normality, and variance heterogeneity. Robust methods of analysis (eg, Huber-White robust standard errors) will be used, as appropriate [ ]. For all multi-item measures, we will evaluate reliability prior to computation of the variable.
The following analyses will be performed for the 3 aim statements:
Aim 1: We will use an LMM to examine the effect of the intervention on trust in science, controlling for specified covariates (see Covariates).
Aim 2: We will first examine the profiles of believability in our control and intervention group using latent profile analyses [, ]. After determining the number of classes and identifying scientific versus nonscientific profiles, we will conduct path analyses linking the brief digital intervention to believability profiles and the mediator (trust in science).
Aim 3: We will first examine the outcome variables for Aim 3—intention to engage in NPBs. Although we expect these intention-based items to function as a monotonic scale that can be collapsed into a single variable, we will need to conduct exploratory factor analysis to determine whether this is the case. The number of factors identified will affect how this outcome is treated in the subsequent part of Aim 3.
We will also examine the mediation effect of believability profiles on the association between behavioral intentions, trust in science, and the primary outcome for Aim 3 (ie, intention to engage in NPBs) in the post-test. Next, we will conduct path analyses linking the brief digital intervention to believability profiles and the mediators (trust in science and believability profiles), and behavioral intentions in the post-test.
For both Aims 2 and 3, we will use model fit statistics (eg, root mean square error of approximation, comparative fit index, and Tucker-Lewis Index) and examine specific indices of ill fit (eg, modification indices). To increase power and maintain Type I error, we will adapt the posterior probability method (eg, partial P value) for formal mediation analyses with intervening variables . We aim to elucidate key mediational chains between our mediators (ie, trust in science and believability classes), predictors (ie, behavioral intervention statement), and outcome (ie, intention to engage in NPBs) through this design, instead of simply looking at the link between the intervention and intention to engage in NPBs.
This study was funded in August 2020, approved by the Indiana University Institutional Review Board on September 15, 2020, and prospectively registered with ClinicalTrials.gov. We expect the infographics to be finalized in early December 2020 and plan to run the pilot test in the same month. The entire experiment should be completed, and the results published, in 2021. This protocol was submitted as prepared for review in a grant format to the Indiana Clinical and Translational Sciences Institute in June 2020. Aside from stylistic modifications made in transforming the grant into a paper, and in addressing grant reviewers’ and manuscript reviewers’ comments, the protocol reflects the originally proposed work.
The primary purpose of this study is to test an actionable, brief digital intervention that can modify trust in science and thereby mitigate belief in misinformation and increase behavioral intentions to engage in NPBs. Thus, if any hypotheses are validated, the primary next step will be to disseminate this information and coordinate use of the infographic. Two articles published in Nature [, ], (including one by that journal’s editorial board), acknowledged the need for researchers to honestly and transparently address COVID-19 misinformation, and to specifically look into trust, but the cited research primarily examined networking and spread dynamics, not prevention.
Data Monitoring, Interim Analyses, and Auditing
This study will not have a data monitoring committee because data collection will be automated, stored, and archived. Moreover, no harms are anticipated to occur as a result of the experiment. No interim analyses are planned. Data and analyses will be reviewed by all members of the study team as well as by independent statistical consultants.
This study has several limitations that might affect its conclusions. First, the data are based on self-reporting. For believability and trust in science, this is a less substantive issue; however, it is potentially subject to social desirability bias, which we will attempt to minimize. Although extant behavior models like the Theory of Planned Behavior  indicate that intentions are direct antecedents to behavior, we cannot be certain that intentions will result in actual behavior. Nevertheless, measuring this would be beyond the scope of this protocol. Second, we are unable to control the conditions in which people will participate, but we will incorporate attention checks throughout the process. Furthermore, prior studies have been able to replicate known experimental findings using crowdsourced digital participants, suggesting that the undue influence of inattention can be minimized [ , ]. Third, this experimental approach is likely subject to a Type II error because a real-life application of this process would entail repeated exposures over time, whereas this approach tests the effect of a single exposure. Fourth, other variables that might influence behavioral intentions also likely exist, but by recruiting a large sample and randomizing participants, we will be able to more cleanly isolate the differences attributable to the intervention.
Finally, generalizability to individuals who do not have internet access or who are unable to participate in online surveys might be limited. However, regarding the concern about access, it is noteworthy that approximately 90% of US adults used the internet in 2019, with higher percentages reported for younger adults and lower percentages, for older adults. Of note, 73% of individuals aged >65 years reported they had internet access in 2019 (although it began trending sharply upward in 2018) . Furthermore, as noted throughout our proposal (and in academic papers published since submission [ ]), social media remains a primary source of misinformation about COVID-19. Considering that the online data collection platform introduces bias as a result of internet access, we speculate that controlling for age (which we already do) may attenuate this concern to an extent, in addition to the fact that the intervention itself is designed for distribution on social media.
Regarding the latter point (opt-in participation), we acknowledge that individuals who opt into survey completion may differ systematically in one or more ways from those who do not. However, the data collected through platforms like Prolific are of good quality and have replicated results of multiple established experiments, thereby reducing concerns about the influence of opt-in participation on the results. Further, studies on similar crowdsourced survey platforms such as mTurk have found that the study participants are similar to the overall US population with regard to a number of different characteristics. However, they also found that survey takers are younger and are more educated, on average, than the US population as a whole [- ]. Both variables (ie, age and education level) will be controlled in our study’s analyses. This further mitigates concerns that the participants recruited by Prolific are systematically different from those that might be enrolled in a similar experiment in other circumstances.
This study was made possible, in part, with support from the Indiana Clinical and Translational Sciences Institute funded, in part by Award Number UL1TR002529 from the National Institutes of Health, National Center for Advancing Translational Sciences, Clinical and Translational Sciences Award. The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. We would like to thank the reviewers at the Indiana Clinical and Translational Sciences Institute for their insightful feedback on our initial submission of our proposal.
JA and YX developed the idea for the study. JA, YX, and ET conjointly prepared the protocol. YX and LG prepared the analysis plan. All authors reviewed and approved the final manuscript.
Conflicts of Interest
Review of original grant submission (Reviewer 1, scores redacted by funder).PDF File, 115KB
Review of original grant submission (Reviewer 2, scores redacted by funder).PDF File, 111KB
- Daily confirmed new cases (7-day moving average). Johns Hopkins University & Medicine - Coronavirus Resource Center. URL: https://coronavirus.jhu.edu/data/new-cases-50-states [accessed 2020-07-17]
- COVID-19 United States cases by county. Johns Hopkins University & Medicine - Coronavirus Resource Center. URL: https://coronavirus.jhu.edu/us-map [accessed 2020-07-17]
- Everyone included: social impact of COVID-19. United Nations Department of Economic and Social Affairs - Social Inclusion. URL: https://www.un.org/development/desa/dspd/everyone-included-covid-19.html [accessed 2020-07-18]
- Social impact COVID-19 events in the United States. Centers for Disease Control and Prevention - CDC COVID Data Tracker. URL: https://www.cdc.gov/covid-data-tracker/index.html#social-impact [accessed 2020-07-17]
- Fauci AS, Lane HC, Redfield RR. Covid-19 — Navigating the uncharted. N Engl J Med 2020 Mar 26;382(13):1268-1269. [CrossRef]
- Greenhalgh T, Schmid MB, Czypionka T, Bassler D, Gruer L. Face masks for the public during the covid-19 crisis. BMJ 2020 Apr 09;369:m1435. [CrossRef] [Medline]
- Zhang N, Cheng P, Jia W, Dung C, Liu L, Chen W, et al. Impact of intervention methods on COVID-19 transmission in Shenzhen. Build Environ 2020 Aug;180:107106 [FREE Full text] [CrossRef] [Medline]
- Gordon A. Dozens of Mississippi lawmakers have coronavirus after weeks of refusing to wear masks. CNN. 2020 Jul 10. URL: https://www.cnn.com/2020/07/10/us/mississippi-coronavirus-legislature-trnd/index.html [accessed 2020-07-11]
- McGinnis C, Sue T. Airline passengers refuse to wear masks, mayhem ensues. SFGate. 2020 Jun 18. URL: https://www.sfgate.com/travel/article/Airline-passenger-refuses-mask-15349202.php [accessed 2020-07-11]
- Meyersohn N. Stores want shoppers to wear masks, but some customers refuse. KCTV News. 2020 Apr 23. URL: https://tinyurl.com/yy4ukcz3 [accessed 2020-07-11]
- Associated Press. Church linked to Oregon’s largest outbreak as daily coronavirus count hits record high. Los Angeles Times. 2020 Jun 17. URL: https://www.latimes.com/world-nation/story/2020-06-17/church-linked-oregon-largest-coronavirus-outbreak [accessed 2020-07-11]
- Mian A, Khan S. Coronavirus: the spread of misinformation. BMC Med 2020 Mar 18;18(1):89 [FREE Full text] [CrossRef] [Medline]
- Kouzy R, Abi Jaoude J, Kraitem A, El Alam MB, Karam B, Adib E, et al. Coronavirus goes viral: quantifying the COVID-19 misinformation epidemic on Twitter. Cureus 2020 Mar 13;12(3):e7255 [FREE Full text] [CrossRef] [Medline]
- Brennen JS, Simon FM, Howard PN, Nielsen RK. Types, sources, and claims of COVID-19 misinformation. The Reuters Institute for the Study of Journalism. 2011 Apr 07. URL: https://reutersinstitute.politics.ox.ac.uk/types-sources-and-claims-covid-19-misinformation [accessed 2020-11-30]
- Verspoor K. ACL 2020 Workshop on Natural Language Processing for COVID-19 (NLP-COVID).: Association for Computational Linguistics; 2020. URL: https://openreview.net/group?id=aclweb.org/ACL/2020/Workshop/NLP-COVID [accessed 2020-07-12]
- Tso R, Cowling B. Importance of face masks for COVID-19: a call for effective public education. Clin Infect Dis 2020 Nov 19;71(16):2195-2198 [FREE Full text] [CrossRef] [Medline]
- Agley J, Xiao Y. Misinformation about COVID-19: evidence for differential latent profiles and a strong association with trust in science. Research Square. Preprint posted online on Oct 21, 2020. [FREE Full text] [CrossRef]
- Sutton RM, Douglas KM. Conspiracy theories and the conspiracy mindset: implications for political ideology. Current Opinion in Behavioral Sciences 2020 Aug;34:118-122. [CrossRef]
- Jasinskaja-Lahti I, Jetten J. Unpacking the relationship between religiosity and conspiracy beliefs in Australia. Br J Soc Psychol 2019 Oct;58(4):938-954. [CrossRef] [Medline]
- Pennycook G, McPhetres J, Bago B, Rand D. Predictors of attitudes and misperceptions about COVID-19 in Canada, the U.K., and the U.S.A. PsyArXiv. Preprint posted online April 14, 2020. [FREE Full text]
- Agley J. Assessing changes in US public trust in science amid the COVID-19 pandemic. Public Health 2020 Jun;183:122-125 [FREE Full text] [CrossRef] [Medline]
- Vraga EK, Tully M, Bode L. Empowering users to respond to misinformation about Covid-19. MaC 2020 Jun 25;8(2):475-479 [FREE Full text] [CrossRef]
- Koetke J, Schumann K, Porter T. Trust in science increases conservative support for social distancing. OSF Preprints. Preprint posted online June 24, 2020. [FREE Full text] [CrossRef]
- Bennett M. Should I do as I’m told? Trust, experts, and COVID-19. Kennedy Institute of Ethics Journal 2020;30(3-4):243-263. [CrossRef]
- Painter M, Qiu T. Political beliefs affect compliance with COVID-19 social distancing orders. SSRN Journal. Preprint posted online July 3, 2020. [CrossRef]
- Makridis C, Rothwell J. The real cost of political polarization: evidence from the COVID-19 pandemic. SSRN Journal. Preprint posted online Jun 30, 2020. [FREE Full text] [CrossRef]
- Simonov A, Sacher S, Dube J, Biswas S. Non-compliance with social distancing during the COVID-19 pandemic. National Bureau of Economic Research. Working Paper Series 2020 May [FREE Full text] [CrossRef]
- Solomou I, Constantinidou F. Prevalence and predictors of anxiety and depression symptoms during the COVID-19 pandemic and compliance with precautionary measures: age and sex matter. Int J Environ Res Public Health 2020 Jul 08;17(14) [FREE Full text] [CrossRef] [Medline]
- Biddlestone M, Green R, Douglas KM. Cultural orientation, power, belief in conspiracy theories, and intentions to reduce the spread of COVID-19. Br J Soc Psychol 2020 Jul;59(3):663-673 [FREE Full text] [CrossRef] [Medline]
- Clark C, Davila A, Regis M, Kraus S. Predictors of COVID-19 voluntary compliance behaviors: an international investigation. Glob Transit 2020;2:76-82 [FREE Full text] [CrossRef] [Medline]
- Yıldırım M, Güler A. COVID-19 severity, self-efficacy, knowledge, preventive behaviors, and mental health in Turkey. Death Stud 2020 Jul 16:1-8. [CrossRef] [Medline]
- Kigatiira KK. Efficacy of fear appeals on adoption of COVID-19 preventive measures: a case of boda boda riders in Nairobi county, Kenya. Int J Res Granthaalayah 2020 Jun;8(6):219-228. [CrossRef]
- Pennycook G, McPhetres J, Zhang Y, Lu J, Rand D. Fighting COVID-19 misinformation on social mediaxperimental evidence for a scalable accuracy-nudge intervention. PsyArXiv Preprints. Preprint posted online March 17, 2020. [CrossRef]
- Sumnall HR, Bellis MA. Can health campaigns make people ill? The iatrogenic potential of population-based cannabis prevention. J Epidemiol Community Health 2007 Nov;61(11):930-931 [FREE Full text] [CrossRef] [Medline]
- Allara E, Ferri M, Bo A, Gasparrini A, Faggiano F. Are mass-media campaigns effective in preventing drug use? A Cochrane systematic review and meta-analysis. BMJ Open 2015 Sep 03;5(9):e007449 [FREE Full text] [CrossRef] [Medline]
- Lane HC, Fauci AS. Research in the context of a pandemic. N Engl J Med 2020 Jul 17 [FREE Full text] [CrossRef] [Medline]
- Nadelson L, Jorcyk C, Yang D, Jarratt Smith M, Matson S, Cornell K, et al. I just don't trust them: the development and validation of an assessment instrument to measure trust in science and scientists. Sch Sci Math 2014 Jan 19;114(2):76-86. [CrossRef]
- Chambon M, Dalege J, Elberse J, Harreveld F. A psychological network approach to factors related to preventive behaviors during pandemics: A European COVID-19 study. PsyArXiv Preprints. Preprint posted online July 01, 2020. [FREE Full text] [CrossRef]
- Shumaker L. Florida sets one-day record with over 15,000 new COVID cases, more than most countries. Reuters. 2020 Jul 12. URL: https://www.reuters.com/article/us-health-coronavirus-usa-idUSKCN24D0HE [accessed 2020-07-16]
- Tanner C. In separate rallies, Utahns protest mask mandate and demand in-person classes. The Salt Lake Tribune. 2020 Jul 15. URL: https://www.sltrib.com/news/education/2020/07/15/packed-meeting-utah/ [accessed 2020-07-16]
- Hatfield M. Masks off! Protestors claim face mask order unconstitutional. ABC13 Eyewitness News. 2020 Jun 28. URL: https://abc13.com/face-mask-order-protest-coronavirus-covid-19/6277878/ [accessed 2020-07-16]
- Palan S, Schitter C. Prolific.ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance 2018 Mar;17:22-27. [CrossRef]
- Peer E, Brandimarte L, Samat S, Acquisti A. Beyond the Turk: alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology 2017 May;70:153-163. [CrossRef]
- Representative samples on Prolific. Prolific - Researcher Help Centre. 2019 Mar 5. URL: https://researcher-help.prolific.co/hc/en-gb/articles/360019236753-Representative-Samples-on-Prolific-limited-release- [accessed 2020-07-16]
- Prolific. Prolific vs. MTurk. Prolific - Researcher Help Centre. URL: https://www.prolific.co/prolific-vs-mturk/ [accessed 2020-07-19]
- Geldsetzer P. Knowledge and perceptions of COVID-19 among the general public in the United States and the United Kingdom: a cross-sectional online survey. Annals of Internal Medicine 2020 Jul 21;173(2):157-160. [CrossRef]
- Kim HS, Hodgins DC. Are you for real? Maximizing participant eligibility on Amazon's Mechanical Turk. Addiction 2020 Oct;115(10):1969-1971. [CrossRef] [Medline]
- Huang G, Li K, Li H. Show, not tell: The contingency role of infographics versus text in the differential effects of message strategies on optimistic bias. Science Communication 2019 Nov 19;41(6):732-760. [CrossRef]
- Rodríguez Estrada FC, Davis LS. Improving visual communication of science through the incorporation of graphic design theories and practices into science communication. Science Communication 2014 Dec 26;37(1):140-148. [CrossRef]
- Lazard A, Atkinson L. Putting environmental infographics center stage: The role of visuals at the elaboration likelihood model’s critical point of persuasion. Science Communication 2014 Oct 30;37(1):6-33. [CrossRef]
- Dobos AR, Orthia LA, Lamberts R. Does a picture tell a thousand words? The uses of digitally produced, multimodal pictures for communicating information about Alzheimer's disease. Public Underst Sci 2015 Aug;24(6):712-730. [CrossRef] [Medline]
- Krakow M, Yale R, Jensen J, Carcioppolo N, Ratcliff C. Comparing mediational pathways for narrative- and argument-based messages: believability, counterarguing, and emotional reaction. Human Communication Research 2018;44(3):299-321. [CrossRef]
- Herzberg KN, Sheppard SC, Forsyth JP, Credé M, Earleywine M, Eifert GH. The Believability of Anxious Feelings and Thoughts Questionnaire (BAFT): a psychometric evaluation of cognitive fusion in a nonclinical and highly anxious community sample. Psychol Assess 2012 Dec;24(4):877-891. [CrossRef] [Medline]
- Lynas M. COVID: Top 10 current conspiarcy theories. Cornell - Alliance for Science. URL: https://allianceforscience.cornell.edu/blog/2020/04/covid-top-10-current-conspiracy-theories/ [accessed 2020-05-25]
- Coronavirus disease (COVID-19) advice for the public: Mythbusters. World Health Organization. URL: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public/myth-busters#oxygen [accessed 2020-09-07]
- Kaufman BG, Whitaker R, Mahendraratnam N, Smith VA, McClellan MB. Comparing associations of state reopening strategies with COVID-19 burden. J Gen Intern Med 2020 Oct 06 [FREE Full text] [CrossRef] [Medline]
- Cheng VC, Wong S, Chuang VW, So SY, Chen JH, Sridhar S, et al. The role of community-wide wearing of face mask for control of coronavirus disease 2019 (COVID-19) epidemic due to SARS-CoV-2. J Infect 2020 Jul;81(1):107-114 [FREE Full text] [CrossRef] [Medline]
- Centers for Disease Control and Prevention - Coronavirus Disease 2019 (COVID-19). URL: https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html [accessed 2020-07-17]
- Ajzen I. Theory of Planned Behavior Questionnaire. Measurement Instrument Database for Social Science. URL: https://www.midss.org/sites/default/files/tpb.construction.pdf [accessed 2020-07-17]
- deBruin WB. Age differences in COVID-19 risk perceptions and mental health: evidence from a national U.S. survey conducted in March 2020. The Journals of Gerontology: Series B 2020 May 29 [FREE Full text] [CrossRef]
- Wolf EJ, Harrington KM, Clark SL, Miller MW. Sample size requirements for structural equation models: an evaluation of power, bias, and solution propriety. Educ Psychol Meas 2013 Dec;76(6):913-934. [CrossRef] [Medline]
- Randomizer. QualtricsXM. URL: https://www.qualtrics.com/support/survey-platform/survey-module/survey-flow/standard-elements/randomizer/ [accessed 2020-08-05]
- Graham JW. Missing data analysis: making it work in the real world. Annu Rev Psychol 2009;60:549-576. [CrossRef] [Medline]
- Freedman DA. On the so-called “Huber Sandwich Estimator” and “Robust Standard Errors”. The American Statistician 2006 Nov;60(4):299-302. [CrossRef]
- Tein J, Coxe S, Cham H. Statistical power to detect the correct number of classes in latent profile analysis. Struct Equ Modeling 2013 Oct 01;20(4):640-657 [FREE Full text] [CrossRef] [Medline]
- Berlin K, Williams N, Parra G. An introduction to latent variable mixture modeling (part 1): overview and cross-sectional latent class and latent profile analyses. J Pediatr Psychol 2014 Mar;39(2):174-187. [CrossRef] [Medline]
- Biesanz JC, Falk CF, Savalei V. Assessing mediational models: testing and interval estimation for indirect effects. Multivariate Behav Res 2010 Aug 06;45(4):661-701. [CrossRef] [Medline]
- Ball P, Maxmen A. The epic battle against coronavirus misinformation and conspiracy theories. Nature 2020 May;581(7809):371-374. [CrossRef] [Medline]
- Coronavirus misinformation needs researchers to respond. Nature 2020 May;581(7809):355-356. [CrossRef] [Medline]
- Internet/broadband fact sheet. Pew Research Center - Internet & Technology. URL: https://www.pewresearch.org/internet/fact-sheet/internet-broadband/ [accessed 2020-08-05]
- Vraga EK, Jacobsen KH. Strategies for effective health communication during the coronavirus pandemic and future emerging infectious disease events. World Medical & Health Policy 2020 Jul 29;12(3):233-241. [CrossRef]
- Chandler J, Shapiro D. Conducting clinical research using crowdsourced convenience samples. Annu Rev Clin Psychol 2016;12:53-81. [CrossRef] [Medline]
- Merz ZC, Lace JW, Eisenstein AM. Examining broad intellectual abilities obtained within an mTurk internet sample. Curr Psychol 2020 Apr 22. [CrossRef]
- Keith MG, Tay L, Harms PD. Systems perspective of Amazon Mechanical Turk for organizational research: review and recommendations. Front Psychol 2017;8:1359 [FREE Full text] [CrossRef] [Medline]
|CDC: Centers for Disease Control and Prevention|
|LMM: linear mixed model|
|mTurk: Amazon Mechanical Turk|
|NPB: nonpharmaceutical preventive behavior|
Edited by G Eysenbach; submitted 16.09.20; peer-reviewed by L Sbaffi, PCI Pang; comments to author 07.11.20; revised version received 10.11.20; accepted 10.11.20; published 07.12.20Copyright
©Jon Agley, Yunyu Xiao, Esi E Thompson, Lilian Golzarri-Arroyo. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 07.12.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.