Published on in Vol 6, No 7 (2017): July

An Intervention Using Gamification to Increase Human Immunodeficiency Virus and Sexually Transmitted Infection Screening Among Young Men Who Have Sex With Men in California: Rationale and Design of Stick To It

An Intervention Using Gamification to Increase Human Immunodeficiency Virus and Sexually Transmitted Infection Screening Among Young Men Who Have Sex With Men in California: Rationale and Design of Stick To It

An Intervention Using Gamification to Increase Human Immunodeficiency Virus and Sexually Transmitted Infection Screening Among Young Men Who Have Sex With Men in California: Rationale and Design of Stick To It


1Public Health Division, AIDS Healthcare Foundation, Los Angeles, CA, United States

2Goldman School of Public Policy, University of California, Berkeley, CA, United States

3School of Public Health, University of California, Berkeley, CA, United States

4David Geffen School of Medicine, University of California, Los Angeles, CA, United States

5Fielding School of Public Health, University of California, Los Angeles, CA, United States

Corresponding Author:

Sandra I McCoy, PhD, MPH

School of Public Health

University of California

779 University Hall

MS #7360

Berkeley, CA, 94720-7360

United States

Phone: 1 510 642 0513


Background: In the United States, young men who have sex with men (YMSM) remain disproportionately affected by human immunodeficiency virus (HIV) and other sexually transmitted infections (STIs). Although routine HIV/STI screening is pivotal to the timely diagnosis of HIV and STIs, initiation of appropriate treatment, and reduced onward disease transmission, repeat screening is underused. Novel interventions that incorporate elements of games, an approach known as gamification, have the potential to increase routinization of HIV/STI screening among YMSM.

Objective: The study aims to test the hypothesis that an incentive-based intervention that incorporates elements of gamification can increase routine HIV/STI screening among YMSM in California.

Methods: The study consists of a formative research phase to develop the intervention and an implementation phase where the intervention is piloted in a controlled research setting. In the formative research phase, we use an iterative development process to design the intervention, including gathering information about the feasibility, acceptability, and expected effectiveness of potential game elements (eg, points, leaderboards, rewards). These activities include staff interviews, focus group discussions with members of the target population, and team meetings to strategize and develop the intervention. The final intervention is called Stick To It and consists of 3 components: (1) online enrollment, (2) Web-based activities consisting primarily of quizzes and a countdown “timer” to facilitate screening reminders, and (3) in-person activities that occur at 2 sexual health clinics. Participants earn points through the Web-based activities that are then redeemed for chances to win various prizes during clinic visits. The pilot study is a quasi-experimental study with a minimum of 60 intervention group participants recruited at the clinics, at community-based events, and online. We will compare outcomes in the intervention group with a historical control group consisting of individuals meeting the inclusion criteria who attended study clinics in the 12 months prior to intervention implementation. Eligible participants in the pilot study (1) are 18 to 26 years old, (2) were born or identify as male, 3) report male sexual partners, and 4) have a zip code of residence within defined areas in the vicinity of 1 of the 2 implementation sites. The primary outcome is repeat HIV/STI screening within 6 months.

Results: This is an ongoing research study with initial results expected in the fourth quarter of 2017.

Conclusions: We will develop and pilot test a gamification intervention to encourage YMSM to be regularly screened for HIV/STIs. The results from this research will provide preliminary evidence about the potential effectiveness of using gamification to amplify health-related behavioral change interventions. Further, the research aims to determine the processes that are essential to developing and implementing future health-related gamification interventions.

Trial Registration: NCT02946164; (Archived by WebCite at

JMIR Res Protoc 2017;6(7):e140



New and innovative strategies for human immunodeficiency virus (HIV) infection prevention are urgently needed to increase the uptake of health services among men who have sex with men (MSM). A growing body of evidence suggests that, under the right circumstances, financial incentives can increase the demand for HIV screening, change short-term sexual behavior, enhance linkage to care after HIV diagnosis, and promote adherence to antiretroviral therapy [1-11]. Several studies have demonstrated that incorporating elements of games into incentive-based programs—an approach known as gamification—can be more effective and cost effective than simple financial incentives alone [12-14]. Ongoing and completed studies in the United States and elsewhere, including several targeting MSM, have demonstrated the feasibility and acceptability of gamification for improving engagement in HIV infection prevention and care [15-20].

We hypothesize that gamification for HIV and sexually transmitted infection (STI) prevention may work well to reengage and motivate a new generation of MSM in a cost-effective and sustainable way. A focus on young men who have sex with men (YMSM) is particularly warranted because this population remains disproportionally affected by HIV infection. Although HIV diagnoses among gay and bisexual men have stabilized in recent years, YMSM continue to experience the greatest burden of HIV compared with any other group in the United States, with young men 13 to 24 years of age accounting for 27% of new diagnoses among all gay and bisexual men [21]. Additionally, MSM are at increased risk for STIs [22]. In 2015, 59.6% of all primary and secondary syphilis diagnosis were among MSM [22]. Rectal chlamydia and gonorrhea are also common among MSM and are associated with increased HIV acquisition [23-25]. Moreover, a growing body of evidence suggests that the Internet and social media are effective ways to share sexual health information with MSM, and YMSM in particular [26-33]. To our knowledge, at least two gamification interventions for diverse YMSM in the United States are underway or have been completed, including an intervention to reduce sexual risk behaviors (healthMpowerment) [15,16] and a mobile phone-based intervention to improve antiretroviral therapy adherence (Epic Allies) [17].

To test the potential of gamification for HIV/STI prevention, our 2-year study developed and will pilot Stick To It, an HIV/STI prevention intervention for YMSM (18-26 years of age) that incorporates elements of gamification. The objective of the intervention is to increase repeat HIV/STI screening, defined as screening at least every 3 months. HIV/STI screening is critical both as the gateway to HIV/STI treatment, which can lead to reduced onward transmission, and as a critical first step to access prevention strategies such as preexposure prophylaxis [34]. This paper describes the methodology and protocol of our study.

The study consists of 2 distinct phases. The first phase is the formative research phase, in which we used focus groups, structured interviews, and rapid prototyping of intervention elements to determine the most effective and appropriate design for the intervention. The second phase consists of the implementation of the intervention in a controlled research design. First, we present the methodology of the formative phase and the insights gained from the different elements therein, leading to the final design of the intervention. Second, we present the protocol for the implementation phase.

Phase 1: Formative Intervention Design

Our intervention was implemented at 2 sexual health clinics, 1 in northern California (Oakland), and 1 in southern California (Hollywood). Phase 1 formative research took place at the 2 clinics and at meeting locations on a university campus from November 2015 to July 2016. Our goal in phase 1 was to design Stick To It, an intervention incorporating gamification intended to motivate increased repeat HIV/STI screening among YMSM. Gamification, “the use of game design elements in non-game contexts,” [35] is hypothesized to amplify the motivational power of financial and nonfinancial incentives. It is informed by self-determination theory, which posits that external rewards can be internalized and generate lasting intrinsic motivation (defined as engaging in activities “because of the positive feelings resulting from the activities themselves”) if they are experienced in a context that satisfies three basic psychological needs: autonomy, competence, and relatedness [36]. Gamification scholars have argued that games and gamification can create such a context [36,37]. At the beginning of phase 1, we selected the key game elements that we intended to use in the intervention. These included a defined theme and a number of core “game mechanics,” basic mechanisms that define how the gamification intervention works. The game theme is a concept that serves to synchronize all components of the game and is critical to maximizing participant engagement [38]. Figure 1 outlines how potential gamification interventions may facilitate improved health outcomes among YMSM.

Because the formative phase of gamification design involves an iterative process of gathering information about the effectiveness of game elements and refining the intervention, we simultaneously present both the design of the formative phase of our study and the results generated by each stage of the formative research.

Figure 1. Theory of change for how a potential gamification intervention may facilitate improved health outcomes among young men who have sex with men (YMSM). HIV: human immunodeficiency virus; STI: sexually transmitted infection.
View this figure
Formative Phase Methods and Results

The phase 1 research activities consisted of 4 components: (1) 10 strategy and development meetings of the entire project team, consisting of the research team, clinic staff, and a gamification consultant from the private sector; (2) 4 in-depth interviews with clinic staff about the clinic and its population; (3) 11 focus groups with 29 members of the target population (YMSM, 18-26 years of age); and (4) a series of weekly meetings of a smaller design team, consisting of 3 members of the research team and 1 of the site coordinators.

Strategy and Development Meetings

We held 10 meetings of the entire project team. Participants were the research team, including the project manager, the 2 site-specific project coordinators, the regional coordinators and research director of the implementation partner, and the gamification consultant. At the initial meetings, the emphasis was on broad game design-related issues such as the theme and aesthetics of the intervention design. Subsequent meetings consisted of extensive discussion of the game elements to be included, such as how to use points; whether to use a leaderboard; the frequency of participant interaction with the intervention; how much interaction between participants to include, if any; when or if to incorporate prizes; and how to use chance elements to determine prizes or other outcomes. Over time, these discussions were increasingly informed by the focus group findings. An important goal of these discussions was to integrate (1) extensive testing of game elements among the target population to determine the most suitable, engaging, and effective intervention design, as this is the cornerstone of gamification [38]; (2) knowledge of gamification and game design principles; (3) the research team’s expertise on established public health behavior change strategies, such as reminders, incentives, and education; and (4) clinic staff’s expertise on patient experience, clinic management, and overall feasibility, including the logistical realities and constrains of the participating clinics.

Staff Interviews

We conducted structured in-depth interviews with 4 staff members from both implementation sites. Interviewees were purposely selected to include testing counselors and clinic managers. The primary purpose of the interviews was to assess the kinds of game elements that would be feasible, appropriate, and best suited to increase engagement between clinic visits based on staff knowledge of the institutional setting and target population. The primary game elements discussed were theme and core game mechanics. The potential core game mechanics that we initially considered were (1) games of chance (eg, dice or spin wheels, gumball machines) and a system of points awarded for achievement of tasks and goals within the intervention, (2) prizes awarded based on accumulated points and outcomes of the chance elements, and (3) leaderboards that display the relative ranking of participants’ points or accomplishments and are used to create an element of competition. We used the insights gained from the staff interviews to develop the set of potential themes, core mechanics, and game elements presented at the initial focus groups. We also asked staff members to briefly pilot the games of chance in the clinic and report their perspectives about the feasibility and acceptability of each approach.

Focus Groups

After staff interviews, we conducted an initial set of 4 focus groups with the target population in both implementation areas. These discussions followed a standard structured format designed to elicit detailed information about preferences and characteristics, with particular focus on use of smartphones and Internet technology, as well as experiences with different types of games. Special emphasis was placed on issues of concern to race and ethnic minorities and the lesbian, gay, bisexual, transgender, questioning community. In addition, we tested the themes and game mechanics that emerged from the staff interviews with focus group participants to determine those that would be most appropriate and effective.

We conducted 7 additional focus groups, which focused more narrowly on the specific game elements of the intervention and followed a progression of increasing detail and specificity of game elements as the design and content of the intervention evolved. This stage of the formative research was intended to implement the iterative game design testing process, an important component of game testing [38]. In these focus groups, we presented mock-ups of specific game elements, in the sequence in which they would be encountered by participants in the intervention, and elicited input on each of the elements and their interaction in the overall intervention design. An additional procedure was to present multiple versions of a particular game element and elicit rankings of the alternatives in terms of potential engagement and effectiveness. While we emphasized the effectiveness of game elements and their potential for participant engagement, we also prioritized issues of sensitivity and appropriateness for the target population during focus group discussions. Namely, we aimed to ensure that the proposed game mechanics and messaging were nonstigmatizing, meaningful, and relevant to the target population. Focus group facilitators used structured focus group guides but encouraged a flexible conversation based on participant input during the sessions to facilitate friendly and open communication that allowed participants to be honest about their experiences with sexual health screenings and game design. Participant input on novel designs of specific game elements or the intervention as a whole was also encouraged.

Among the 11 focus groups held, a total of 29 participants attended, with between 2 and 9 participants per group session. The mean age of participants was 24 years old. The self-identified race/ethnicity of focus group participants was as follows: American Indian/Alaskan native: 1 (3%); Asian/Pacific Islander: 3 (10%); black/African American: 4 (14%); Hispanic/Latino: 10 (34%); white: 9 (31%); mixed or other race/ethnicity: 2 (7%).

Table 1 summarizes the key results of the focus groups: the game elements that were discussed, the main insights gained with respect to each element, and the decisions that were made about each element in the iterative design process.

Table 1. Focus group results: key insights and decisions about game elements to be used in the Stick To It intervention for young men who have sex with men (YMSM) in California, 2015-2016.
Game elementKey insights from focus group discussionsDesign decisions for Stick To It
ThemePrefer contemporary to retro themes.
Prefer fun, lighthearted themes to highly sexualized themes.
Prefer simplicity in design.
Base theme on experiences and interests of YMSM.
Use bright colors, whimsical humor, and fun content.
Reminders to screen at self-scheduled timeReminders need to be fun and rewarding.
Prefer text messages (SMSa) over email and social media-based reminders.
Frequency matters.
Send communications via SMS.
Link to a fun online activity for points.
Decrease communication intervals from 3 weeks to 1 week between screenings.
Incorporate a “countdown timer” to remind participants when next quarterly screening is due.
Web-based activities to accompany reminders and earn pointsPrefer polls and quizzes to games.
Activities should not be competitive.
Interest was expressed in useful information about sexual health with a fun or whimsical approach.
Time is limited, so online activities should be designed to be completed in a short period of time, such as waiting in line, commuting to work or school.
Use multiple-choice quizzes, with points for answering questions, and additional points for correct answers.
Limit questions to 5 per quiz.
Mix informational with fun and whimsical activities.
Make questions short and to the point.
Chance element for awarding prizes for screeningGuarantee participants always win at least a small prize as a means to deflect from the anxiety and stress associated with screening encounters.
Prefer a combination of high-probability small prizes and low-probability large prizes.
Participants need to believe the chance mechanism isn’t rigged.
Use gumball machine at the clinic as a game of chance.
More points lead to more gumballs.
Base prizes on color combinations of gumballs.
Award small prizes for common combinations and large prizes for rare combinations.

aSMS: short message service.

Design Team Meetings

The smaller design team held a series of weekly meetings. The design team iteratively redesigned components of the intervention on the basis of the results of the other formative research components. Each meeting of the design team generated a revised set of game elements, which were then taken into the subsequent focus groups and strategy and development meetings for consideration.

Final Intervention Design

The final intervention consists of 3 components: online enrollment, Web-based activities, and in-person activities that occur at the clinic. Participants earn points through the Web-based activities, which they then redeem for a chance to win prizes during clinic visits. All content was developed prior to the implementation phase of the intervention.

In the first component, participants go through an online enrollment process and introduction to the intervention. The enrollment process consists of answering study eligibility questions, providing consent to participate in the study, and answering basic demographic questions. The introduction to the intervention consists of reading a short tutorial, inputting the date of last HIV/STI screening—used to set the countdown timer for the next recommended screening—and being invited to answer a 5-question multiple-choice quiz on a topic related to sexual health. Quizzes include questions that test knowledge of sexual health information that members of the target population may value (eg, STI symptoms, treatment, transmission risk, and prevention strategies) and questions primarily intended to be humorous or whimsical. Participants earn points for each step of this process (except for providing consent).

The second component, Web-based activities, occur on the intervention website ( and are intended to accomplish two primary goals. The first is to provide multiple reminders of the approaching quarterly screening date, which is accomplished through the prominent display of the countdown timer on the personalized webpage dashboard using a graphic presentation that was designed as part of the aesthetic theme of the intervention. The second is to provide participants the opportunity to accumulate points, which increases their chance of winning more valuable prizes at the clinic, thus increasing their motivation to receive quarterly HIV/STI screening. The online activities are primarily triggered by a series of short message service (SMS) text messages, inviting participants to visit the intervention website and complete various activities, such as taking the latest quiz, viewing their screening countdown timer, and inviting friends to join the intervention. Messages are sent with increasing frequency as the participant approaches their quarterly screening date, beginning with a 3-week interval, increasing to a 1-week interval. To complete each quiz, participants click a link in the SMS message directing them to their home page on the intervention website, where participants can also view their countdown timer. Points are awarded for answering the quiz questions, with additional points awarded for answering correctly. Participants are provided with an opportunity to take a new quiz every 3 weeks throughout the course of the intervention. In addition, participants earn points when they invite eligible friends who subsequently enroll in the intervention. Invited friends must meet all participant inclusion criteria.

The third component of the intervention involves the game of chance and takes place at the clinic. Participants also receive additional points for visiting the clinic for screening. Participants’ accumulated points are redeemable for prizes only at the clinic (for any reason, including screening or treatment). The redemption process works as follows. Gumball machines are located at each clinic. This was both the game of chance overwhelmingly preferred by the target population through our focus group testing and the most feasible to implement at the clinics. During their clinic visit, participants can draw between 1 and 5 gumballs, depending on their accumulated points. The number of points earned corresponds with the number of gumball draws available to the participant. The color combinations of the gumballs that are drawn determine the prize received. Points redeemed for gumball draws are deducted from the participant’s accumulated points at the time of redemption. The system of points and prizes was designed to result in an average prize cost of US $5 per screening visit. After screening, the countdown timer is reset for the next quarterly screening date. Multimedia Appendix 1 presents the points system and prizes used in the intervention. Multimedia Appendix 2 presents screenshots of the intervention website. Figure 2 shows a schematic of the final intervention design.

Figure 2. Schematic of final intervention design. HIV: human immunodeficiency virus; STI: sexually transmitted infection.
View this figure

Phase 2 Intervention Pilot

Phase 2 will take place from October 2016 to June 2017 and will address the following aims: (1) evaluate the preliminary effectiveness of the Stick To It intervention on repeat HIV/STI screening in a pilot study, (2) evaluate the acceptability of and participant engagement with the Stick To It intervention, (3) evaluate the feasibility and cost of implementing an effectiveness evaluation on a larger scale, and (4) evaluate the cultural competence of the intervention as designed and implemented.

Study Population and Eligibility Criteria

Our study targets individuals who (1) are 18 to 26 years old, (2) were born or identify as male, (3) report male sexual partners at the time of enrollment, and (4) have a zip code of residence within defined areas in the vicinity of 1 of the 2 implementation sites. We selected YMSM aged 18 to 26 years as inclusion criteria for this pilot study given the substantial variations in the intervention design that would be required to include adolescent-aged YMSM. Additionally, a minimum inclusion age of 18 years will ensure that participants can provide independent consent to participate in the intervention.

Recruitment, Screening, and Sample Size

Participants will be recruited through three main channels. Multimedia Appendix 3 (flyers) and Multimedia Appendix 4 (video) present the recruitment tools used for the intervention.

In-Clinic Recruitment

Clinic staff will identify eligible individuals who visit the clinics for screenings, treatment, or other purposes and will inform them of the study and invite them to participate. In addition, placing flyers in the clinics and displaying the gumball machines in examination rooms will facilitate conversations about the intervention with potential participants. Those who are interested will be given a handout directing them to the intervention website, where they will be invited to complete an eligibility questionnaire and, if eligible, provide consent to participate in the study. Those with a smartphone will be invited to sign up while still at the clinic. Those who are eligible and provide consent will enter the study.

Community-Based Recruitment

We will place flyers in bars, other clinics, on mobile testing vans, and at selected community events. The advertisements and flyers include links to the intervention website, where individuals are invited to take the eligibility questionnaire and provide consent.

Online Recruitment

We will post advertisements on social networking platforms such as Facebook, Twitter, Instagram, Craigslist, and Grindr (a geolocation-based social networking app used by YMSM).

Study Design

Owing to potential spillover effects between the treatment and control groups within local peer groups, we deemed a randomized design to be impracticable. Instead, our study will use a quasi-experimental study design with a historical control group consisting of eligible participants identified within the medical records of the same clinics over the 12 months prior to the intervention phase of the study. We preregistered the study ( NCT02946164). All eligible individuals recruited at the clinics, at community-based events, and online will be assigned to the treatment group. We will recruit a minimum of 60 participants and a maximum of 200 participants into the treatment group and identify an identical number of individuals to serve as a historical control group. To our knowledge, this will be the first intervention using gamification to target YMSM for repeat HIV/STI screening. Accordingly, we chose a large sample size range as the upper bound, as it is not possible for us to predict what the demand will be for such an intervention among the target population.

Given the pilot nature of the study, the proposed analyses will likely be underpowered for formal testing purposes. Nevertheless, if we assume that half of the men in the comparison group will attend 1 or more repeat HIV/STI screening visits over 6 months of follow-up, with 60 men in each group, we will have 80% power to detect a minimum change of at least 27 percentage points (from 50% to 77%), which is comparable to the increase observed in an incentive-based intervention previously implemented by the community-based organization that operates the 2 clinics in our study [39].


For the primary outcome of repeat HIV/STI screening, we will use routinely collected client data from the clinics’ electronic patient tracking and medical record system. A set of quantitative and qualitative secondary outcomes will be used to determine the acceptability and feasibility of Stick To It, and whether the intervention is suitable for a future effectiveness study on a larger scale. These implementation outcomes are as follows. First is the ability to recruit the study population: the percentage of eligible men approached who are willing to enroll and whether we can fully enroll the cohort within 6 weeks of study initiation. Second is participant engagement with the intervention: among Stick To It participants, we will determine (1) the proportion who successfully invited 1 or more other eligible individuals to the intervention, (2) the number of completed quizzes, and (3) the average number of quizzes participants completed. Third is cost: we will track the incremental cost of the program, relative to standard of care, which we will identify using the ingredients approach, in accordance with guidance from the World Health Organization [40]. Fourth is the potential for adverse events; that is, whether any adverse events occurred and the potential for adverse events in a larger effectiveness study. Fifth is the cultural competence of staff who implemented Stick To It services and activities and the cultural relevance of the intervention for members of the target population.

Data Collection

For each participant, we will collect data for a period of 6 months following their entry into the study. The primary outcome is repeat HIV/STI screening, defined as the number of screenings participants receive over the 6-month observation period, an indicator of retention in the program and its effectiveness.

In addition, we will collect qualitative information about the experiences of participants and clinic staff, described below. We will use the following methods to collect our secondary outcomes.

Postpilot Questionnaire, Administered to All Participants

A survey offered to all participants at the end of the study period will assess HIV/STI screening at other clinics (self-reported), barriers and facilitators to screening, and perceptions of the Stick To It intervention.

In-Depth Interviews

We will conduct up to 10 in-depth interviews with members of the clinic staff at the end the study period. We will gather detailed information on their perceptions of the intervention, including implementation challenges and successes, adaptability and integration with clinic operations, perceptions of effectiveness, and suggestions for future improvements.

We will also conduct in-depth exit interviews with up to 40 Stick To It participants from both locations. Our goal is to gather detailed information about whether the intervention was relevant, motivating, and culturally competent. Further, we hope to assess the level of satisfaction with the process of earning and redeeming points, the rewards used, and the program’s potential applicability to others. We will purposefully select a diverse group of men from different racial and ethnic groups and men who had different levels of engagement with the intervention. Topical areas will include the participant’s perception of HIV/STI risk, perceptions about and experience with HIV/STI screening, willingness to discuss sexual health with others, particularly motivating elements of the game, and any unexpected challenges or adverse events.

We will examine whether Stick To It was culturally appropriate, relevant, and nonstigmatizing for YMSM, with an emphasis on African American and Latino YMSM, who are disproportionately affected by HIV [41]. Thus, at the completion of the in-depth interviews, we will administer a short interviewer-administered survey that includes the validated 20-item Public Perception of Physician’s Cultural Competence Scale, which measures client perceptions of physicians’ cultural competence [42]. We will adapt the scale to also refer to the clinic staff involved in Stick To It, and the design of the intervention itself. In addition, given the issues of stigma and discrimination that may adversely affect health care access among YMSM, we will include questions adapted from the validated Experiences of Discrimination [43] scale and the Discrimination in Medical Settings Scale [44] to determine whether the intervention or its implementation was stigmatizing.

In-depth interviews will be conducted in English by trained staff and will follow standard procedures [45,46]. A semistructured interview guide will cover predetermined issues (to ensure systematic data collection), but the interviewer will be free to change the sequence and wording of questions to ensure that unexpected themes can emerge [47]. Interviews will be audio recorded, with participant’s consent, and later transcribed verbatim.

Figure 3 summarized our study design, screening and recruitment process, and data collection.

Figure 3. Summary of study design, screening and recruitment process, and data collection.
View this figure

Data Analysis

For quantitative outcomes, we will first assess missing data, describe participant characteristics using frequency tables, and provide descriptive statistics (eg, means, standard deviations, medians, ranges, proportions) for the primary and secondary outcomes stratified by group. For the binary outcomes, we will compare the intervention and comparison groups by computing unadjusted and adjusted prevalence ratios and 95% confidence intervals using generalized linear models, such as log-binomial regression [48-50]. For the continuous and discrete outcomes, we will use ordinary least squares and zero-inflated Poisson regression (depending on the distribution of the outcomes) to determine the change in outcomes associated with participation in the intervention, before and after controlling for baseline characteristics. All regression analyses will include robust standard errors to adjust for heteroscedasticity and clustering. We will also perform a per-protocol analysis limited to participants that had a minimum level of engagement with the intervention during the study period (eg, ≥25th percentile of points earned during the game).

Qualitative data analysis will be conducted with ATLAS.ti (Scientific Software Development GmbH) following standard procedures [47,48]. In brief, we will combine inductive and deductive techniques to strengthen the validity of the coding system and our conclusions during the multistage analysis process [46-50]. Two researchers will independently review the transcripts and code according to broad, a priori, and emergent themes related to gamification theory and barriers and facilitators of regular screening. Interviewer notes capturing contextual details, quality, inconsistencies, and impressions will also be incorporated. Data reduction will be based on coding sorts of the most central themes (eg, engagement with the intervention) followed by a systematic analysis of related themes (eg, motivation, risk perception, stigma) using coding matrices to identify relationships [44,51]. To reduce threats to validity, we will solicit feedback on our hypotheses from participants (respondent validation) [52] and rigorously examine theoretical validity—whether the data are consistent or inconsistent with the underlying theoretical models [47].

This is an ongoing research study with initial results expected in the fourth quarter of 2017.

The ongoing study is intended to explore the effectiveness of incorporating gamification into an incentive-based intervention to increase repeat HIV/STI screening among YMSM. Given that this population remains disproportionally affected by HIV and STIs, novel approaches that supplement proven engagement strategies may serve an important public health function. Namely, early HIV diagnosis, as achieved through repeat HIV screening, has been shown to facilitate timely linkage to medical care, initiation of antiretroviral therapy, and attainment of viral suppression, resulting in reduced transmission risk [53-55].

Given the success of nongamification interventions involving incentives, and considering the potential of gamification to harness and amplify intrinsic motivation for both short-term and long-term behavior change, we hypothesize that this approach may be particularly effective in increasing routine HIV/STI screening among YMSM. We emphasize that a fundamental principle of gamification is to enhance the motivational effectiveness of known behavior change mechanisms, such as reminders and incentives, and to amplify the existing motivation of individuals to accomplish desired behaviors. However, we do not expect stand-alone gamification interventions to replace or compete with existing types of interventions, but rather to enhance established strategies. Accordingly, we coupled gamification with an incentive-based structure for this very reason.

A secondary but important goal of this study is to explore the process of designing gamification interventions for health-related behavior change and to learn best practices for future intervention design. This is crucial, due to the fact that gamification interventions potentially require an extensive formative design process to determine the optimal set of game elements and intervention design required to incorporate such elements. One conjecture is that the success of any given health-related gamification intervention, and thus of health-related gamification more generally, may depend on highly context-specific and population-specific formative research. Thus, determining best practices for the formative phase is necessary to achieve the goal of being able to consistently and predictably design cost-effective interventions in a range of different circumstances. This paper, in addition to presenting the protocol of the study, is an attempt at documenting what formative research to develop a gamification-based intervention entails.

It is also important to recognize the limitations of this study. As previously discussed, we will use a historical control group for comparison. Although a randomized design has advantages, it was not practical because participants randomly assigned to the control group would have been exposed to some components of the intervention, such as recruitment materials posted in the clinic waiting and examination rooms (eg, comparison group contamination). Additionally, given the pilot state of the intervention, a cluster randomized design was neither warranted nor feasible. Although a historical comparison group will help us understand whether there is a preliminary signal of the intervention’s effectiveness and whether further study is warranted, it has important limitations. Most importantly, we will be unable to make causal statements about the effect of the intervention because we will be unable to adequately control for time trends between the intervention and comparison groups and the selection bias that may result from differences between the 2 groups. Nevertheless, a historical comparison group will still provide us with baseline data on repeat testing among YMSM in the same geographic region and therefore a benchmark to which we can make preliminary comparisons about the intervention’s potential effectiveness.


The authors thank the following individuals for their contributions to this work: Sagar Amin; Branden Barger; Dale Gluth, MA; Jamie Murkey, MPH; Sean Young, MS, PhD; Gabe Zichermann; and the Stick To It Community Advisory Board. Publication was made possible in part by support from the Berkeley Research Impact Initiative sponsored by the University of California, Berkeley Library. Funding for the study was provided by the National Institute of Mental Health (McCoy R34MH106359). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health or the National Institutes of Health.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Stick To It points system/prize decoder.

PDF File (Adobe PDF File), 181KB

Multimedia Appendix 2

Intervention website screenshots.

PDF File (Adobe PDF File), 746KB

Multimedia Appendix 3

Recruitment tools (flyers).

PDF File (Adobe PDF File), 7MB

Multimedia Appendix 4

Recruitment tools (video).

MP4 File (MP4 Video), 10MB

  1. McCoy SI, Shiu K, Martz TE, Smith CD, Mattox L, Gluth DR, et al. Improving the efficiency of HIV testing with peer recruitment, financial incentives, and the involvement of persons living with HIV infection. J Acquir Immune Defic Syndr 2013 Jun 01;63(2):e56-e63. [CrossRef] [Medline]
  2. Björkman-Nyqvist M, Corno L, de Walque D, Svensson J. Evaluating the impact of short term financial incentives on HIV and STI incidence among youth in Lesotho: a randomized trial. 2013 Presented at: 7th IAS Conference on HIV Pathogenesis, Treatment, and Prevention; June 30-July 3, 2013; Kuala Lumpur, Malaysia.
  3. Baird SJ, Garfein RS, McIntosh CT, Ozler B. Effect of a cash transfer programme for schooling on prevalence of HIV and herpes simplex type 2 in Malawi: a cluster randomised trial. Lancet 2012 Apr 07;379(9823):1320-1329. [CrossRef] [Medline]
  4. de Walque D, Dow WH, Nathan R, Abdul R, Abilahi F, Gong E, et al. Incentivising safe sex: a randomised trial of conditional cash transfers for HIV and sexually transmitted infection prevention in rural Tanzania. BMJ Open 2012;2:e000747 [FREE Full text] [CrossRef] [Medline]
  5. Galárraga O, Genberg BL, Martin RA, Barton Laws M, Wilson IB. Conditional economic incentives to improve HIV treatment adherence: literature review and theoretical considerations. AIDS Behav 2013 Sep;17(7):2283-2292 [FREE Full text] [CrossRef] [Medline]
  6. Solomon SS, Srikrishnan AK, Vasudevan CK, Anand S, Kumar MS, Balakrishnan P, et al. Voucher incentives improve linkage to and retention in care among HIV-infected drug users in Chennai, India. Clin Infect Dis 2014 Aug 15;59(4):589-595 [FREE Full text] [CrossRef] [Medline]
  7. Haukoos JS, Witt MD, Coil CJ, Lewis RJ. The effect of financial incentives on adherence with outpatient human immunodeficiency virus testing referrals from the emergency department. Acad Emerg Med 2005 Jul;12(7):617-621 [FREE Full text] [CrossRef] [Medline]
  8. Thornton RL. The demand for, and impact of, learning HIV status. Am Econ Rev 2008 Dec 01;98(5):1829-1863 [FREE Full text] [CrossRef] [Medline]
  9. Sibanda EL, Tumushime M, Mufuka J, Gudukeya S, Napierala Mavedzenge S, Bautista-Arredondo S, et al. Results of a cluster-randomized trial of non-financial incentives to increase uptake of couples counseling and testing among clients attending PSI mobile HIV services in rural Zimbabwe. J Int AIDS Soc 2016 Jul 22;19(Suppl 5):62. [CrossRef]
  10. McNairy ML, Gachuhi AB, Lamb MR, Nuwagaba-Biribonwoha H, Burke S, Ehrenkranz P, et al. The Link4Health study to evaluate the effectiveness of a combination intervention strategy for linkage to and retention in HIV care in Swaziland: protocol for a cluster randomized trial. Implement Sci 2015 Jul 19;10:101 [FREE Full text] [CrossRef] [Medline]
  11. Björkman Nyqvist M, Corno L, de Walque D, Svensson J. Using lotteries to incentivize safer sexual behavior: evidence from a randomized controlled trial on HIV prevention. Policy research working paper. Washington, DC: World Bank; 2015 Mar.   URL: https:/​/elibrary.​​action/​doSearch?AllField=Using+Lotteries+to+Incentivize+Safer+Sexual+Behavior%3A+Evidence+from+a+Randomized+Controlled+Trial+on+HIV+Prevention [accessed 2017-04-15] [WebCite Cache]
  12. Haisley E, Volpp KG, Pellathy T, Loewenstein G. The impact of alternative incentive schemes on completion of health risk assessments. Am J Health Promot 2012;26(3):184-188. [CrossRef] [Medline]
  13. Ashraf N, Bandiera O, Jack K. No margin, no mission? A field experiment on incentives for pro-social tasks. 2012 Jan 24.   URL: [accessed 2017-04-15] [WebCite Cache]
  14. Mehrotra A, An R, Patel DN, Sturm R. Impact of a patient incentive program on receipt of preventive care. Am J Manag Care 2014 Jun;20(6):494-501 [FREE Full text] [Medline]
  15. Pike E. Level Up! Using gamification to spark competition, collaboration,community in online interventions. 2014 Presented at: Youth + Tech + Health Live Conference; April 6-8, 2014; San Francisco, CA.
  16. Hightow-Weidman LB, Fowler B, Kibe J, McCoy R, Pike E, Calabria M, et al. development of a theory-based HIV/STI website for young black MSM. AIDS Educ Prev 2011 Feb;23(1):1-12 [FREE Full text] [CrossRef] [Medline]
  17. LeGrand S, Muessig KE, McNulty T, Soni K, Knudtson K, Lemann A, et al. Epic allies: development of a gaming app to improve antiretroviral therapy adherence among young HIV-positive men who have sex with men. JMIR Serious Games 2016 May 13;4(1):e6 [FREE Full text] [CrossRef] [Medline]
  18. Enah C, Moneyham L, Vance DE, Childs G. Digital gaming for HIV prevention with young adolescents. J Assoc Nurses AIDS Care 2013;24(1):71-80. [CrossRef] [Medline]
  19. Black S, Wallace M, Middelkoop K, Robbertze D, Bennie T, Wood R, et al. Improving HIV testing amongst adolescents through an integrated youth centre rewards program: insights from South Africa. Child Youth Serv Rev 2014 Oct;45:98-105. [CrossRef]
  20. Horvath KJ, Michael Oakes J, Simon Rosser B, Danilenko G, Vezina H, Rivet Amico K, et al. Feasibility, acceptability and preliminary efficacy of an online peer-to-peer social support ART adherence intervention. AIDS Behav 2013 Apr 4;17(6):2031-2044 [FREE Full text] [CrossRef] [Medline]
  21. Centers for Disease Control and Prevention. HIV Surveillance Report, 2014; vol. 26. Atlanta, GA: U.S. Department of Health and Human Services; 2015.   URL: [accessed 2017-04-10] [WebCite Cache]
  22. Centers for Disease Control and Prevention. Sexually Transmitted Disease Surveillance 2015. Atlanta, GA: U.S. Department of Health and Human Services; 2016.   URL: [accessed 2017-04-10] [WebCite Cache]
  23. Pathela P, Braunstein SL, Blank S, Schillinger JA. HIV incidence among men with and those without sexually transmitted rectal infections: estimates from matching against an HIV case registry. Clin Infect Dis 2013 Jun 25;57(8):1203-1209. [CrossRef] [Medline]
  24. Patton ME, Kidd S, Llata E, Stenger M, Braxton J, Asbel L, et al. Extragenital gonorrhea and chlamydia testing and infection among men who have sex with men--STD Surveillance Network, United States, 2010-2012. Clin Infect Dis 2014 Jun;58(11):1564-1570 [FREE Full text] [CrossRef] [Medline]
  25. Bernstein KT, Marcus JL, Nieri G, Philip SS, Klausner JD. Rectal gonorrhea and chlamydia reinfection is associated with increased risk of HIV seroconversion. J Acquir Immune Defic Syndr 2010 Apr 01;53(4):537-543. [CrossRef] [Medline]
  26. Mustanski B, Garofalo R, Monahan C, Gratzer B, Andrews R. Feasibility, acceptability, and preliminary efficacy of an online HIV prevention program for diverse young men who have sex with men: the keep it up! intervention. AIDS Behav 2013 Nov;17(9):2999-3012 [FREE Full text] [CrossRef] [Medline]
  27. Rosser BR, Wilkerson JM, Smolenski DJ, Oakes JM, Konstan J, Horvath KJ, et al. The future of Internet-based HIV prevention: a report on key findings from the Men's INTernet (MINTS-I, II) Sex Studies. AIDS Behav 2011 Apr;15 Suppl 1:S91-100 [FREE Full text] [CrossRef] [Medline]
  28. Margolis AD, Joseph H, Hirshfield S, Chiasson MA, Belcher L, Purcell DW. Anal intercourse without condoms among HIV-positive men who have sex with men recruited from a sexual networking web site, United States. Sex Transm Dis 2014 Dec;41(12):749-755. [CrossRef] [Medline]
  29. Christensen JL, Miller LC, Appleby PR, Corsbie-Massay C, Godoy CG, Marsella SC, et al. Reducing shame in a game that predicts HIV risk reduction for young adult MSM: a randomized trial delivered nationally over the Web. J Int AIDS Soc 2013 Nov 13;16(3 Suppl 2):18716 [FREE Full text] [Medline]
  30. Patel V, Lounsbury D, Berrios A, Riviera S, Vega C. Developing a peer-led social media based HIV testing intervention with YMSM of color - a feasibility study. 2014 Presented at: Youth + Tech + Health Live Conference; April 6-8, 2014; San Francisco, CA.
  31. Jones V, Whitlock M. What's going on online with hard to reach population? Social media use among black gay men. 2014 Presented at: Youth + Tech + Health Live Conference; April 6-8, 2014; San Francisco, CA.
  32. Hirshfield S, Chiasson MA, Joseph H, Scheinmann R, Johnson WD, Remien RH, et al. An online randomized controlled trial evaluating HIV prevention digital media interventions for men who have sex with men. PLoS One 2012;7(10):e46252 [FREE Full text] [CrossRef] [Medline]
  33. Lewis MA, Uhrig JD, Ayala G, Stryker J. Reaching men who have sex with men for HIV prevention messaging with new media: recommendations from an expert consultation. Ann Forum Collaborative HIV Res 2011 Oct 03;13(3):10-16 [FREE Full text]
  34. Cohen MS, Chen YQ, McCauley M, Gamble T, Hosseinipour MC, Kumarasamy N, HPTN 052 Study Team. Prevention of HIV-1 infection with early antiretroviral therapy. N Engl J Med 2011 Aug 11;365(6):493-505 [FREE Full text] [CrossRef] [Medline]
  35. Deterding S, Dixon D, Khaled R, Nacke L. From game design elements to gamefulness: defining “gamification”. 2011 Presented at: 15 International Academic MindTrek Conference: Envisioning Future Media Environments; Sept 28-30, 2011; Tampere, Finland. [CrossRef]
  36. Deci EL, Ryan RM. Facilitating optimal motivation and psychological well-being across life's domains. Can Psychol 2008;49(1):14-23. [CrossRef]
  37. Deterding S. Google Tech Talk. 2011 Jan 24. Meaningful play: getting gamification right   URL: [accessed 2017-05-14] [WebCite Cache]
  38. Schell J. The Art of Game Design: A Book of Lenses. Burlington, MA: Morgan Kaufmann; 2008.
  39. McGrath M, Anderson L, Beymer M, Cohen A, Natoli L, Javanbakht M, et al. Testing with benefits: results from a contingency management program to reduce recurrent STIs in high-risk MSM. 2015 Presented at: Annual Meeting of the American Public Health Association; Oct 31-Nov 4, 2015; Chicago, IL.
  40. Tan-Torres Edejer T, Baltussen R, Adam T, Hutubessy R, Acharya A, Evans DB, et al, editors. Making Choices in Health: WHO Guide to Cost-Effectiveness Analysis. Geneva, Switzerland: World Health Organization; 2003.
  41. Centers for Disease Control and Prevention. HIV Surveillance Report, 2015; vol. 27. Atlanta, GA: U.S. Department of Health and Human Services; 2016 Nov.   URL: [accessed 2017-04-10] [WebCite Cache]
  42. Ahmed R, Bates BR. Development of scales to assess patients' perception of physicians' cultural competence in health care interactions. J Transcult Nurs 2012 Jul;23(3):287-296. [CrossRef] [Medline]
  43. Krieger N, Smith K, Naishadham D, Hartman C, Barbeau EM. Experiences of discrimination: validity and reliability of a self-report measure for population health research on racism and health. Soc Sci Med 2005 Oct;61(7):1576-1596. [CrossRef] [Medline]
  44. Peek ME, Nunez-Smith M, Drum M, Lewis TT. Adapting the everyday discrimination scale to medical settings: reliability and validity testing in a sample of African American patients. Ethn Dis 2011;21(4):502-509 [FREE Full text] [Medline]
  45. Ulin P, Robinson E, Tolley E. Qualitative Methods in Public Health: A Field Guide for Applied Research. New York, NY: John Wiley & Sons; 2004.
  46. Miles M, Huberman A. Qualitative Data Analysis: An Expanded Sourcebook. 2nd edition. Thousand Oaks, CA: Sage Publications; 1994.
  47. Maxwell J. Qualitative Research Design: An Interactive Approach. 2nd edition. Thousand Oaks, CA: Sage Publications; 2005.
  48. Zocchetti C, Consonni D, Bertazzi PA. Relationship between prevalence rate ratios and odds ratios in cross-sectional studies. Int J Epidemiol 1997 Feb;26(1):220-223. [Medline]
  49. Greenland S. Interpretation and choice of effect measures in epidemiologic analyses. Am J Epidemiol 1987 May;125(5):761-768. [Medline]
  50. Barros AJD, Hirakata VN. Alternatives for logistic regression in cross-sectional studies: an empirical comparison of models that directly estimate the prevalence ratio. BMC Med Res Methodol 2003 Oct 20;3:21 [FREE Full text] [CrossRef] [Medline]
  51. Ryan G, Bernard H. Data management and analysis methods. In: Denzin N, Lincoln Y, editors. Handbook of Qualitative Research. 2nd edition. Thousand Oaks, CA: Sage Publications; 2000:769-802.
  52. Bryman A. Quantity and Quality in Social Science Research. London, UK: Unwin Hyman; 1998.
  53. Castel AD, Greenberg AE, Befus M, Willis S, Samala R, Rocha N, et al. Temporal association between expanded HIV testing and improvements in population-based HIV/AIDS clinical outcomes, District of Columbia. AIDS Care 2014;26(6):785-789. [CrossRef] [Medline]
  54. Weinhardt LS, Carey MP, Johnson BT, Bickham NL. Effects of HIV counseling and testing on sexual risk behavior: a meta-analytic review of published research, 1985-1997. Am J Public Health 1999 Sep;89(9):1397-1405. [Medline]
  55. Marks G, Crepaz N, Janssen RS. Estimating sexual transmission of HIV from persons aware and unaware that they are infected with the virus in the USA. AIDS 2006 Jun 26;20(10):1447-1450. [CrossRef] [Medline]

HIV: human immunodeficiency virus
MSM: men who have sex with men
SMS: short message service
STI: sexually transmitted infection
YMSM: young men who have sex with men

Edited by G Eysenbach; submitted 17.05.17; peer-reviewed by L Hightow-Weidman, D Vance; comments to author 31.05.17; revised version received 21.06.17; accepted 22.06.17; published 17.07.17


©Christopher M Mejia, Daniel Acland, Raluca Buzdugan, Reva Grimball, Lauren Natoli, Mark R McGrath, Jeffrey D Klausner, Sandra I McCoy. Originally published in JMIR Research Protocols (, 17.07.2017.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.