This paper is in the following e-collection/theme issue:

NIH mHealth - funded projects (29) HIV/AIDS/STI Prevention and Care (819) Mobile Health (mhealth) (2765) Knowledge Translation and Implementation Science (373) RCTs - Protocols/Proposals (eHealth) (508) Focus Groups and Qualitative Research for Human Factors Research (772)

Published on in Vol 11, No 4 (2022): April

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/37748, first published .
Evaluation of the Implementation and Effectiveness of a Mobile Health Intervention to Improve Outcomes for People With HIV in the Washington, DC Cohort: Study Protocol for a Cluster Randomized Controlled Trial

Evaluation of the Implementation and Effectiveness of a Mobile Health Intervention to Improve Outcomes for People With HIV in the Washington, DC Cohort: Study Protocol for a Cluster Randomized Controlled Trial

Evaluation of the Implementation and Effectiveness of a Mobile Health Intervention to Improve Outcomes for People With HIV in the Washington, DC Cohort: Study Protocol for a Cluster Randomized Controlled Trial

Authors of this article:

Jacqueline Hodges1 Author Orcid Image ;   Sylvia Caldwell1 Author Orcid Image ;   Wendy Cohn2 Author Orcid Image ;   Tabor Flickinger3 Author Orcid Image ;   Ava Lena Waldman1 Author Orcid Image ;   Rebecca Dillingham1 Author Orcid Image ;   Amanda Castel4 Author Orcid Image ;   Karen Ingersoll5 Author Orcid Image
Article Authors Cited by (2) Tweetations (7) Metrics

    Protocol

    1Division of Infectious Diseases and International Health, University of Virginia, Charlottesville, VA, United States

    2Public Health Sciences, University of Virginia, Charlottesville, VA, United States

    3Department of Medicine, University of Virginia, Charlottesville, VA, United States

    4Milken Institute School of Public Health, George Washington University, Washington, DC, United States

    5Department of Psychiatry and Neurobehavioral Sciences, University of Virginia, Charlottesville, VA, United States

    Corresponding Author:

    Jacqueline Hodges, MD, MPH

    Division of Infectious Diseases and International Health

    University of Virginia

    1215 Lee St

    Charlottesville, VA, 22903

    United States

    Phone: 1 4349240000

    Email: jchodges03@gmail.com


    Background: Gaps remain in achieving retention in care and durable HIV viral load suppression for people with HIV in Washington, DC (hereafter DC). Although people with HIV seeking care in DC have access to a range of supportive services, innovative strategies are needed to enhance patient engagement in this setting. Mobile health (mHealth) interventions have shown promise in reaching previously underengaged groups and improving HIV-related outcomes in various settings.

    Objective: This study will evaluate the implementation and effectiveness of a clinic-deployed, multifeature mHealth intervention called PositiveLinks (PL) among people with HIV enrolled in the DC Cohort, a longitudinal cohort of people with HIV receiving care in DC. A cluster randomized controlled trial will be conducted using a hybrid effectiveness-implementation design and will compare HIV-related outcomes between clinics randomized to PL versus usual care.

    Methods: The study aims are threefold: (1) We will perform a formative evaluation of PL in the context of DC Cohort clinics to test the feasibility, acceptability, and usability of PL and tailor the platform for use in this context. (2) We will conduct a cluster randomized controlled trial with 12 DC Cohort clinics randomized to PL or usual care (n=6 [50%] per arm) and measure the effectiveness of PL by the primary outcomes of patient visit constancy, retention in care, and HIV viral load suppression. We aim to enroll a total of 482 participants from DC Cohort clinic sites, specifically including people with HIV who show evidence of inconsistent retention in care or lack of viral suppression. (3) We will use the Consolidated Framework for Implementation Research (CFIR) and the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework to measure implementation success and identify site, patient, provider, and system factors associated with successful implementation. Evaluation activities will occur pre-, mid-, and postimplementation.

    Results: Formative data collection was completed between April 2021 and January 2022. Preliminary mHealth platform modifications have been performed, and the first round of user testing has been completed. A preimplementation evaluation was performed to identify relevant implementation outcomes and design a suite of instruments to guide data collection for evaluation of PL implementation throughout the trial period. Instruments include those already developed to support DC Cohort Study activities and PL implementation in other cohorts, which required modification for use in the study, as well as novel instruments designed to complete data collection, as guided by the CFIR and RE-AIM frameworks.

    Conclusions: Formative and preimplementation evaluations will be completed in spring 2022 when the trial is planned to launch. Specifically, comprehensive formative data analysis will be completed following data collection, coding, preliminary review, and synthesis. Corresponding platform modifications are ready for beta testing within the DC Cohort. Finalization of the platform for use in the trial will follow beta testing.

    Trial Registration: ClinicalTrials.gov NCT04998019; https://clinicaltrials.gov/ct2/show/NCT04998019

    International Registered Report Identifier (IRRID): PRR1-10.2196/37748

    JMIR Res Protoc 2022;11(4):e37748

    doi:10.2196/37748

    Keywords

    human immunodeficiency virus; HIV; mobile health; mHealth; implementation science; cluster randomized controlled trial; smartphone 


    Background

    Despite progress made toward addressing the HIV epidemic within the United States, concerted efforts are needed to address substantial gaps in the care continuum for people with HIV [1]. Retention in HIV care and achievement and maintenance of viral suppression are critical steps of the continuum [2]; however, currently less than half of all people with HIV in the United States are considered retained in care, and even fewer have achieved viral suppression [3,4]. Missed outpatient visits are an important early marker of failure to achieve suppression and are associated with increased mortality [5-7].

    National gaps persisting within the HIV care continuum are similar for people with HIV in Washington, DC (hereafter DC), which is a priority jurisdiction of the Ending the Epidemic Initiative [8]. Specific subpopulations of people with HIV have previously demonstrated higher discontinuity of care, including those who are male, Black, younger, uninsured, and with injection drug use as a risk factor for exposure to HIV [9,10]. People with HIV in DC face various barriers to retention in care, including limited transportation, lack of comprehensive medical case management and adherence-related services, absence of a medical home, and gaps in health literacy [11,12].

    The DC Cohort

    The DC Cohort is the largest citywide prospective cohort of people with HIV in the United States, with 11,700 participants having consented at 15 partnering clinics across DC [13]. The DC Cohort Study is a National Institutes of Health (NIH)-funded study conducted in partnership between the DC Center for AIDS Research (CFAR), DC Department of Health (DOH), and the NIH/National Institute of Allergy and Infectious Diseases (NIAID) as part of the DC Partnership for AIDS Progress [12]. Among the 15 sites, 14 (93%) agreed to participate in the study at the time of funding.

    DC Cohort clinics range in size, characteristics, patients, and populations served and include federal, academic, and community-based clinics, as well as pediatric and adult clinics. Through a network of research assistants (RAs) located at these partner clinics, approximately 450 people with HIV enroll in the DC Cohort annually. DC Cohort participants represent about 75% of people with HIV cared for at these clinics and are demographically similar to the broader HIV population in DC; in 1 interim assessment, 4258 (82%) of 5193 participants were black, 3531 (68%) male, and 1973 (38%) men who have sex with men (MSM) [14].

    Mobile Health and HIV

    Smartphone accessibility is high overall in the United States [15]. Mobile health (mHealth) interventions developed to encourage self-management and social support and enhance the mental health of people with HIV have been associated with improvement in antiretroviral therapy (ART) adherence and retention in care for at-risk groups [16-25]. A range of retention-in-care services are offered across DC Cohort sites [26,27]; however, no mHealth interventions have been systematically deployed or studied within these sites.

    PositiveLinks (PL) is a clinic-associated mHealth platform available to providers (in outpatient settings, including clinical and nonclinical) and patients through a smartphone app. Clinic providers also access a web portal to manage patient cohorts, an online implementation guide, and an online learning management system for training and certification. PL features were designed using psychological theories of behavior change (information-motivation-behavioral skills model [28] and social action theory [29]) and principles of motivational interviewing that encourage self-directed behavior change [30], and are informed by user-based design [31-38]. The PL platform delivers appointment reminders; daily queries (“check-ins”) of mood, stress, and medication adherence with self-monitoring feedback; display of recent cluster of differentiation (CD)4 and viral load lab results; access to PL support staff for assistance/troubleshooting; secure communication with providers and clinic staff using in-app messaging; tailored educational resources; and the ability to interact with other users on a secure, anonymous community message board (Figure 1). The PL platform has also been adapted and translated for use in Spanish-speaking populations [39].

    ‎
    Figure 1. Screenshots of the home page and several features of PL. Features shown include a discreet home screen (A) and a dashboard (B), daily queries for medication administration (C), and stress and mood (D), along with discreet push notifications. The “How am I?” screen (E) provides graphical feedback based on user answers to daily queries for self-monitoring. Additional features include a lab results page (F), an anonymous community message board (G), and secure in-app messaging between patients and providers/staff (H). Users can also upload documents that support Ryan White eligibility. PL: PositiveLinks.
    View this figure

    A prospective study of poorly retained people with HIV in Virginia found that PL usage increased retention in care and improved the cohort rate of viral suppression by over 30% at 12 months [40]. Additionally, PL was included as an evidence-based strategy to improve engagement in care for people with HIV in a 2020 update to guidelines by the HIV Medicine Association of the Infectious Diseases Society of America [41]. Although PL is a promising tool for people with HIV, efficacy has not been evaluated with a rigorous randomized trial or in urban populations. We describe a protocol for a cluster randomized controlled trial using a type II hybrid effectiveness-implementation design to test PL against usual care for people with HIV receiving outpatient HIV care in the DC Cohort.

    The aims of this study are threefold. We first aim to determine the feasibility, acceptability, and usability of PL within DC Cohort clinic sites in order to tailor the PL platform for use in this context. Our second aim is to determine the effectiveness of PL in relation to key clinical outcomes of viral suppression, visit constancy, and retention in care. Our third aim is to use validated implementation science frameworks to rigorously measure the success of implementation of the PL program within participating DC Cohort sites and identify site, patient, provider, and system-level factors critical for successful implementation.


    Outcome Measures

    Primary outcomes to be evaluated at 12 months following participation include viral suppression (HIV viral load <200 copies/mL), visit constancy (proportion of 4-month time intervals, with 1 visit with an HIV care provider completed in the 12-month period of study participation), and retention in care by the Health Resources and Services Administration (HRSA)-1 measure (keeping 2 HIV care appointments separated by 90 days within the 12-month time period of study participation) [42].

    The primary hypotheses to be tested are that compared to participants at DC Cohort sites randomized to usual care, participants at sites randomized to receive the intervention (tailored PL platform with associated program activities) will demonstrate, on average:

    • 15% better viral suppression at 12 months
    • 25% greater rate of visit constancy at 12 months
    • 25% greater rate of retention in care at 12 months

    The study will also test the impact of PL versus usual care on secondary outcomes related to patient psychosocial characteristics, including markers of mental health, stigma, social support, and drug use. Finally, relevant implementation-centered outcome measures identified during the preimplementation evaluation (described in further detail later) will be evaluated during the mid- and postimplementation phases of the study.

    Study Aim 1: Formative Evaluation

    To determine the feasibility, acceptability, and usability of PL within DC Cohort clinics, we performed a series of focus groups and in-depth interviews with stakeholders.

    DC Regional Planning Commission, DC Cohort Executive Committee, and DC Cohort Site Providers

    Focus groups were conducted with members of the DC Cohort Executive Committee (DC Cohort site principal investigators (PIs), NIH and DOH representatives; n=10) and the DC Regional Planning Commission on Health and HIV (COHAH; n=50). Both focus groups met online during COVID-19 surges in DC. Members were included in focus groups based on availability to participate; all members were invited to participate and provided informed consent. In-depth interviews were also conducted with 2 providers from 14 DC Cohort sites (eg, clinicians, nurses, case managers, social workers, and support staff; n=28).

    Provider focus groups were semistructured. An interview guide designed by the study team was used to collect perspectives on barriers to and facilitators of retention in care and viral suppression, input on app features based on experiences with the population of people with HIV they serve, and potential modifications felt to be most useful for enhancing retention in care. In-depth interviews were also semistructured and conducted using the same interview guides.

    People With HIV Receiving Care in the DC Cohort

    Focus groups were also conducted with a subset of people with HIV from 14 DC Cohort sites (5 focus groups, n=32 patients representing 8 sites). Eligible people with HIV were those who were aged 16 years or older, were receiving care at a DC Cohort clinical site, spoke English, could provide legal informed consent, and could participate virtually (due to the COVID-19 pandemic restrictions). Participants were identified, recruited, and consented from DC Cohort sites by site RAs. Participants were remunerated with a US $25 gift card. Individual think-aloud user testing has also been completed (approximately 14/482 [2.9%] expected to enroll). COVID-19 precautions were observed during all sessions. Participants were remunerated with US $50 gift cards and US $10 metro cards. Following the user testing phase, 1 DC Cohort site withdrew from the study and 2 DC Cohort clinics merged, leaving 12 active sites. The site that withdrew cited staffing issues during the COVID-19 era that would present a challenge to fully participate. After user testing, final beta testing to detect any bugs, glitches, or data loss issues will be conducted with an additional 14 participants who will use the DC Cohort PL platform with assistance from their clinic RA for 1 month. People with HIV who participate in the beta testing will be remunerated with a US $50 gift card and a US $10 metro card for each session.

    Focus Group Testing Among People With HIV in the DC Cohort

    Focus groups were conducted using semistructured interview guides designed to elicit patient knowledge and perspectives surrounding engagement in care and viral suppression, assess comfort with and use of technology and smartphone apps, and elicit perceptions about PL and its potential role in supporting engagement in care. Following demonstration of the PL platform features, interviewers elicited feedback on interest in the app and preferences for particular app features. Brief surveys were distributed at the end of the focus groups and included questions relating to self-reported adequacy of retention in care and care-seeking behaviors, unmet needs, comorbidities, perceptions of the patient-provider relationship, and levels of user experience/comfort with smartphones.

    Themes elicited from focus group and in-depth interviews were synthesized by the study team and presented to the DC Cohort Executive Committee, and consensus was reached on app modifications. Requested modifications were then provided to the PL development team.

    Think-Aloud User Testing Among People With HIV in the DC Cohort

    The modified app was iteratively tested with DC Cohort participants using the think-aloud protocol [43,44]. In total, 14 users provided input during 1-hour task-focused individual sessions and completed surveys at the end of the session, demonstrating how they navigate app features, while voicing their opinions about their experience of PL.

    Beta Testing Among People With HIV in the DC Cohort

    Investigators and developers discussed modifications to the app based on formative work following a preliminary review. The development team made modifications agreed on by the team. Beta testing of the near-finalized app will be conducted with 14 people with HIV, and the participating DC Cohort site RAs will be assigned to oversee patient enrollment, training on PL use, and ongoing app troubleshooting for the study. Participant interviews will be performed to solicit input after the first week of PL use in order to identify any issues with logins, navigation, functionality, or technical issues. Interviews will be repeated after the 1-month testing period concludes. A postparticipation survey will be performed to elicit feedback on the participants’ usage of the app over the 1-month period using the System Usability Scale [45]. Paradata metrics collected automatically by the app will be reviewed for the period as well. The research and PL development team will review all output from beta testing and make final app modifications.

    Formative Data Analysis

    All focus groups and interviews were audio-recorded and transcribed verbatim. Analysis of text files will be completed in spring 2022 using qualitative analysis software (Dedoose) with an a priori open coding process to identify themes and categories. Coding will be performed by at least 2 independent RAs to achieve consensus. Descriptive statistics will be used to analyze participant survey data for focus groups, in-depth interviews, and user and beta testing.

    Study Aim 2: Cluster Randomized Controlled Trial

    This study is a cluster randomized controlled trial whereby 12 clinics will be randomized to PL (n=6, 50%) or usual care (n=6, 50%); see Figure 2. Participants from clinics randomized to PL will receive access to the smartphone app following training on use provided by the site RA. Clinic providers (ie, clinicians, nonclinical care providers, support staff) will have access to the provider online learning management system for training on PL use, the PL provider online portal and smartphone app, and remote assistance provided by the PL program team. On-site administration will be supervised by site RAs. Patients at clinics randomized to PL will be able to use the app for at least 12 months up to the date of trial completion. Participants from clinics randomized to usual care will receive usual clinic retention and medication adherence support services for 12 months. Trial activities will complete in 2025.

    ‎
    Figure 2. Cluster randomized controlled trial recruitment and participant flow diagram. DC: Washington, DC; DOH: Department of Health; PL: PositiveLinks.
    View this figure
    Recruitment

    Participants in the cluster randomized controlled trial will be recruited from 12 clinics participating in the DC Cohort Study. Informed consent will be obtained for all DC Cohort participants under a protocol approved by the George Washington University Institutional Review Board (Protocol NCR202829; ClinicalTrials.gov NCT04998019). Inclusion criteria are people with HIV who (1) are enrolled in the DC Cohort study; (2) are aged 16 years or more; (3) if a minor, are in charge of their own HIV care (with waiver of parental consent); (4) speak and read English or Spanish at the fourth-grade level or above; (5) can provide informed consent; (6) plan to reside in the DC metro area for 12 months following enrollment; and (7) have at least 1 of the following putative indicators of poor retention (in order of priority): (1) detectable viral load, (2) not retained in care, (3) returning to care after a gap of ≥6 months, (4) no visit constancy in the 12 months prior to enrollment, (5) newly diagnosed or initiating HIV care, (6) recently transferred from a different HIV care site, or (7) evidence of simultaneous HIV care receipt at a DC Cohort site and a non-DC Cohort site based on DC DOH surveillance data. Exclusion criteria include people with HIV who are (1) aged below 16 years or if 16-17 years old have a parent in charge of HIV care and (2) unable to provide informed consent. To minimize cross-site contamination, people with HIV receiving care at 2 DC Cohort sites will be excluded. Eligible patients will be identified by the study team with monthly review of the DC Cohort database as well as input from site providers, patient navigators, and RAs. Patients who do not own a smartphone on enrollment will be provided a study smartphone to use for the duration of the study.

    Randomization

    In total, 6 clinics will be randomized to the intervention arm, and 6 clinics will be randomized to the usual care arm. Randomization will maximize the balance between arms in terms of clinics’ predominant patient population characteristics (adolescent vs adult), panel size (eg, the 2 largest clinics will be randomized, 1 to each arm), and availability of Ryan White Clinical services, with all other clinics randomized to each arm by the study statistician. Given the nature of the intervention, neither researchers nor participants will be blinded to the outcome of randomization of clinic sites.

    Sample Size Determination

    Sample size calculation and power analysis are based on data from the PL 12-month prospective outcome study [40] showing a 30% increase in viral suppression at 12 months and on DC Cohort data showing that 55% of participants achieved viral suppression within a 12-month period [27]. We would need a mean cluster size of 64 or 768 participants to detect a 15% difference. We would need to enroll 432 participants to detect a more conservative 17.5% increase in viral suppression at 12 months, for 80% power to detect a true difference between PL and usual care as 36 per condition, assuming an intraclass correlation coefficient of .02 and a coefficient of variation of cluster sizes of 0.5. To achieve a sample size with sufficient power, we aim to enroll a total of 482 participants.

    We estimate 60% of the people with HIV approached will be interested in the study based on the mean study consent rates for prior DC Cohort studies [46,47]. We plan to approach 945 people with HIV and enroll 482 (51%) participants overall (approximately n=40, on average, per cluster). Based on prior experience in recruiting people with HIV for studies in DC, recruiting 482 people with HIV over 20 months at the 12 clinics (at least 2 people per clinic per month) will be achievable, given the percentage of DC Cohort participants (7839/11,700, 67%) meeting the inclusion criteria. Based on previous experiences with PL and mHealth interventions [16,40], we anticipate approximately 15% participants lost to follow-up, resulting in 410 (85.1%) of 482 participants for analysis at the study endpoint.

    Trial Data Collection

    At study enrollment, baseline surveys will be completed by trial participants to evaluate specific sociodemographic measures relevant to retention in care [9,10], including age, sex, race, injection drug use, social determinants of health (eg, food insecurity, financial and housing instability), and changes in contact information in the past year. At study enrollment, 6 and 12 months, patients will be surveyed on risk factors for poor retention or lack of viral suppression, including medication nonadherence, self-efficacy, depression, and stressful life events [48] and psychosocial measures relevant to outcomes in PL studies, including experiences with stigma [49], social support, mental health, perceived stress, quality of life, self-efficacy related to substance use, and patient-provider communication. The site RA will administer baseline surveys using REDCap software during the enrollment visit, while 6- and 12-month surveys will be administered at a clinic visit or over the telephone. All data will be entered into REDCap.

    Efficacy Outcomes

    All DC Cohort sites have RAs designated to collect and export patient laboratory values (eg, CD4 count, HIV viral load), sociodemographics, comorbid diagnoses, and encounter data into a central database (DC Cohort Study Database). Data exported for our study will be restricted to a 45-day window surrounding each participant’s baseline, 6-month, and 12-month dates.

    Characterizing Usual Care in DC Cohort Clinics

    A standardized site assessment form was distributed as part of DC Cohort Study activities in 2016 to characterize the range of services comprising usual care delivered by DC Cohort clinic sites [27]. Services queried include clinic staffing (size, provider training, work experience), on-site clinical services, and activities to support patient ART adherence or linkage and retention in HIV care. Based on this assessment, the usual care condition across sites ranges from no ancillary support to comprehensive services (case management, adherence support, patient navigation, mental health, substance use, dental services, and food banks). This site assessment form will be redistributed electronically via REDCap to all sites (n=12, 100%) prior to initiation of the trial.

    Trial Statistical Analysis

    To compare the primary outcomes of viral suppression, visit constancy, and retention in care at 12 months between clusters, we will perform logistic regression using a mixed effects model (MEM), accounting for heterogeneity, including unequal cluster size between sites and correlations between participants from the same clinic site and between repeated measurements on the same participant group [50-52]. The comparisons between the conditions will be adjusted for both individual-level (gender, race, age, mode of HIV transmission) and cluster-level (differences in availability of specific adherence, retention and counseling services provided by sites) characteristics.

    The MEM will also compare each secondary outcome of interest (psychosocial characteristics at 6- and 12-month follow-up assessments) between clusters within the PL and usual care arms. Linear regression and logistic regression based on MEMs will be used to analyze continuous outcomes and binary outcomes, respectively, adjusted for the same individual- and cluster-level characteristics as with primary outcome analyses.

    Adverse Events

    Site RAs will routinely monitor and moderate activity on the PL platform by the respective site’s patient panel, including any inflammatory content or disclosure of identifying information posted on the community message board. Should such activity be identified, procedures to notify the site PIs will be instituted to determine whether additional actions are necessary to prevent further dissemination of inappropriate content.

    Ethics Approval

    Informed consent will be obtained for all trial providers and participants using an approved protocol, with ethical approval provided by the George Washington University Institutional Review Board (Protocol NCR202829; ClinicalTrials.gov NCT04998019).

    Study Aim 3: Implementation Evaluation

    Implementation of PL will be evaluated in parallel with efficacy of the intervention. Validated implementation science frameworks will be used pre-, mid- and post-PL implementation to determine factors that influence the relative success of the implementation strategy used to recruit, train, and retain patients and providers in the PL intervention. Specifically, we used the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework [53] to identify relevant implementation-centered outcomes during the preimplementation evaluation and will apply the framework mid- and postimplementation to measure and compare implementation success at each step of the implementation strategy executed across DC Cohort sites. The Consolidated Framework for Intervention Research (CFIR) [54] also guided preimplementation activities, as described later, and will be applied mid- and postimplementation toward identification of relevant barriers and facilitators of implementation success within DC Cohort sites at the patient, provider, clinic, and broader organizational levels.

    Preimplementation Evaluation

    During the preimplementation period, the DC Cohort Executive Committee, DC Cohort site leadership, and provider concerns related to the process of PL implementation were evaluated in conjunction with formative phase activities (focus groups, in-depth interviews). The output of both rounds of user testing conducted with people with HIV during the formative phase is currently being analyzed for any implementation-related concerns. During the preimplementation evaluation, we also used the RE-AIM framework to identify data points required to evaluate PL implementation based on predefined outcomes of interest across sites (Figure 3). A suite of instruments was then developed to support data collection to adequately capture all identified outcomes of interest and corresponding data points.

    ‎
    Figure 3. RE-AIM framework dimensions. Dimensions of interest for PL implementation in DC Cohort clinics are listed along with corresponding outcome data requiring collection, as well as the instruments that will be utilized. For each framework dimension, outcome measures evaluated using multiple different instruments are denoted with a corresponding symbol matched to the instrument used. DC: Washington, DC; HRSA: Health Resources and Services Administration; ORIC: Organizational Readiness for Implementing Change; PL: PositiveLinks; RE-AIM: Reach Effectiveness Adoption Implementation Maintenance.
    View this figure
    Mid- and Postimplementation Evaluations

    During the mid- and postimplementation evaluations, we will conduct in-depth interviews with a subset of site providers and RAs (n=24). Postimplementation, an additional focus group will be repeated with the DC Cohort Executive Committee. We will also conduct 4 focus groups postimplementation with a subset of PL trial participants (32/482 [6.6%] expected to enroll). People with HIV will be sampled to ensure demographic diversity and include a range of users (eg, frequent vs infrequent users) who will be remunerated with a US $25 gift card and a US $10 metro card. For each interview/focus group conducted mid- and postimplementation, a semistructured interview guide will be used, developed based on our prior application of the CFIR to evaluate PL implementation [55]. Postimplementation interview guides will be updated in an iterative fashion based on analysis of midimplementation feedback.

    Implementation Evaluation Data Collection

    A suite of data collection instruments relevant to our implementation evaluation and planned for deployment to stakeholders mid- and postimplementation were developed, as described during the preimplementation evaluation (Table 1). Informed consent to participate in the implementation evaluation phase of the study will be obtained during the trial consent process for patients. Providers will be offered open enrollment in PL across sites randomized to the intervention, and informed consent will be obtained to participate in implementation evaluation activities.

    Table 1. Data collection instruments identified during the preimplementation phase to support collection of relevant data points for the implementation evaluation phase of cluster randomized controlled trial.
    InstrumentDescription of instrumentFrequency and timing of data collection and export
    DCa Cohort Study DatabasePatient laboratory values, encounter data, and sociodemographic data are routinely collected for all people with HIV in the DC Cohort.Data are collected continuously by site RAsb and uploaded/exported to the database monthly for all people with HIV in the DC Cohort, including trial participants.
    Patient consent logsPatient responses are tracked to site RAs, who consent and enroll participants in DC Cohort studies. These logs will be modified for use in the trial to track patient enrollment and completion of various steps of PLc program implementation.Log responses are uploaded and updated within the DC Cohort Study Database by RAs monthly.
    DC Cohort site assessment formsSite assessment forms query various site-level characteristics and will be modified to include items related to PL implementation (eg, site-level use of telemedicine, other mHealthd tools). Forms will be deployed electronically and completed by site PIse at the start of the study period.Interim distribution of the site assessment forms to site leadership (to be completed with site staff/provider assistance) will be performed throughout the study period at a yearly interval.
    Provider baseline and follow-up surveysSurveys will assess provider characteristics of interest, including specialty, time employed at the site, specific training, and baseline technology use. Follow-up survey items include perspectives on provider roles within the program, program adaptations, and individual usage of PL.Baseline surveys will be distributed to site providers at the start of the study period and then every 6 months to providers newly employed during the study period and consenting to participation. Follow-up surveys will be redistributed to providers at 6-month intervals throughout the study period.
    PL posttraining feedback surveyProviders completing the training step of PL implementation will be tracked by completion of a posttraining survey. Feedback elicited will include perceptions of the online learning management system (eg, modules). The survey was modified to include items from the ORICf measure.An electronic feedback survey immediately follows completion of learning modules via the online learning management system. Survey responses will be exported for mid- and postimplementation evaluations.
    PL paradataPlatform paradata metrics include user logins, screens viewed, features used, and screen time. In-app content includes patient responses to daily queries, messages posted on the community message board, and secure messages exchanged between patients and clinic providers.Paradata metrics are collected and stored in the platform automatically and continuously. In-app content will be exported for analysis for the postimplementation evaluation.

    aDC: Washington, DC.

    bRA: research assistant.

    cPL: PositiveLinks.

    dmHealth: mobile health.

    ePI: principal investigator.

    fORIC: Organizational Readiness for Implementing Change.

    Existing data instruments were identified, including those developed to support the broader DC Cohort Study (DC Cohort Study Database, patient consent logs, site assessment forms), as well as those developed and used for evaluations across different clinic sites implementing PL in various contexts (PL posttraining feedback survey, PL paradata). Modifications to these instruments were planned to further support collection of specific data points required to capture all RE-AIM outcome measures that were elucidated during the preimplementation phase. For example, the PL posttraining feedback survey follows providers’ completion of learning modules included in the online learning management system used to train them on how to use PL. For providers in this trial, we plan to modify the survey to include validated items from the Organizational Readiness for Implementing Change (ORIC) measure [56], which assesses providers’ perceptions of the collective psychological readiness of their DC Cohort sites to implement organizational changes necessary to incorporate the PL program within their clinic’s activities (a characteristic that can be examined at the level of an individual provider and in aggregate at the site level, which is important for the “adoption” dimension of RE-AIM).

    Novel instruments were also designed to complete necessary data collection for corresponding RE-AIM dimensions, including both the provider baseline and provider follow-up surveys. Provider follow-up surveys, for example, probe for shifting roles and adaptations related to program implementation made by providers, different ways in which providers engage with available PL features, and provider perspectives on materials and processes supporting program implementation within the DC Cohort sites (all relevant to the “implementation” dimension of RE-AIM).

    Analysis of Implementation Outcomes

    All interviews and focus groups conducted for implementation evaluations will be audiotaped, transcribed, and analyzed using Dedoose, as described for formative data analysis. We will use a deductive approach built around CFIR constructs for analysis of stakeholder interviews, with specific constructs selected from our prior evaluation [55] as the a priori categories to assign codes. Two or more investigators will independently code each interview/focus group. Additional codes will be iteratively added in an inductive fashion following coder consensus. We will analyze PL paradata metrics using Google Analytics to characterize user activity for multiple features over the study period (eg, community message board, daily check-ins, provider messaging), as well as establish any associations between frequency/dose of user activity and differences in primary outcomes.

    Descriptive statistics will be used to analyze the proportions of patients and providers completing implementation steps (eg, reach, adoption dimensions of RE-AIM). Pearson correlation and binary logistic regression will be used to for exploratory analysis of associations between patient-specific covariates (eg, demographics) and outcome measures for each dimension (eg, reach, including PL usage by patients). We will also examine associations between site-specific covariates (reach measures attained for their patients, adoption measures achieved for their providers, site-level measures, for example, organizational readiness), and differences in patients’ primary outcomes (viral suppression, retention in care) observed on average for sites randomized to PL. Provider survey responses will be analyzed using descriptive statistics for Likert responses and with qualitative analysis, as described before for open-ended responses.


    Formative Phase

    Interviews and focus groups for the formative phase were completed between April and December 2021. Qualitative analysis of interview and focus group transcripts is ongoing as of March 2022. Preliminary app modification requests have been finalized following consensus reached between study team members and the DC Cohort Executive Committee, and the first round of user testing of the modified app with people with HIV has been completed. Recruitment for beta testing with people with HIV was initiated in February 2022.

    Cluster Randomized Controlled Trial

    Randomization will occur in spring 2022. Patient recruitment for the cluster randomized controlled trial is planned to start in spring 2022.

    Implementation Evaluation

    Following completion of the preimplementation evaluation, a series of instruments were designed using the RE-AIM framework to support planned data collection for relevant implementation outcomes during the mid- and postimplementation evaluations. The preimplementation evaluation process, including the design of instruments guided by several relevant implementation science frameworks, and logistical planning surrounding the method and timing of instrument distribution, data export/access, and analysis in conjunction with trial activities, will be described in further detail in a separate publication.

    We have previously applied the CFIR toward examination of PL implementation in another cohort seeking care at a Ryan White Clinic in Virginia, using a rigorous process of in-depth interviews with participating stakeholders [55]. CFIR domains (with corresponding constructs) that emerged from our prior evaluation as most relevant to PL implementation included Inner Setting (Compatibility, Access to Knowledge and Information), Outer Setting (Patient Needs and Resources, External Policy and Incentives), Characteristics of Individuals (Knowledge and Beliefs), Innovation Characteristics (Adaptability, Complexity), and Implementation Process (Planning, Engagement of Key Stakeholders). Provider surveys (provider baseline and follow-up surveys) were designed to incorporate items that assess these constructs.


    Principal Results

    This is a novel, large-scale cluster randomized controlled trial with a hybrid efficacy-implementation design examining the impact of the PL mHealth intervention on HIV-related patient outcomes with a direct comparison arm of usual care. Prior work to evaluate the clinical effectiveness of PL has been limited to single-arm prospective cohort studies within a nonurban population in Central Virginia. This project will use a randomized design to test the effectiveness of PL against usual care in a diverse urban cohort of people with HIV not achieving durable viral suppression or retention in care. We hypothesize that compared to usual care, clusters participating in the PL intervention will experience improved rates in viral suppression, visit constancy, and retention in care at 12 months.

    Comparison With Prior Work

    This study will significantly extend the evidence base for this intervention beyond more rural samples by testing its efficacy in a vulnerable urban sample using a robust study design. Further, evaluation of PL implementation to date has been limited to a small subset of clinics within a nonurban context. This project builds on prior preliminary work using implementation science frameworks to identify best practices for implementing PL in a range of different urban HIV care settings and corresponding determinants of implementation success and will inform future disseminations of PL and other mHealth tools at scale in order to improve the lives and health of people with HIV.

    Limitations

    Limitations exist in this study design. Although formative work for this distinct patient population to date does not suggest significant changes to the platform will be required based on preliminary review, beta testing is underway and further modifications may be suggested by users. The study research and development teams will prioritize the most feasible changes prior to trial initiation. DC Cohort sites are heterogenous in the services provided to support patient adherence and retention and the patient populations they primarily serve, presenting a potential challenge in assessment of the impact of PL when implemented in conjunction with variable services across clinics. The usual care condition will require assessment at baseline and periodically throughout the trial, and site-level characteristics must be adjusted for during statistical analyses.

    Conclusion

    Output from the formative phase is currently being analyzed, and corresponding preliminary modifications to the platform are being tested by people with HIV within the DC Cohort. Modifications will be finalized by the app development team following beta testing. Trial activities are expected to begin in spring 2022.

    Acknowledgments

    Drs Castel and Ingersoll are multiple principal investigators (PIs) on this project which is funded by the National Institute of Mental Health (R01MH122375). The DC Cohort is funded by the National Institute of Allergy and Infectious Diseases (NIAID; UM1AI069503 and 1R24AI152598-01). Dr Hodges receives additional support from the National Institutes of Health (NIH; T32AI007046).

    Authors' Contributions

    The cosenior authors are KI and AC. KI, AC, RD, and WC are responsible for study conceptualization and planning; AC and KI, funding acquisition; KI, AC, RD, and ALW, project administration; JH, WC, SC, TF, KI, and AC, investigation; and JH, SC, KI, and AC, manuscript preparation. All authors have reviewed and approved the final manuscript.

    Data included in this manuscript were collected by the DC Cohort Study Group with investigators and research staff located at the Children’s National Hospital Adolescent (Lawrence D’Angelo) and Pediatric (Natella Rakhmanina) clinics, the Interim Senior Deputy Director of the DC Department of Health (DOH) HIV/AIDS, Hepatitis, STD and TB Administration (HAHSTA; Clover Barnes), the Family and Medical Counseling Service (Michael Serlin); Georgetown University (Princy Kumar), the George Washington University Biostatistics Center (Marinella Temprosa, Vinay Bhandaru, Tsedenia Bezabeh, Nisha Grover, Lisa Mele, Susan Reamer, Alla Sapozhnikova, Greg Strylewicz, and Jiayang Xiao), the George Washington University Department of Epidemiology (Shannon Barth, Morgan Byrne, Amanda Castel, Alan Greenberg, Shannon Hammerlund, Paige Kulie, Anne Monroe, James Peterson, and Bianca Stewart) and the Department of Biostatistics and Bioinformatics (Yan Ma), the George Washington University Medical Faculty Associates (Hana Akselrod), the Howard University Adult Infectious Disease Clinic (Jhansi L. Gajjala) and Pediatric Clinic (Sohail Rana), Kaiser Permanente Mid-Atlantic States (Michael Horberg), the La Clinica Del Pueblo (Ricardo Fernandez), MetroHealth (Duane Taylor), the Washington Health Institute (Jose Bordon), Unity Health Care (Gebeyehu Teferi), the Veterans Affairs Medical Center (Debra Benator), the Washington Hospital Center (Maria Elena Ruiz), and the Whitman-Walker Institute (Stephen Abbott).

    Conflicts of Interest

    RD, SC, ALW and KI have active consulting agreements with Warm Health Technology, Inc.

    ‎
    Multimedia Appendix 1

    CONSORT-EHEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 1305 KB
    1. Fauci AS, Redfield RR, Sigounas G, Weahkee MD, Giroir BP. Ending the HIV epidemic: a plan for the United States. JAMA 2019 Mar 05;321(9):844-845. [CrossRef] [Medline]
    2. Colasanti J, Kelly J, Pennisi E, Hu Y, Root C, Hughes D, et al. Continuous retention and viral suppression provide further insights into the HIV care continuum compared to the cross-sectional HIV care cascade. Clin Infect Dis 2016 Mar 01;62(5):648-654 [FREE Full text] [CrossRef] [Medline]
    3. Division of HIV/AIDS Prevention, National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention. Understanding the HIV Care Continuum. Centers for Disease Control and Prevention. 2019 Jul.   URL: https://www.cdc.gov/hiv/pdf/library/factsheets/cdc-hiv-care-continuum.pdf [accessed 2022-02-01]
    4. Division of HIV Prevention, National Center for HIV, Viral Hepatitis, STD, and TB Prevention. Status of HIV in the U.S. 2021. Centers for Disease Control and Prevention.   URL: https://www.cdc.gov/hiv/policies/strategic-priorities/mobilizing/status-of-hiv.html [accessed 2022-02-01]
    5. Simoni JM, Frick PA, Pantalone DW, Turner BJ. Antiretroviral adherence interventions: a review of current literature and ongoing studies. Top HIV Med 2003;11(6):185-198 [FREE Full text] [Medline]
    6. Giordano TP, Gifford AL, White AC, Suarez-Almazor ME, Rabeneck L, Hartman C, et al. Retention in care: a challenge to survival with HIV infection. Clin Infect Dis 2007 Jun 01;44(11):1493-1499. [CrossRef] [Medline]
    7. Yehia BR, French B, Fleishman JA, Metlay JP, Berry SA, Korthuis PT, Research Network HIV. Retention in care is more strongly associated with viral suppression in HIV-infected patients with lower versus higher CD4 counts. J Acquir Immune Defic Syndr 2014 Mar 01;65(3):333-339 [FREE Full text] [CrossRef] [Medline]
    8. Office of Infectious Disease and HIV/AIDS Policy. Ending the HIV Epidemic: About Ending the HIV Epidemic in the U.S.: Priority Jurisdictions: Phase I. HIV.gov. 2020.   URL: https://www.hiv.gov/federal-response/ending-the-hiv-epidemic/jurisdictions [accessed 2022-02-01]
    9. Colasanti J, Stahl N, Farber EW, Del Rio C, Armstrong WS. An exploratory study to assess individual and structural level barriers associated with poor retention and re-engagement in care among persons living with. J Acquir Immune Defic Syndr 2017 Feb 01;74 Suppl 2:S113-S120 [FREE Full text] [CrossRef] [Medline]
    10. Rebeiro PF, Abraham AG, Horberg MA, Althoff KN, Yehia BR, Buchacz K, et al. Sex, race, and HIV risk disparities in discontinuity of HIV care after antiretroviral therapy initiation in the United States and Canada. AIDS Patient Care STDS 2017 Mar;31(3):129-144 [FREE Full text] [CrossRef] [Medline]
    11. Castel AD, Kalmin MM, Hart RLD, Young HA, Hays H, Benator D, et al. Disparities in achieving and sustaining viral suppression among a large cohort of HIV-infected persons in care - Washington, DC. AIDS Care 2016 Nov;28(11):1355-1364. [CrossRef] [Medline]
    12. Greenberg AE, Hays H, Castel AD, Subramanian T, Happ LP, Jaurretche M, DC Cohort Executive Committee. Development of a large urban longitudinal HIV clinical cohort using a web-based platform to merge electronically and manually abstracted data from disparate medical record systems: technical challenges and innovative solutions. J Am Med Inform Assoc 2016 May;23(3):635-643 [FREE Full text] [CrossRef] [Medline]
    13. Castel AD, Terzian A, Opoku J, Happ LP, Younes N, Kharfen M, et al. Defining care patterns and outcomes among persons living with HIV in Washington, DC: linkage of clinical cohort and surveillance data. JMIR Public Health Surveill 2018 Mar 16;4(1):e23. [CrossRef]
    14. Opoku J, Doshi RK, Castel AD, Sorensen I, Horberg M, Allston A, et al. Comparison of clinical outcomes of persons living with HIV by enrollment status in Washington, DC: evaluation of a large longitudinal HIV cohort study. JMIR Public Health Surveill 2020 Apr 15;6(2):e16061. [CrossRef]
    15. Pew Research Center. Mobile Fact Sheet. 2021.   URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2022-02-01]
    16. Ingersoll KS, Dillingham RA, Hettema JE, Conaway M, Freeman J, Reynolds G, et al. Pilot RCT of bidirectional text messaging for ART adherence among nonurban substance users with HIV. Health Psychol 2015 Dec;34 Suppl:1305-1315 [FREE Full text] [CrossRef] [Medline]
    17. Amico KR. Evidence for technology interventions to promote ART adherence in adult populations: a review of the literature 2012-2015. Curr HIV/AIDS Rep 2015 Dec;12(4):441-450 [FREE Full text] [CrossRef] [Medline]
    18. Finitsis DJ, Pellowski JA, Johnson BT. Text message intervention designs to promote adherence to antiretroviral therapy (ART): a meta-analysis of randomized controlled trials. PLoS One 2014;9(2):e88166 [FREE Full text] [CrossRef] [Medline]
    19. Simoni JM, Kutner BA, Horvath KJ. Opportunities and challenges of digital technology for HIV treatment and prevention. Curr HIV/AIDS Rep 2015 Sep 28;12(4):437-440. [CrossRef] [Medline]
    20. da Costa TM, Barbosa BJ, Gomes e Costa DA, Sigulem D, de Fatima Marin H, Filho AC, et al. Results of a randomized controlled trial to assess the effects of a mobile SMS-based intervention on treatment adherence in HIV/AIDS-infected Brazilian women and impressions and satisfaction with respect to incoming messages. Int J Med Inform 2012 Apr;81(4):257-269 [FREE Full text] [CrossRef] [Medline]
    21. Kiwanuka N, Mpendo J, Asiimwe S, Ssempiira J, Nalutaaya A, Nambuusi B, et al. A randomized trial to assess retention rates using mobile phone reminders versus physical contact tracing in a potential HIV vaccine efficacy population of fishing communities around Lake Victoria, Uganda. BMC Infect Dis 2018 Nov 21;18(1):591. [CrossRef]
    22. Venter W, Coleman J, Chan VL, Shubber Z, Phatsoane M, Gorgens M, et al. Improving linkage to HIV care through mobile phone apps: randomized controlled trial. JMIR Mhealth Uhealth 2018 Jul 17;6(7):e155 [FREE Full text] [CrossRef] [Medline]
    23. Cooper V, Clatworthy J, Whetham J, Consortium E. mHealth interventions to support self-management in HIV: a systematic review. Open AIDS J 2017;11:119-132 [FREE Full text] [CrossRef] [Medline]
    24. Lester RT, Mills EJ, Kariri A, Ritvo P, Chung M, Jack W, et al. The HAART cell phone adherence trial (WelTel Kenya1): a randomized controlled trial protocol. Trials 2009 Sep 22;10(1):87. [CrossRef]
    25. Govindasamy D, Meghij J, Kebede Negussi E, Clare Baggaley R, Ford N, Kranzer K. Interventions to improve or facilitate linkage to or retention in pre-ART (HIV) care and initiation of ART in low- and middle-income settings: a systematic review. J Int AIDS Soc 2014;17:19032 [FREE Full text] [Medline]
    26. Happ LP, Monroe AK, Young HA, Ma Y, Greenberg AE, Horberg MA, DC Cohort Executive Committee. Individual-level and clinic-level factors associated with achieving glycemic control in a large cohort of people with HIV in care-Washington, DC. J Acquir Immune Defic Syndr 2020 Sep 01;85(1):113-122 [FREE Full text] [CrossRef] [Medline]
    27. Monroe AK, Happ LP, Rayeed N, Ma Y, Jaurretche MJ, Terzian AS, et al. Clinic-level factors associated with time to antiretroviral initiation and viral suppression in a large, urban cohort. Clin Infect Dis 2020 Oct 23;71(7):e151-e158 [FREE Full text] [CrossRef] [Medline]
    28. Amico KR, Barta W, Konkle-Parker DJ, Fisher JD, Cornman DH, Shuper PA, et al. The information-motivation-behavioral skills model of ART adherence in a Deep South HIV+ clinic sample. AIDS Behav 2009 Feb;13(1):66-75 [FREE Full text] [CrossRef] [Medline]
    29. Ewart CK. Social action theory for a public health psychology. Am Psychol 1991 Sep;46(9):931-946. [CrossRef] [Medline]
    30. William RM, Stephen R. Motivational Interviewing: Helping People Change, Third Edition. New York: Guilford Press; 2012.
    31. Laurence C, Wispelwey E, Flickinger TE, Grabowski M, Waldman AL, Plews-Ogan E, et al. Development of PositiveLinks: a mobile phone app to promote linkage and retention in care for people with HIV. JMIR Form Res 2019 Mar 20;3(1):e11578 [FREE Full text] [CrossRef] [Medline]
    32. Draper SW, Norman DA. User Centered System Design: New Perspectives on Human-Computer Interaction. Hillsdale, NJ: Lawrence Erlbaum Associates; 1986.
    33. Nielsen J. Designing Web Usability. Indianapolis, IN: New Riders; 2000.
    34. Nielsen J, Loranger H. Prioritizing Web Usability. Berkeley, CA: New Riders; 2006.
    35. Nielsen J. Usability Engineering. Cambridge, MA: AP Professional; 1993.
    36. Kinzie MB, Cohn WF, Julian MF, Knaus WA. A user-centered model for web site design: needs assessment, user interface design, and rapid prototyping. J Am Med Inform Assoc 2002;9(4):320-330 [FREE Full text] [Medline]
    37. Hilgart MM, Ritterband LM, Thorndike FP, Kinzie MB. Using instructional design process to improve design and development of Internet interventions. J Med Internet Res 2012 Jun;14(3):e89 [FREE Full text] [CrossRef] [Medline]
    38. Harte R, Glynn L, Rodríguez-Molinero A, Baker PM, Scharf T, Quinlan LR, et al. A human-centered design methodology to enhance the usability, human factors, and user experience of connected health systems: a three-phase methodology. JMIR Hum Factors 2017 Mar 16;4(1):e8 [FREE Full text] [CrossRef] [Medline]
    39. Flickinger TE, Sherbuk JE, Petros de Guex K, Añazco Villarreal D, Hilgart M, McManus KA, et al. Adapting an m-Health intervention for Spanish-speaking Latinx people living with HIV in the nonurban Southern United States. Telemed Rep 2021 Feb;2(1):46-55 [FREE Full text] [CrossRef] [Medline]
    40. Dillingham R, Ingersoll K, Flickinger TE, Waldman AL, Grabowski M, Laurence C, et al. PositiveLinks: a mobile health intervention for retention in HIV care and clinical outcomes with 12-month follow-up. AIDS Patient Care STDS 2018 Jun;32(6):241-250 [FREE Full text] [CrossRef] [Medline]
    41. Thompson M, Horberg M, Agwu A, Colasanti J, Jain M, Short W. Primary care guidance for persons with human immunodeficiency virus: 2020 update by the HIV Medicine Association of the Infectious Diseases Society of America. Clin Infect Dis 2021;73(11):e3572-e3605. [CrossRef]
    42. Mugavero MJ, Davila JA, Nevin CR, Giordano TP. From access to engagement: measuring retention in outpatient HIV clinical care. AIDS Patient Care STDS 2010 Oct;24(10):607-613 [FREE Full text] [CrossRef] [Medline]
    43. Lewis C, Rieman J. Task-Centered User Interface Design: A Practical Introduction. Boulder, CO: University of Colorado, Department of Computer Science; 1993.
    44. Nielsen J. Thinking Aloud: The #1 Usability Tool. 2012.   URL: https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/ [accessed 2022-01-01]
    45. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum-Comput Interact 2008 Jul 30;24(6):574-594. [CrossRef]
    46. Castel AD, Tang W, Peterson J, Mikre M, Parenti D, Elion R, et al. Sorting through the lost and found: are patient perceptions of engagement in care consistent with standard continuum of care measures? J Acquir Immune Defic Syndr 2015 May 01;69 Suppl 1:S44-S55 [FREE Full text] [CrossRef] [Medline]
    47. Castel AD, Qasmieh S, Greenberg D, Ellenberger N, Howell TH, Griffith C, et al. Digital gaming to improve adherence among adolescents and young adults living with HIV: mixed-methods study to test feasibility and acceptability. JMIR Serious Games 2018 Oct 15;6(4):e10213 [FREE Full text] [CrossRef] [Medline]
    48. Corless IB, Hoyt AJ, Tyer-Viola L, Sefcik E, Kemppainen J, Holzemer WL, et al. 90-90-90-Plus: maintaining adherence to antiretroviral therapies. AIDS Patient Care STDS 2017 May;31(5):227-236 [FREE Full text] [CrossRef] [Medline]
    49. Berger BE, Ferrans CE, Lashley FR. Measuring stigma in people with HIV: psychometric assessment of the HIV stigma scale. Res Nurs Health 2001 Dec;24(6):518-529. [Medline]
    50. Demidenko E. Mixed Models: Theory and Applications with R, Second Edition. Hoboken, NJ: Wiley; 2013:433-486.
    51. Detry MA, Ma Y. Analyzing repeated measurements using mixed models. JAMA 2016 Jan 26;315(4):407-408. [CrossRef]
    52. Heo M, Leon AC. Performance of a mixed effects logistic regression model for binary outcomes with unequal cluster size. J Biopharm Stat 2005;15(3):513-526. [CrossRef] [Medline]
    53. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019;7:64 [FREE Full text] [CrossRef] [Medline]
    54. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50 [FREE Full text] [CrossRef] [Medline]
    55. Cohn WF, Canan CE, Knight S, Waldman AL, Dillingham R, Ingersoll K, et al. An implementation strategy to expand mobile health use in HIV care settings: rapid evaluation study using the Consolidated Framework for Implementation Research. JMIR Mhealth Uhealth 2021 Apr 28;9(4):e19163 [FREE Full text] [CrossRef] [Medline]
    56. Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci 2014;9:7 [FREE Full text] [CrossRef] [Medline]

    ‎
    ART: antiretroviral therapy
    CD: cluster of differentiation
    CFAR: Center for AIDS Research
    CFIR: Consolidated Framework for Intervention Research
    DC: Washington, DC
    DOH: Department of Health
    HRSA: Health Resources and Services Administration
    MEM: mixed effects model
    mHealth: mobile health
    NIH: National Institutes of Health
    ORIC: Organizational Readiness for Implementing Change
    PI: principal investigator
    PL: PositiveLinks
    RA: research assistant
    RE-AIM: Reach Effectiveness Adoption Implementation Maintenance

    Edited by T Leung; This paper was externally peer reviewed by the Population and Public Health Approaches to HIV/AIDS Study Section (PPAH) at the National Institute of Mental Health (National Institutes of Health, USA).submitted 09.03.22; accepted 29.03.22; published 22.04.22

    Copyright

    ©Jacqueline Hodges, Sylvia Caldwell, Wendy Cohn, Tabor Flickinger, Ava Lena Waldman, Rebecca Dillingham, Amanda Castel, Karen Ingersoll. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 22.04.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.