Background: Psychotic disorders are among the most disabling of all mental disorders. The first-episode psychosis (FEP) often occurs during adolescence or young adulthood. Young people experiencing FEP often face multiple barriers in accessing a comprehensive range of psychosocial services, which have predominantly been delivered in person. New models of service delivery that are accessible, sustainable, and engaging are needed to support recovery in youth diagnosed with FEP.
Objective: In this paper, we describe a protocol to implement and evaluate the acceptability, safety, and potential efficacy of an online psychosocial therapeutic intervention designed to sustain recovery and prevent relapses in young adults diagnosed with FEP. This intervention was originally developed and tested in Australia and has been adapted for implementation and evaluation in Canada and is called Horyzons-Canada (HoryzonsCa).
Methods: This cohort study is implemented in a single-center and applies a pre-post mixed methods (qualitative-quantitative convergent) design. The study involves recruiting 20 participants from a specialized early intervention program for psychosis located in Montreal, Canada and providing them with access to the HoryzonsCa intervention for 8 weeks. Data collection includes interview-based psychometric measures, self-reports, focus groups, and interviews.
Results: This study received funding from the Brain and Behavior Research Foundation (United States), the Quebec Health Research Funding Agency (Canada), and the Canada Research Chairs Program. The study was approved by the Research Ethics Board of the Centre intégré universitaire de santé et de services sociaux de l'Ouest-de-l'Île-de-Montréal on April 11, 2018 (#IUSMD 17-54). Data were collected from August 16, 2018, to April 29, 2019, and a final sample of 20 individuals participated in the baseline and follow-up interviews, among which 9 participated in the focus groups. Data analysis and reporting are in process. The results of the study will be submitted for publication in 2021.
Conclusions: This study will provide preliminary evidence on the acceptability, safety, and potential efficacy of using a digital health innovation adapted for the Canadian context to deliver specialized mental health services to youth diagnosed with FEP.
Trial Registration: ISRCTN Registry ISRCTN43182105; https://www.isrctn.com/ISRCTN43182105
International Registered Report Identifier (IRRID): RR1-10.2196/28141
Psychotic disorders have a lifetime prevalence of 3%  and include symptoms such as delusions, hallucinations, disorganized thoughts and behaviors, poverty of thought and affect, apathy, and deficits in verbal memory and executive functioning [ , ]. The onset of psychosis typically occurs during adolescence or young adulthood (ie, between the ages of 15-25 years), often leading to substantial impairments in social and community functioning, and ultimately derailing transitions toward life goals. Psychosis is associated with three of the top five leading causes of disability in the world (ie, major depression, bipolar disorder, and schizophrenia) for adolescents and young adults [ ], incurring substantial costs for the health care system and society in terms of loss of productivity. As such, psychotic disorders have been described in the literature as among the most scientifically challenging and disabling mental disorders [ , ].
Over the past two decades, a specialized early intervention approach has been developed for youth diagnosed with first-episode psychosis (FEP) with the ultimate goal of achieving symptom remission, relapse prevention, and social recovery [, ]. Many of these programs accept patients with schizophrenia spectrum psychoses and affective psychoses. Specialized early intervention services typically involve medication, psychosocial services (eg, illness education, lifestyle management, employment and education support, or family education and support), and case management [ , ]. The short-term (ie, 1-2 years) benefits of this specialized approach compared to routine care have been reported in several randomized controlled trials, quasi-experimental studies, and reviews of the literature [ - ]. Randomized controlled trials have confirmed that young people treated in a specialized early intervention service have higher rates of adherence; lower relapse and hospitalization rates; better quality of life; and improved outcomes at 12, 18, and 24 months compared to patients in routine care [ - ].
Although the specialized early intervention approach has shown promise in improving outcomes in youth with FEP, the need to develop better psychological, social, and vocational interventions for this population continues to be a priority. This is because there are several clinical challenges associated with treating this population that are of high concern for clinicians, patients, and families. These include high rates of relapse (estimated at 30%) [, ] and high rates of service disengagement, which range between 20% to 40% across specialized early intervention programs [ , ]. There are also challenges in sustaining improvements in symptoms and global functioning beyond the first 2 years of receiving specialized services [ ].
To date, specialized early intervention services have been restricted to models of care that are predominantly delivered in person. However, research suggests that after an initial period of receiving intensive treatment, some youth begin to experience the phenomenon of overengagement or engulfment by services and prefer a less intensive form of follow-up [, ]. Moreover, relying entirely on services delivered in person may not be feasible given that specialized early intervention is resource-intensive, with typical case manager-to-patient ratios being 1:25 and services often delivered in the community, which presents a range of challenges including, for example, loss of productivity and lack of access to transportation [ ]. Research is needed on new models of service delivery that are accessible, sustainable, engaging, and effective in supporting recovery in youth diagnosed with FEP. This is especially relevant during the first 5 years of the illness as this phase is considered as the “critical period” when clinical and psychosocial interventions have the highest impact [ ]. There is also a need for sustainable models of service delivery that can meet the need for psychosocial support beyond the time frame of receiving specialized early intervention for psychosis services.
Information and communication technologies (ICTs) offer a promising avenue for addressing the aforementioned clinical challenges by increasing access and quality of mental health services, and ultimately supporting the process of recovery for youth experiencing FEP [- ]. These technologies can potentially deliver services using a less intensive and more engaging format that is compatible with the culture of young people growing up in the 21st century. Our prior research conducted in a Canadian specialized early intervention for psychosis setting has shown that more than 90% of youth diagnosed with FEP receiving specialized services have regular access to the internet through computers and mobile devices, and many are receptive to the idea of using these technologies for mental health care [ , ]. However, despite the potential of technology for improving mental health care for youth, this area has received limited research attention in the Canadian field of early intervention for psychosis. Consequently, few studies exist on the acceptability, safety, and efficacy of digital health innovations to support recovery in youth diagnosed with FEP. Country-specific research on digital mental health care is relevant for advancing policy and practice that is contextualized to the needs of a nation’s population. For example, the Canadian health care landscape has particular characteristics that justify locally driven digital mental health service research. According to Martin and colleagues [ ], Canada has a health care system that is managed through provincial and territorial insurance plans; a vast geography; and a significant population diversity in terms of density, culture, and language [ ]. These characteristics are important considerations for adapting, implementing, and evaluating digital health service innovations in the Canadian context.
The Horyzons Intervention Platform
Members of our team have developed an innovative web-based therapeutic intervention platform, called Horyzons, designed to sustain the treatment benefits of early intervention for psychosis and to promote long-term social functioning . This intervention delivers evidence- and strengths-based targeted psychosocial interventions and is enhanced by a moderated online social networking environment. Youth diagnosed with FEP are guided through interactive games to identify, discuss, and develop key personal strengths in an online environment and in real life to address relapse risk factors and psychological well-being.
Specifically, Horyzons consists of interactive psychosocial interventions that are informed by evidence-based psychosocial interventions targeting key risk factors and salient domains in the early recovery process (including psychoeducation, vocational recovery, early warning signs of relapse, depression, social anxiety, personal strengths). Youth with FEP are prompted to practice newly acquired skills through over 350 purpose-developed behavioral activities, which are designed to bridge the gap between online therapy and real-world outcomes (eg, tips on how to use personal strengths to cope with stress, enhance well-being, or improve social connectedness). Horyzons also includes peer-to-peer web-based social networking that includes a web feed (or news feed) where youth with FEP and moderators can post comments and information, upload pictures and videos, and like different types of content, and includes moderation that is conducted by clinicians and peer support workers.
Clinician moderators provide guidance, monitor clinical status, and ensure safety of the social networking environment. Clinician moderators develop brief case formulations that are presented at weekly supervision meetings with the senior clinical research team. Clinician moderators send each youth tailored content suggestions (eg, step or action) weekly based on the young person’s needs, interests, and strengths. Suggestions appear on the young person’s home page, and they receive a text notification via the intervention platform’s SMS text messaging feature. Peer support moderators are young adults with lived experience who have received peer support training and who have been stable and in remission for a minimum of 2 years. Their role includes assisting with orientation to the Horyzons intervention platform, providing support, and fostering engagement. For example, peer support moderators reach out to new users or users that have not logged into the intervention platform after a week. They also post icebreakers and comments, like the posts of users, and engage in role modeling activities. They receive supervision from the clinical research team. The integrity of the moderation is ensured through a detailed moderation manual, and all moderators participate in weekly to biweekly group supervision sessions with senior members of the team. The intervention platform includes a comprehensive safety protocol following best practices in internet research involving vulnerable people , which considers 3 levels of security (ie, online safety, clinical safety, and system security). This safety protocol (including criteria and the process for withdrawal from the study) was successfully used in previous research on Horyzons [ , ].
Research and Development of Horyzons and Canadian Adaptations
A detailed description of the intervention, its development, and its subsequent adaptation for the Canadian context can be found in our previous published work [- ]. The first version of Horyzons was developed iteratively over a 30-month period following participatory design principles involving patients, clinical psychologists, computer programmers, health informatics experts, web and graphic designers, and professional writers. It was then pilot-tested on a sample of 20 young adults for its feasibility, acceptability, utility, and safety considerations over 1 month [ ]. Results showed that there were no dropouts and incidents (ie, adverse events or inappropriate use) during the pilot, and system use was high, with a total of 275 log-ins. Specifically, 70% of participants used the system for at least 3 weeks, 95% used the social networking features, and 60% completed at least 3 therapy modules. Moreover, all participants found the system easy-to-use, 70% considered it to be a useful long-term treatment option beyond discharge from early interventions services, 100% considered it to be safe, 85% would recommend it to others, and 100% reported moderation to be helpful. With respect to potential clinical improvements, 70% considered the therapy modules to be helpful, and 60% reported that Horyzons significantly increased their social connectedness. Paired samples t tests also revealed a significant reduction in depressive symptoms at follow-up (d=0.60; P=.02) [ ]. The safety strategy pertaining to the intervention study addressed both static (eg, personality trait variables) and dynamic factors (mental health status), and results indicated no clinical or security problems related to the use of Horyzons during the pilot study [ , ]. Additionally, a second version of Horyzons was evaluated in Australia within the context of a 5-year follow-up randomized controlled trial [ ], with results recently published. Horyzons has also been piloted in the United States, and preliminary analyses of within-group effect sizes demonstrated the greatest improvements in psychotic symptoms (d=0.65-0.81), followed by depressive symptoms (d=0.14-0.30) and social functioning (d=0.18) [ ]. Horyzons has also been adapted for young people at ultra-high risk for psychosis, and preliminary research suggests that it is effective in improving social functioning in this population [ ].
In this paper, we describe the research protocol informing this phase 2 live pilot study of HoryzonsCa. The protocol is largely informed by the Technology Acceptance Model (TAM) [, ], which is a well-established model focusing on individual perceptions pertaining to ease of use and usefulness of a technology, and the eHealth Adaptation Framework [ ], which considers broader factors such as language, culture, and context pertaining to localized implementation of an eHealth innovation. Our attention to the adaptation aspects of Horyzons is in alignment with strategies recommended in the literature to better capture complexities pertaining to the implementation of eHealth innovations [ ]. Moreover, during our analysis and interpretation of the results, we will also begin to consider models stemming from sociotechnically informed theories of change at the levels of individual, organization, and system, such as the nonadoption, abandonment, spread, scale-up, sustainability framework, which is intended to facilitate reflection on factors influencing adoption, nonadoption, abandonment, spread, scale-up, and sustainability of eHealth technologies [ ]. Such reflections will be important to inform the design of a scaled-up evaluation of the intervention, including the types of data that will be important to collect.
Objectives and Hypotheses
The aim of this study is to determine the acceptability, safety, and potential efficacy of HoryzonsCa in supporting recovery in young adults receiving specialized services for FEP. Specifically, our objectives are:
- To determine the acceptability of an online therapeutic platform (HoryzonsCa). Primary hypothesis: HoryzonsCa will be acceptable to patients. Our conceptualization of acceptability is partially informed by the TAM [ , ], which posits that attitudes toward using a technology influence intention (“acceptance”) to use a technology, which is a direct determinant of behavior [ , ]. The attitude construct is composed of perceived usefulness (an individual’s perception that using an information technology [IT] system will be helpful to them) and perceived ease of use (an individual’s perception that using an IT system will be free of effort) [ ]. Patients’ acceptance regarding the use of technologies in health care delivery has been extensively assessed using the TAM model and its derivatives, which highlight the importance of considering factors beyond the individual’s attitudes [ - ]. As such, we also include the concept of adoption in our conceptualization of acceptability, which “is defined as the intention, initial decision, or action to try or employ an innovation or evidence-based practice” [ ]. Thus, using this combined understanding of acceptability, in this study, the intervention will be considered acceptable if at least 70% of participants provide positive reports on general experience of the platform, 60% provide positive reports on perceived usefulness (helpfulness), 60% provide positive reports on ease of use, and 60% log onto the site at least 4 times over the 8-week follow-up. These percentages are hypothesized based on the results obtained from the original Horyzons pilot study [ ]. We will also determine acceptability through qualitatively assessing perceptions of HoryzonsCa in relation to likes, dislikes, perceived barriers and facilitators to using the platform, and ease of use/experiences of navigating and using its key features (eg, various psychoeducational modules or café) and through exploratory analysis of website use analytics.
- To assess the safety of the HoryzonsCa platform. Secondary hypothesis: HoryzonsCa will be safe, defined as no adverse events, reports, or incidents (eg, hospitalization, suicidal ideation, or disclosure to treatment team regarding harm) related to use of the platform from baseline assessment to 8 weeks follow-up, and at least 70% of participants report that they agree or strongly agree with the perceived safety of the platform and perceived confidentiality of information shared on the platform. We will also determine safety through qualitatively assessing perceptions of the HoryzonsCa platform (eg, experiences, concerns).
- To assess the potential efficacy of the HoryzonsCa intervention/platform. Secondary hypotheses: Participants will show moderate to large improvements (Cohen d≥0.5) on measures of social functioning (Social and Occupational Functioning Assessment Scale; Personal and Social Performance Scale) and either improvement of 1 point or no deterioration on the Clinical Global Impression Scale, from baseline to 8 weeks follow-up, as our primary foci for assessing potential efficacy. We will also assess for improvements in social support, self-esteem and perceived strengths, and symptoms. The analysis will help determine pre-post effect sizes on a number of variables conceptually targeted by the platform. These effect sizes will be used in a future clinical trial (eg, randomized controlled trial) for the purpose of statistical power calculation and estimation of sample size.
Study Design and Setting
This cohort study is implemented in a single center and applies a pre-post mixed methods (qualitative-quantitative convergent) design. Participants are recruited from Prevention and Early Intervention Program for Psychosis (PEPP)–Montreal, the Douglas Mental Health University Institute in Montreal, Quebec. This program provides a comprehensive range of services for young people diagnosed with FEP and follows best practice guidelines for real-world settings [, ]. Treatment includes psychiatric evaluation and follow-up, modified assertive case management tailored to meet the needs of young patients in the early phase of illness [ ] including support toward treatment goals (eg, illness education, return to work or school, and crisis intervention), family support and intervention, and psychosocial group interventions (eg, physical, recreational, recovery, and support).
Participants, Sample Size Considerations, Recruitment, and Orientation Process
The target sample is 20 to 25 participants that are patients receiving services from PEPP-Montreal. This sample size was determined by the following factors: feasibility of participant recruitment within a 2- to 3-month time frame, budget related to staffing of moderators, and ensuring adequate number of active users on the platform for the social networking features to function effectively. This sample size has also shown to be sufficient to pilot-test the feasibility and acceptability of Horyzons, as illustrated in the prior pilot study  while also acknowledging the limitation of this small sample size for efficacy testing (which is not the main objective of this pilot study). Participant inclusion and exclusion criteria are presented in and , respectively.
- Diagnosis of a psychotic disorder (including affective or nonaffective psychoses) by a clinician
- Receiving specialized services for a first-episode psychosis at the recruitment site
- Considered symptomatically stable and capable of interacting on the online platform and participating in focus groups and semistructured interviews, as judged by their primary clinicians (ie, psychiatrist, case manager)
- 18 years or older
- At low or at most moderate severity score (4 or below) on the suicidality item of the Brief Psychiatric Rating Scale, version 4 [ ] for the month preceding study entry. This criterion is to minimize risk for the need of urgent intervention regarding comments pertaining to suicidal ideation and plans posted on the social network or to the moderation team (given that the platform is not monitored 24/7) and to reduce potential anxiety or distress in other participants from overexposure to content focused on suicide.
- Able to nominate an emergency contact
- Intellectual disability
- Hospitalized at the time of recruitment
- Unable to speak or read English
- Diagnosis of antisocial personality disorder or borderline personality disorder
- In the acute phase of mania or psychosis to the extent that their mental status may soon require hospitalization or would impede the participant’s ability to provide informed consent or to participate in interviews and focus groups
In terms of the recruitment process, a member of the research team met with clinicians at the recruitment site to describe the project, including the rationale, objectives, inclusion and exclusion criteria, and methods. The treating clinicians screened all participants. The treating clinicians referred patients who fulfilled the aforementioned inclusion and exclusion criteria. The treating clinicians provided a copy of the study information handout to prospective participants when introducing the study so that they had time to consider the study and ask questions about it. Once participants completed screening for eligibility, were identified as being eligible to participate, and expressed interest to learn more about the study, they were contacted by a member of the research team to receive further details about the project, their participation, and the informed consent process. A study information brochure was provided to participants as a summary of the information contained in the consent form for user-friendly access to key information about the project.
Participants received detailed information concerning their participation including date, time, and duration of scheduled meetings, interviews, and activities related to study participation. In addition, they were contacted 24 hours prior to any scheduled interviews to confirm their participation. They were contacted according to their preferences either by phone, e-mail, or in person when attending the clinic. After providing written informed consent, the research assistant administered the baseline measures, the results of which were also used to confirm clinical stability and eligibility to participate. In a subsequent HoryzonsCa Orientation Meeting, participants were oriented to the HoryzonsCa website and provided with login information. Participants were then able to access the website, at their convenience, over a period of 8 weeks. They were encouraged to log into the website at least 1 time per week for a minimum of 15 minutes per visit. Access to HoryzonsCa was in addition to the services that participants already receive from PEPP-Montreal. The Horyzons safety protocol  was adapted for this study and included an assessment procedure for risk of suicide attempt. This assessment has been implemented by members of the team in the context of other psychosocial intervention research projects with the same clinical population. All research team members were trained in the safety protocol. Given that it was the first time the website and safety protocol was used in a Canadian context, the results of this study will help identify potential modifications to improve the safety protocol for future research on HoryzonsCa. The details of the adapted safety protocol and withdrawal criteria are provided in .
We collected quantitative and qualitative data through interview-based psychometric measures, self-reports, focus groups, and interviews. An initial interview consisted of completing a self-reported sociodemographic questionnaire, the Technology Access, Use, and Competency Questionnaire (TAUC-Q), and a combination of self-reported and interviewer-administered clinical measures (for social functioning, global improvement and therapeutic response, social support, self-esteem and perceived strengths, and symptoms). The TAUC-Q and the clinical measures were also administered during the exit interview at the 8-week follow-up along with the interviewer-administered Horyzons-Canada Acceptability, Usability, Safety, and Impact Questionnaire (HC-AUSI-Q). Participants were also invited to a HoryzonsCa Meet-up and Focus Group during the follow-up period.provides an overview of the schedule of assessments that took place across the duration of the follow-up.
Sociodemographic Characteristics and Access and Use of Technology
Participants were asked to complete a sociodemographic questionnaire (self-report) consisting of nine questions regarding gender, age, length of service use, highest level of education completed, ethnicity, vocational status, living situation, marital status, and annual income. Participants’ access, use, and attitudes in relation to technology was assessed using the TAUC-Q (self-report), which includes 10 questions regarding participants’ access and use of internet and mobile technology (eg, smartphone, computer, social media, text, or email) and perceived competency of technology. Their responses on these questionnaires will allow us to estimate the transferability of our results and to better understand participant experiences, use, and perceptions of the platform.
Acceptability (TAM components of perceived ease of use and perceived usefulness) was measured through the HC-AUSI-Q and website use analytics. The HC-AUSI-Q is an adaptation of questionnaires used in the HoryzonsCa phase 1 adaptation study [, ], the Horyzons Usability Questionnaire from the original Horyzons pilot study [ ], and the Website Analysis and Measurement Inventory [ ]. The interviewer-administered HC-AUSI-Q consists of a questionnaire and a semistructured interview that includes 16 close-ended and 10 open-ended questions on perceived ease of use, perceived usefulness, enjoyment, and safety. The questions pertaining to acceptability address the topics of general experience (eg, “I had a positive experience on Horyzons-Canada”), usefulness (eg, “Horyzons was useful to identify my warning signs for relapse”), and ease of use (eg, “Overall, the platform is easy to use”). Acceptability in terms of adoption was assessed using website use analytics (eg, frequency of log-ins and patterns of use over 8 weeks).
Safety, which considers 3 levels of security (ie, online safety, clinical safety, system security), was assessed using two specific questions (ie, “I felt safe on Horyzons-Canada” and “I felt like the information shared on Horyzons-Canada was confidential”) in the HC-AUSI-Q. In addition, any adverse events, reports, or incidents (eg, hospitalization, suicidal ideation, or disclosure to treatment team regarding harm) in relation to the use of the online system were carefully monitored and quantified over the study duration. The causal relationship between adverse events and use of HoryzonsCa was determined based on detailed documentation of each event and through discussion by members of the research and moderation team. All adverse events, reports, and incidents (eg, hospitalization, major deterioration in ability to function) were also submitted to the ethics review board for further examination.
In terms of primary clinical measures, social functioning was measured using the interviewer-administered Social and Occupational Functioning Assessment Scale that comprises a 100-point single item to assess social and occupational functioning  and the interviewer-administered Personal and Social Performance Scale that consists of a 100-point single item to assess functioning in four domains: socially useful activities (including work and study), personal and social relationships, self-care, and disturbing and aggressive behaviors [ ]. In terms of secondary clinical measures, global improvement and therapeutic response was assessed using the clinician-administered Clinical Global Impression Scale that consists of two items to assess global improvement and severity of illness [ ]. We also assessed social support using the self-reported Multidimensional Scale of Perceived Social Support that includes 12 items (eg, “I get the emotional help and support I need from my family) [ ]; self-esteem and perceived strengths using the self-reported Self-Esteem Rating Scale that consists of 40 items (eg, “I feel that I am a very competent person”) [ ], the self-reported Strengths Knowledge Scale that includes 8 items (eg, “I know what I do best”) [ ], and the self-reported Strengths Use Scale that consists of 14 items (eg, “I am regularly able to do what I do best”) [ , ]; and symptoms using the following interviewer-administered assessments: the Scale for the Assessment of Positive Symptoms that consists of 34 items in four symptom domains (hallucinations, delusions, bizarre behavior, and positive formal thought disorder) [ ]; the Scale for the Assessment of Negative Symptoms that includes 25 items in five symptom domains (affective flattening or blunting, alogia, avolition-apathy, anhedonia-asociality, and attention) [ ]; the Brief Psychiatric Rating Scale that includes 24 items assessing psychiatric symptoms such as somatic concern, anxiety, depression, and suicidality [ ]; and the Calgary Depression Scale that includes nine items assessing depressive symptoms such as depressed mood, hopelessness, and suicide [ ].
Qualitative data on the acceptability, safety, and impact of the intervention was mainly obtained at the 4 weeks follow-up through the HoryzonsCa Meet-up and Focus Group discussion meeting.
The research assistant took field notes on any additional comments from participants related to their experiences and perspectives of the intervention during the HoryzonsCa Initial Interview and Orientation Meeting (which occur at baseline) and the HoryzonsCa Exit Interview (which occur at the 8 weeks follow-up). Moreover, participants were asked open-ended questions at the end of the HC-AUSI-Q, which were conducted during the HoryzonsCa Exit Interview.
The HoryzonsCa Meet-Up and Focus Group (120 minutes) occurred midway through the 8 weeks of follow-up. We invited all participants to this Meet-up and Focus Group Meeting, and aimed to have approximately 4 to 8 participants each session. The focus groups were facilitated by the project lead with the support of a research assistant. Before having a focused discussion on the platform, the facilitator (who does not have any clinical role or therapeutic relationship with the participants) welcomed participants, provided an overview of the meet-up objectives, and provided logistic information (eg, duration of group and scheduled breaks). After that, participants were invited to share their experiences and perspectives of the system, and feedback on factors that support or hinder its use. Specifically, the topics of the focus group were general impressions of the HoryzonsCa intervention platform (eg, likes and dislikes), how easy it is to use, how useful the intervention platform has been for well-being, how the intervention platform can be improved to better meet the needs of Canadian youth, and any other suggestions in implementing and evaluating HoryzonsCa. The interview guides for the focus groups were adapted from our phase 1 adaptation study protocol, which was informed by the eHealth Adaptation Framework . The facilitator encouraged an open discussion on the perceptions of the platform. A moderator was then invited to the room to present participants with the key components of the platform and provide tips on its use. At the end of the meeting, the lead facilitator reiterated the purpose of the focus group, summarized what was said, described the next steps, and gave participants the opportunity to bring up discussion points that were not addressed previously but which were of importance to them. These meetings were audio recorded.
During the HoryzonsCa Exit Interview (30 minutes), the TAUC-Q and the clinical measures used for the initial interview and the HC-AUSI-Q were completed. The research assistant took field notes including any additional comments that the participant provided related to the website or participation in the project.
We also collected data on recruitment rates, appropriateness of eligibility criteria, and the project team’s experience of preparing for and implementing the intervention (based on team meeting notes) mainly to inform the feasibility and design for conducting a larger implementation study in the future.
This study is currently in the data management and analysis phase. The quantitative data from the outcome measures was entered into an Excel file (Microsoft Corporation). All outcome measures at baseline and at the 8 weeks follow-up are either self-reported or interviewer-administered, and those that were completed by the participant were checked by the interviewer for completeness, as sometimes questions are not answered simply because of rushing through, filling in errors, or oversight. Any missing data will be discussed using a team approach to first understand its nature (eg, participant nonresponse, research assistant error in data entry, or participant dropout). We will consider the extent and the type of the missing data (eg, missing completely at random, missing at random, and missing not at random), and then determine the best approach (eg, multiple imputation or regression imputation) to handle the missing data in consultation with a statistician . Once all participants completed the HoryzonsCa Exit Interview, website use data was exported to an Excel file.
The qualitative data from the focus group audio recordings will be transcribed and anonymized. The field notes from the HoryzonsCa Initial Interview and Orientation Meeting and the HoryzonsCa Exit Interview taken by the research assistant will be typed by the research assistant. All qualitative data will be uploaded into the latest version of Atlas.ti software (ATLAS.ti Scientific Software Development GmbH), a coding software package.
Following the convergent mixed methods model, the quantitative and qualitative data will first be analyzed separately and then considered for an integrated analysis of the findings . The quantitative data (including website use data) will first be assessed using descriptive statistics (eg, frequencies). Specifically, to evaluate the acceptability and safety of HoryzonsCa, we will analyze quantitative feedback from the HC-AUSI-Q and website use by calculating proportions (ie, percentage of participants who indicate agree or strongly agree for specific items related to acceptability and safety; percentage of participants with at least 4 log-ins over the 8 weeks follow-up). To assess the potential efficacy of HoryzonsCa, paired samples t tests will be conducted on social functioning and clinical measures, and within-group effect sizes (Cohen d) will be reported for statistically significant changes between baseline and the 8 weeks follow-up. We will make Bonferroni corrections to reduce the inflation of alpha for multiple comparisons and determine a minimal clinically important difference to detect important changes over time. In addition, exploratory analysis of potential meditators and moderators of treatment effects will be conducted; for example, we will analyze the associations between website use and treatment effects to estimate the moderating role of website use. We will conduct the checks of assumptions prior to conducting analysis, including normality of the data, and will transform the data if needed. Two members of the research team (eg, a postdoctoral fellow and a research assistant) will review data quality (eg, missing data) and determine the best approach (eg, multiple imputation) to handle the missing data in consultation with a statistician and senior members of the research team. Quantitative data analysis will be supported using SPSS (IBM Corp; or R [R Foundation for Statistical Computing]).
Qualitative data (including qualitative feedback from the HC-AUSI-Q and the qualitative data from the interviews and focus groups) will be analyzed and reviewed for themes related to acceptability, perceived benefits, safety, barriers, and facilitators of using HoryzonsCa. Two members of the research team will review all the transcripts, codevelop a coding framework, and conduct a thematic analysis  in consultation with the project lead. These will be identified based on their salience with the research objectives and in relation to patterned responses. The coding categories will reflect questions asked during the interview and perspectives that emerged frequently in the data. Qualitative data analysis for the focus groups will be supported using the latest version of Atlas.ti software, a coding software package.
This study was funded by the Brain and Behavior Research Foundation (United States), Quebec Health Research Funding Agency (Canada), and the Canada Research Chairs Program (Canada), and was approved by the Research Ethics Board of the Centre intégré universitaire de santé et de services sociaux de l'Ouest-de-l'Île-de-Montréal on April 11, 2018 (#IUSMD 17-54). The study was registered as a clinical trial at  (ISRCTN43182105). Recruitment was initiated on May 10, 2018, and data collection occurred between August 16, 2018, to April 29, 2019. A total of 48 individuals were approached for the study, from which 20 were excluded (17 declined to participate, 1 was not reachable, 1 did not meet inclusion criteria, and 1 met exclusion criteria). The 28 remaining individuals provided informed consent and were invited to complete a baseline assessment, from which 4 did not attend for various reasons (eg, no longer interested, no longer attending clinic, or not feeling well). Among the 24 participants that completed the baseline assessment, 3 dropped out of the study before being given access to the intervention, and 1 was excluded from the intervention soon after team discussion of the baseline assessment due to not meeting inclusion criteria (ie, clinical stability). Upon completion of the baseline assessment and confirmation of eligibility criteria, a final sample of 20 participants were given access to the intervention and none of these were lost at 8 weeks follow-up. Participants were recruited over a 9-month duration (which was 3 times longer than our originally anticipated 3-month timeline), with a recruitment rate of approximately 3 participants per month. A total of 9 participants attended the focus group meetings of approximately 120 minutes each session (4 attended more than once). No adverse events related to the intervention occurred during the live pilot implementation. Further details on the results are expected to be submitted for publication in 2021.
Importance of This Study
Psychotic disorders can have a profound impact on the individual, their caregivers, and society with regard to a loss of quality of life and productivity. Despite advances in early intervention, there remain ongoing challenges in treating this population, including preventing relapse and service disengagement, and sustaining improvements in symptoms and functioning over the long term. As such, accessible, sustainable, and engaging psychosocial services are needed to optimize care and outcomes for this population. The COVID-19 pandemic and public health guidelines for social distancing have compounded the urgent need for new models of service delivery to provide specialized services to patients with FEP. This study addresses this gap in health service research through leveraging HoryzonsCa, an online intervention with the potential to support recovery and prevent relapse in patients with FEP, during and post COVID-19.
In this paper, we have described the study protocol for pilot-testing HoryzonsCa, and it is the first study to implement and evaluate a live version of this online intervention in a Canadian context. This study is distinguished from previous research on Horyzons, as it is the first to be based on a systematic adaptation process [, ] that considers geographical, cultural, and health care contexts. There is evidence to suggest that embarking on an adaptation process of an eHealth intervention (eg, considering language, culture, and context) can contribute positively to its adoption and effectiveness [ ].
Furthermore, in comparison to Australia, the Canadian context lags in terms of implementation of digital health innovations . Moreover, peer support legislation, policy, training, and provision is uneven across Canada [ ]. These factors influenced our implementation of the intervention, particularly in relation to the hiring and training of the moderation team. For example, in previous Australian research on Horyzons, the clinician moderators were already working at the recruitment site delivering face-to-face services, with part of their time allocated to providing online moderation for Horyzons, whereas in the current context, the clinicians were ambivalent about making online moderation as part of their role, which may be due to the lack of experience and training in the delivering of online interventions [ ]. As such, for this pilot study we decided to recruit a moderator external to the setting. Additionally, we had a limited pool of trained and experienced peer support worker workforce to recruit from and had turnover of 2 peer support workers (for reasons unrelated to the intervention) before being able to recruit a peer support worker that completed the duration of the project. However, it is noteworthy that this worker had limited experience in the delivery of online interventions and no employed experience as a peer support worker. As such, this pilot testing provides key information on the factors to consider in the implementation of this complex intervention (including recruitment and training of moderators and participant recruitment, retention, and engagement with the intervention). We will further detail and interpret these aspects of implementation using sociotechnical theory in our upcoming results report. In summary, through this study, we will gain insights into the acceptability, safety, and potential efficacy of HoryzonsCa, as well as key information that will support decision-making regarding scaling up the implementation and evaluation of this online intervention in Canada and abroad.
Strengths and Limitations
There are several strengths to this research. First, in terms of preparing the intervention, we made adaptations to Horyzons to optimize its transferability from Australia to Canada, using a systematic approach and an eHealth adaptation framework, prior to its implementation [, ]. The mixed methods design in this study is expected to provide an in-depth understanding of user experience and perspectives of HoryzonsCa. This mixed methods approach allows researchers to obtain a comprehensive view about the user experience and evaluate the potential impact of implementing such digital health innovations in Canadian health care settings.
Limitations of this study include a small sample size (N=20) and a single group, pre-post design. Due to this small sample size and a lack of a control group, we may not be able to detect treatment effects and impact of the intervention. Additionally, due to the short duration of follow-up (ie, 8 weeks), our findings regarding the acceptability, safety, and potential efficacy of HoryzonsCa will need to be interpreted with caution. Furthermore, our approach to exclude participants that score as moderately severe to extremely severe on suicidality (ie, due to recent suicidal ideations and behaviors, to mitigate the need for urgent intervention on a platform that is not monitored 24/7, and to reduce potential overexposure to comments that may be disruptive or distressing for other participants in the intervention) is not necessarily supported by predictive research and could even potentially exclude participants who may benefit from such an intervention [, ]. Future research on HoryzonsCa should consider how such an online intervention platform can facilitate a recovery-oriented approach for all participants despite their score on a suicidality assessment while at the same time optimizing the safety of the online group environment. In addition, although the field of digital mental health is rapidly growing [ , ], we acknowledge that the limited evidence on the implementation of digital mental health interventions for people diagnosed with severe mental health disorders such as psychosis in Canada may limit the scope of our interpretations regarding the results. Nonetheless, the findings will help inform decision-making for scaling up the evaluation on HoryzonsCa to a larger implementation study.
Considering daily use of ICTs among young people and an increasing need for new models of mental health service delivery in the context of the COVID-19 pandemic, research on digital health innovations such as online psychosocial interventions are needed to prevent relapses and support long-term recovery in young people experiencing FEP. However, limited attention has been given to how digital health innovations can be implemented and evaluated in Canada in the context of early interventions for psychosis. In this study, we have reported on the protocol for pilot-testing HoryzonsCa as a follow-up to our phase 1 adaptation study [, ]. Specifically, this pilot study will provide preliminary evidence on the acceptability (including actual use), safety, and potential efficacy of HoryzonsCa for Canadian young adults with FEP, and an in-depth understanding of user experience and perspectives of HoryzonsCa (eg, perceived barriers and facilitators to using the platform). Our pilot study will inform the development of a research protocol (eg, including pre-post effect sizes on a number of variables conceptually targeted by the intervention, statistical power calculation, or sample size) for a larger implementation study of HoryzonsCa and that also considers sociotechnical factors pertaining to implementation as well as the experiences and perspectives of clinicians and peer support workers on moderating the intervention.
This Phase 2 Pilot Study of the Horyzons-Canada research program was supported by a Young Investigator Grant from the Brain and Behavior Research Foundation (United States) awarded to SL. During this study, SL was supported in part by a Research Scholar Salary Award from the Fonds de recherche du Québec–Santé and subsequently in part by a New Investigator Salary Award from the Canadian Institutes of Health Research. SL is currently funded by the Canada Research Chairs Program. MAJ was supported by an investigator grant (APP1177235) from the National Health and Medical Research Council of Australia.
SL, MAJ, and JG were involved in designing the overall study, with critical input from RJ and ML. SL, SD, and GE contributed to finalizing adaptations to the intervention and preparing it for evaluation (eg, website use analytics). SL wrote the detailed protocol submitted for ethics review, and GE assisted in preparing materials for protocol implementation. SL and HL prepared the initial draft of the manuscript based on content in the protocol. All authors contributed to revising the current manuscript in a critical way.
Conflicts of Interest
SL reports a recent research grant from Hoffman-La Roche, pertaining to an upcoming phase 3 study on the implementation and evaluation of HoryzonsCa; MAJ, JG, and RJ are coinvestigators on this grant. RJ served as speaker and member of advisory board committees for Pfizer, Janssen, Bristol Myers Squibb, Sunovian, Myelin and Associates, Otsuka, Lundbeck, Shire, and Perdue. He also received grants from Janssen, Bristol Myers Squibb, Otsuka, Lundbeck, Astra Zeneca, and HLS Therapeutics Inc. All of these are unrelated to this study. ML reports grants from Otsuka Lundbeck Alliance, Hoffman-La Roche (pertaining to an upcoming phase 3 study on the implementation and evaluation of HoryzonsCa), diaMentis, personal fees from Otsuka Canada, personal fees from Lundbeck Canada, grants and personal fees from Janssen, and personal fees from MedAvante-Prophase outside the submitted work.
Safety protocol and withdrawal criteria.DOCX File , 23 KB
Schedule of assessments.DOCX File , 20 KB
- Perälä J, Suvisaari J, Saarni SI, Kuoppasalmi K, Isometsä E, Pirkola S, et al. Lifetime prevalence of psychotic and bipolar I disorders in a general population. Arch Gen Psychiatry 2007 Jan;64(1):19-28. [CrossRef] [Medline]
- Aas M, Dazzan P, Mondelli V, Melle I, Murray RM, Pariante CM. A systematic review of cognitive function in first-episode psychosis, including a discussion on childhood trauma, stress, and inflammation. Front Psychiatry 2014 Jan 08;4:182. [CrossRef] [Medline]
- Diagnostic and Statistical Manual of Mental Disorders, 4th edition. Washington, DC: American Psychiatric Association; 2000.
- Burden of mental and behavioural disorders. In: The World Health Report 2001: Mental Health: New Understanding, New Hope. Geneva: World Health Organization; 2001:19-45.
- Nature. Where next with psychiatric illness? Nature 1988 Nov 10;336(6195):95-96. [CrossRef] [Medline]
- GBD 2016 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 328 diseases and injuries for 195 countries, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet 2017 Sep 16;390(10100):1211-1259 [FREE Full text] [CrossRef] [Medline]
- Malla A, Lal S, Vracotas NC, Goldberg K, Joober R. Early intervention in psychosis: specialized intervention and early case identification. Encephale 2010 Mar;36 Suppl 3:S38-S45. [CrossRef] [Medline]
- Malla A, Norman R, McLean T, Scholten D, Townsend L. A Canadian programme for early intervention in non-affective psychotic disorders. Aust N Z J Psychiatry 2003 Aug;37(4):407-413. [CrossRef] [Medline]
- Alvarez-Jiménez M, Parker AG, Hetrick SE, McGorry PD, Gleeson JF. Preventing the second episode: a systematic review and meta-analysis of psychosocial and pharmacological trials in first-episode psychosis. Schizophr Bull 2011 May;37(3):619-630 [FREE Full text] [CrossRef] [Medline]
- Petersen L, Jeppesen P, Thorup A, Abel M, Øhlenschlaeger J, Christensen T, et al. A randomised multicentre trial of integrated versus standard treatment for patients with a first episode of psychotic illness. BMJ 2005 Sep 17;331(7517):602 [FREE Full text] [CrossRef] [Medline]
- Harvey PO, Lepage M, Malla A. Benefits of enriched intervention compared with standard care for patients with recent-onset psychosis: a metaanalytic approach. Can J Psychiatry 2007 Jul;52(7):464-472. [CrossRef] [Medline]
- Craig TK, Garety P, Power P, Rahaman N, Colbert S, Fornells-Ambrojo M, et al. The Lambeth Early Onset (LEO) Team: randomised controlled trial of the effectiveness of specialised care for early psychosis. BMJ 2004 Nov 06;329(7474):1067 [FREE Full text] [CrossRef] [Medline]
- Kane JM, Robinson DG, Schooler NR, Mueser KT, Penn DL, Rosenheck RA, et al. Comprehensive versus usual community care for first-episode psychosis: 2-year outcomes from the NIMH RAISE Early Treatment Program. Am J Psychiatry 2016 Apr 01;173(4):362-372 [FREE Full text] [CrossRef] [Medline]
- Addington DE, Patten SB, McKenzie E, Addington J. Relationship between relapse and hospitalization in first-episode psychosis. Psychiatr Serv 2013 Aug 01;64(8):796-799. [CrossRef] [Medline]
- Malla A, Norman R, Bechard-Evans L, Schmitz N, Manchanda R, Cassidy C. Factors influencing relapse during a 2-year follow-up of first-episode psychosis in a specialized early intervention service. Psychol Med 2008 Nov;38(11):1585-1593. [CrossRef] [Medline]
- Doyle R, Turner N, Fanning F, Brennan D, Renwick L, Lawlor E, et al. First-episode psychosis and disengagement from treatment: a systematic review. Psychiatr Serv 2014 May 01;65(5):603-611. [CrossRef] [Medline]
- Lal S, Malla A. Service engagement in first-episode psychosis: current issues and future directions. Can J Psychiatry 2015 Aug;60(8):341-345 [FREE Full text] [CrossRef] [Medline]
- Bertelsen M, Jeppesen P, Petersen L, Thorup A, Øhlenschlaeger J, le Quach P, et al. Five-year follow-up of a randomized multicenter trial of intensive early intervention vs standard treatment for patients with a first episode of psychotic illness: the OPUS trial. Arch Gen Psychiatry 2008 Jul;65(7):762-771. [CrossRef] [Medline]
- Lal S, Ungar M, Malla A, Leggo C, Suto M. Impact of mental health services on resilience in youth with first episode psychosis: a qualitative study. Adm Policy Ment Health 2017 Jan;44(1):92-102. [CrossRef] [Medline]
- Lal S, Abdel-Baki A, Sujanani S, Bourbeau F, Sahed I, Whitehead J. Perspectives of young adults on receiving telepsychiatry services in an urban early intervention program for first-episode psychosis: a cross-sectional, descriptive survey study. Front Psychiatry 2020;11:117. [CrossRef] [Medline]
- Birchwood M, Todd P, Jackson C. Early intervention in psychosis. The critical period hypothesis. Br J Psychiatry Suppl 1998;172(33):53-59. [Medline]
- Álvarez-Jiménez M, Gleeson JF, Bendall S, Lederman R, Wadley G, Killackey E, et al. Internet-based interventions for psychosis: a sneak-peek into the future. Psychiatr Clin North Am 2012 Sep;35(3):735-747. [CrossRef] [Medline]
- Lal S, Adair CE. E-mental health: a rapid review of the literature. Psychiatr Serv 2014 Jan 01;65(1):24-32. [CrossRef] [Medline]
- Lal S, Dell'Elce J, Malla AK. Technology access and use among young adults with a first episode of psychosis. Psychiatr Serv 2015 Jul;66(7):764-765. [CrossRef] [Medline]
- Lal S, Dell'Elce J, Tucci N, Fuhrer R, Tamblyn R, Malla A. Preferences of young adults with first-episode psychosis for receiving specialized mental health services using technology: a survey study. JMIR Ment Health 2015;2(2):e18 [FREE Full text] [CrossRef] [Medline]
- Martin D, Miller AP, Quesnel-Vallée A, Caron NR, Vissandjée B, Marchildon GP. Canada's universal health-care system: achieving its potential. Lancet 2018 Apr 28;391(10131):1718-1735 [FREE Full text] [CrossRef] [Medline]
- Alvarez-Jimenez M, Bendall S, Lederman R, Wadley G, Chinnery G, Vargas S, et al. On the HORYZON: moderated online social therapy for long-term recovery in first episode psychosis. Schizophr Res 2013 Jan;143(1):143-149. [CrossRef] [Medline]
- Sharkey S, Jones R, Smithson J, Hewis E, Emmens T, Ford T, et al. Ethical practice in internet research involving vulnerable people: lessons from a self-harm discussion forum study (SharpTalk). J Med Ethics 2011 Dec;37(12):752-758. [CrossRef] [Medline]
- Alvarez-Jimenez M, Koval P, Schmaal L, Bendall S, O'Sullivan S, Cagliarini D, et al. The Horyzons project: a randomized controlled trial of a novel online social therapy to maintain treatment effects from specialist first-episode psychosis services. World Psychiatry 2021 Jun;20(2):233-243. [CrossRef] [Medline]
- Lal S, Gleeson J, Rivard L, D'Alfonso S, Joober R, Malla A, et al. Adaptation of a digital health innovation to prevent relapse and support recovery in youth receiving services for first-episode psychosis: results from the Horyzons-Canada phase 1 study. JMIR Form Res 2020 Oct 29;4(10):e19887 [FREE Full text] [CrossRef] [Medline]
- Lal S, Gleeson J, Malla A, Rivard L, Joober R, Chandrasena R, et al. Cultural and contextual adaptation of an eHealth intervention for youth receiving services for first-episode psychosis: adaptation framework and protocol for Horyzons-Canada phase 1. JMIR Res Protoc 2018 Apr 23;7(4):e100 [FREE Full text] [CrossRef] [Medline]
- D'Alfonso S, Phillips J, Valentine L, Gleeson J, Alvarez-Jimenez M. Moderated online social therapy: viewpoint on the ethics and design principles of a web-based therapy system. JMIR Ment Health 2019 Dec 04;6(12):e14866 [FREE Full text] [CrossRef] [Medline]
- Gleeson JF, Lederman R, Wadley G, Bendall S, McGorry PD, Alvarez-Jimenez M. Safety and privacy outcomes from a moderated online social therapy for young people with first-episode psychosis. Psychiatr Serv 2014 Apr 01;65(4):546-550. [CrossRef] [Medline]
- Ludwig KA, Browne JW, Nagendra A, Gleeson JF, D'Alfonso S, Penn DL, et al. Horyzons USA: a moderated online social intervention for first episode psychosis. Early Interv Psychiatry 2021 Apr;15(2):335-343. [CrossRef] [Medline]
- Alvarez-Jimenez M, Gleeson JF, Bendall S, Penn DL, Yung AR, Ryan RM, et al. Enhancing social functioning in young people at Ultra High Risk (UHR) for psychosis: a pilot study of a novel strengths and mindfulness-based online social therapy. Schizophr Res 2018 Dec;202:369-377. [CrossRef] [Medline]
- Barrera M, Castro FG, Strycker LA, Toobert DJ. Cultural adaptations of behavioral health interventions: a progress report. J Consult Clin Psychol 2013 Apr;81(2):196-205 [FREE Full text] [CrossRef] [Medline]
- Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989 Sep;13(3):319-340. [CrossRef]
- Holden RJ, Karsh BT. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010 Feb;43(1):159-172 [FREE Full text] [CrossRef] [Medline]
- Shachak A, Kuziemsky C, Petersen C. Beyond TAM and UTAUT: future directions for HIT implementation research. J Biomed Inform 2019 Dec;100:103315 [FREE Full text] [CrossRef] [Medline]
- Greenhalgh T, Abimbola S. The NASSS Framework - a synthesis of multiple theories of technology implementation. Stud Health Technol Inform 2019 Jul 30;263:193-204. [CrossRef] [Medline]
- KL C, Karsh BT. The patient technology acceptance model (PTAM) for homecare patients with chronic illness. Proc Hum Factors Ergonomics Soc Annu Meeting 2016 Nov 06;50(10):989-993. [CrossRef]
- Or CK, Karsh BT. A systematic review of patient acceptance of consumer health information technology. J Am Med Inform Assoc 2009;16(4):550-560 [FREE Full text] [CrossRef] [Medline]
- Wilson EV, Lankton NK. Modeling patients' acceptance of provider-delivered e-health. J Am Med Inform Assoc 2004;11(4):241-248 [FREE Full text] [CrossRef] [Medline]
- Yarbrough AK, Smith TB. Technology acceptance among physicians: a new take on TAM. Med Care Res Rev 2007 Dec;64(6):650-672. [CrossRef] [Medline]
- Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
- Lukoff D, Nuechterlein KH, Ventura J. Appendix A. Manual for Expanded Brief Psychiatric Rating Scale (BPRS). Schizophr Bull 1986;12(4):594-602. [CrossRef]
- A professional service with over 12 years experience of all types of private and public sector websites and portals worldwide. WAMMI. URL: http://www.wammi.com/about.html [accessed 2020-11-18]
- Morosini PL, Magliano L, Brambilla L, Ugolini S, Pioli R. Development, reliability and acceptability of a new version of the DSM-IV Social and Occupational Functioning Assessment Scale (SOFAS) to assess routine social functioning. Acta Psychiatr Scand 2000 Apr;101(4):323-329. [Medline]
- Guy W. Clinical global impression. In: ECDEU Assessment Manual for Psychopharmacology. Rockville, MD: US Department of Health, Education, and Welfare, Public Health Service, Alcohol, Drug Abuse, and Mental Health Administration, National Institute of Mental Health, Psychopharmacology Research Branch, Division of Extramural Research Programs; 1976:217-222.
- Zimet GD, Dahlem NW, Zimet SG, Farley GK. The multidimensional scale of perceived social support. J Pers Assess 1988 Mar;52(1):30-41. [CrossRef]
- Nugent WR, Thomas JW. Validation of a clinical measure of self-esteem. Res Soc Work Pract 2016 Aug 18;3(2):191-207. [CrossRef]
- Govindji R, Linley PA. Strengths use, self-concordance and well-being: implications for strengths coaching and coaching psychologists. Int Coaching Psychol Rev 2007;2(2):143-153 [FREE Full text]
- Wood AM, Linley PA, Maltby J, Kashdan TB, Hurling R. Using personal and psychological strengths leads to increases in well-being over time: a longitudinal study and the development of the strengths use questionnaire. Pers Individual Differences 2011 Jan;50(1):15-19. [CrossRef]
- Andreasen NC. The Scale for the Assessment of Positive Symptoms (SAPS). Iowa City, IA: The University of Iowa; 1984.
- Andreasen NC. The Scale for the Assessment of Negative Symptoms (SANS). Iowa City, IA: The University of Iowa; 1983.
- Addington D, Addington J, Maticka-Tyndale E. Assessing depression in schizophrenia: the Calgary Depression Scale. Br J Psychiatry Suppl 1993 Dec(22):39-44. [Medline]
- Kang H. The prevention and handling of the missing data. Korean J Anesthesiol 2013 May;64(5):402-406 [FREE Full text] [CrossRef] [Medline]
- Creswell JW, Klassen AC, Plano Clark VL, Smith KC. Best Practices for Mixed Methods Research in Health Sciences. Bethesda, MD: National Institutes of Health; 2011.
- Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol 2006 Jan;3(2):77-101 [FREE Full text] [CrossRef]
- Online social therapy to support recovery in youth receiving mental health services. ISRCTN Registry. URL: https://www.isrctn.com/ISRCTN43182105 [accessed 2021-11-12]
- Lal S. E-mental health: promising advancements in policy, research, and practice. Healthc Manage Forum 2019 Mar;32(2):56-62 [FREE Full text] [CrossRef] [Medline]
- Making the case for peer support: report to the peer support project committee of the Mental Health Commission of Canada. Mental Health Commission of Canada. 2010. URL: https://www.mentalhealthcommission.ca/sites/default/files/2016-07/MHCC_Making_the_Case_for_Peer_Support_2016_Eng.pdf [accessed 2021-08-11]
- Mulder R, Newton-Howes G, Coid JW. The futility of risk prediction in psychiatry. Br J Psychiatry 2016 Oct;209(4):271-272. [CrossRef] [Medline]
- Chan MK, Bhatti H, Meader N, Stockton S, Evans J, O'Connor RC, et al. Predicting suicide following self-harm: systematic review of risk factors and risk scales. Br J Psychiatry 2016 Oct;209(4):277-283. [CrossRef] [Medline]
|FEP: first-episode psychosis|
|HC-AUSI-Q: Horyzons-Canada Acceptability, Usability, Safety, and Impact Questionnaire|
|ICT: information and communication technology|
|IT: information technology|
|PEPP: Prevention and Early Intervention Program for Psychosis|
|TAM: Technology Acceptance Model|
|TAUC-Q: Technology Access, Use, and Competency Questionnaire|
Edited by G Eysenbach; submitted 23.02.21; peer-reviewed by N Kozloff, J Bierbooms, L Balcombe, R Lederman; comments to author 31.03.21; revised version received 18.08.21; accepted 07.09.21; published 07.12.21Copyright
©Shalini Lal, John F Gleeson, Simon D'Alfonso, Geraldine Etienne, Ridha Joober, Martin Lepage, Hajin Lee, Mario Alvarez-Jimenez. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 07.12.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.