This paper is in the following e-collection/theme issue:

Methods and Feasibility Studies (379) Usability Evaluation Case Studies (191) Depression and Mood Disorders; Suicide Prevention (1363) Anxiety and Stress Disorders (1053)

Published on in Vol 6, No 8 (2017): August

Conventional Cognitive Behavioral Therapy Facilitated by an Internet-Based Support System: Feasibility Study at a Psychiatric Outpatient Clinic

Conventional Cognitive Behavioral Therapy Facilitated by an Internet-Based Support System: Feasibility Study at a Psychiatric Outpatient Clinic

Conventional Cognitive Behavioral Therapy Facilitated by an Internet-Based Support System: Feasibility Study at a Psychiatric Outpatient Clinic

Authors of this article:

Kristoffer NT Månsson1, 2, 3 Author Orcid Image ;   Hugo Klintmalm4 Author Orcid Image ;   Ragnar Nordqvist4 Author Orcid Image ;   Gerhard Andersson3, 5 Author Orcid Image
Article Authors Cited by (9) Tweetations (5) Metrics

    Original Paper

    1Department of Psychology, Stockholm University, Stockholm, Sweden

    2Department of Adult Psychiatry, PRIMA Barn- och Vuxenpsykiatri, Stockholm, Sweden

    3Centre for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden

    4Division of Psychology, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden

    5Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden

    Corresponding Author:

    Kristoffer NT Månsson, PhD

    Department of Psychology

    Stockholm University

    Frescati Hagväg 8

    Stockholm, 106 91

    Sweden

    Phone: 46 (0)705803267

    Email: kristoffer.mansson@psychology.su.se


    Background: Cognitive behavioral therapies have been shown to be effective for a variety of psychiatric and somatic disorders, but some obstacles can be noted in regular psychiatric care; for example, low adherence to treatment protocols may undermine effects. Treatments delivered via the Internet have shown promising results, and it is an open question if the blend of Internet-delivered and conventional face-to-face cognitive behavioral therapies may help to overcome some of the barriers of evidence-based treatments in psychiatric care.

    Objective: We evaluated the feasibility of an Internet-based support system at an outpatient psychiatric clinic in Sweden. For instance, the support system made it possible to send messages and share information between the therapist and the patient before and after therapy sessions at the clinic.

    Methods: Nine clinical psychologists participated and 33 patients were enrolled in the current study. We evaluated the usability and technology acceptance after 12 weeks of access. Moreover, clinical data on common psychiatric symptoms were assessed before and after the presentation of the support system.

    Results: In line with our previous study in a university setting, the Internet-based support system has the potential to be feasible also when delivered in a regular psychiatric setting. Notably, some components in the system were less frequently used. We also found that patients improved on common outcome measures for depressive and anxious symptoms (effect sizes, as determined by Cohen d, ranged from 0.20-0.69).

    Conclusions: This study adds to the literature suggesting that modern information technology could be aligned with conventional face-to-face services.

    JMIR Res Protoc 2017;6(8):e158

    doi:10.2196/resprot.6035

    Keywords

    cognitive behavioral therapy; Internet-treatment; psychiatry; blended therapy 


    During the last decade, there has been a growing interest in alternative ways of delivering psychological treatments. The development of Internet-delivered interventions targeting common psychiatric and somatic disorders is one promising method [1,2]. Therapist-guided Internet-delivered treatments based on cognitive behavioral therapy (ICBT) have commonly shown promising effects in studies of both research studies (efficacy) [3], and in more clinically representative settings (effectiveness) [4]. A growing body of evidence suggests similar outcomes of ICBT and conventional face-to-face cognitive behavioral therapies (CBT) [2], with therapist-guided ICBT being less time-consuming for the clinician. Using the Internet to deliver health care may open new avenues to treatment, especially in societies where the distance to care is far away. Thus, ICBT has the potential to increase access to evidence-based psychological treatment [3].

    In primary or psychiatric care, there may be some obstacles of providing conventional CBT delivered face-to-face. For instance, therapists may be prone to drift away from implementing effective interventions (ie, therapist drift) [5], and they may also fail to adhere to evidence-based treatment manuals [6]. One way to overcome such obstacles could be to provide computer-assisted support in therapeutic work [7]. In a previous study, we developed an Internet-based support system to facilitate the delivery of conventional CBT [8]. The basic idea of the system is to support the delivery of CBT in a clinical setting where the therapist meets their patients face to face. By providing support, our objective was to improve the delivery of regular treatment components present in CBT, for example, homework assignments. A potential strength of the approach is that it conceptually shifts the focus of research away from specific digital interventions towards the system level (ie, capable of delivering many interventions). The approach also highlights the potential impact of introducing digital communication channels in face-to-face psychotherapy. The initial study showed some promising findings in terms of user experiences (eg, the ease of providing written information as a complement to the therapy sessions), and we observed reliable reductions of depressive and anxious symptoms. The study was conducted in a university setting, and there is a need to test the support system in clinical psychiatric care (eg, with a more severe clinical population and across different disorders).

    This feasibility study aimed at evaluating the experiences and effects of an Internet-based support system used as an adjunct to conventional CBT delivered face to face. The system was designed to support the delivery of face-to-face CBT and not replace in-session treatment activities. The system was used for communication between therapy sessions, sharing media, and clarifying homework assignments [8]. Clinicians and patients were recruited from a psychiatric clinic in Sweden, and the users were given access to the support system during 12 weeks. At follow-up, we evaluated support system usability and technology acceptance. Moreover, self-report questionnaires targeting clinical symptoms at baseline and 12-week follow-up were also administered.


    Procedure

    Nine clinical psychologists participated as therapists in the study. The clinicians were asked to recruit patients from the clinic in accordance with the standard procedures at the clinic. In line with the ethics committee agreement (ID: 2013/452-31), all patients were informed about the objectives of the study via a document printed on paper and asked to provide written informed consent before inclusion. All patients answered questionnaires regarding clinical and demographic characteristics via the Internet. After inclusion, the clinician registered the patient in the support system and distributed an online follow-up survey after 12 weeks access of the support system. Mean time between assessments was 91 days (range 61-116).

    The Support System

    The Internet-based support system used in this study was previously developed and tested in a pilot study conducted in a university clinic setting [8]. Also, the support system has been used in audiologic practice in supporting first-time hearing aid clients [9]. In brief, the support system was accessible via personal computers through an encrypted secured socket layer connection to the Internet. Users were assigned personal login identifications via email. Also, to increase security, an additional temporary password was sent via mobile phone text messages at each attempt to log on.

    The support system facilitated a variety of functions and the therapists decided themselves on how to use the content, tailored to the patients’ needs, and components included communication between sessions with the ability to send mobile phone text messages. Via the support system, the therapist also had the opportunity to send mobile phone text messages to the patients. The support system included a library that mainly provided text documents, but also other media such as audio and movies were made accessible. These resources were compiled primarily from prior studies on Internet-delivered CBT for anxiety and depression [10], and they were not presented as separate treatments but rather as part of the face-to-face treatment (eg, as online handouts). Topics covered in the online handouts contained supplemental information on CBT, such as behavioral activation, activity scheduling, exposure therapy, common cognitive biases, and maintenance of avoidance via safety behaviors. We also provided some audio files, such as relaxation instructions. In addition, the support system included common questionnaires and forms used in homework assignments, such as guides to create a fear hierarchy, daily thought records, and sleep diaries. For an overview of all the functions, see Table 1.

    Table 1. List of included components in the Internet-based support system.

    		Editable bya

    		PatientClinician
    Formulate homework assignmentsNoYes
    Library (sharing content)NoYes
    Registration formsYesYes
    Sending internal messagesYesYes
    Sending mobile phone text messagesNoYes
    Setting an agendaNoYes
    Setting goalsYesYes
    Uploading files to the patient’s personal libraryYesYes
    Uploading new files to the libraryNoYes
    Writing memosYesYes

    aThe term editable means that the user was able to edit, add, or delete content within that specific component/function in the support system.

    Technical Issues

    During the study period, we had one main technical problem with the support system. As a way of warranting the security of the support system, it was designed to automatically log out inactive users (as determined by no clicks with the pointer). First, the support system automatically disconnected users after 10 minutes of inactivity. A number of users gave us feedback that text had been lost due to this function (eg, while writing a long message exceeding 10 minutes, the user was incorrectly disconnected). Consequently, we increased this time frame to 40 minutes during the study period.

    Before study initiation, we invited a group of clinicians to a 2-hour workshop offering a brief overview of the support system. Also, the clinicians logged in to the system and were instructed to complete five tasks in order to acquire some knowledge on basic functions in the support system, for example, log in to the system, create a new user (patient), send the patient a message, share a file from the library with the patient, as well as a registration form for behavioral experiments.

    Participants and Recruitment

    The included clinicians’ professional status and demographic characteristics are presented in Table 2. The clinicians volunteered and did not receive any compensation for their participation.

    During the study period, 52 patients were registered in the support system. However, data from 4 patients were missing at the baseline assessment, 12 patients were missing at follow-up, and for 7 patients assessment data were completely missing (ie, both at baseline and follow-up). In total, 29 patients contributed with complete data from the pre- and posttreatment assessments. The patients’ demographic characteristics and computer experience at baseline are presented in Table 3. Participants self-rated their level of experience of using computers on a 5-point Likert scale, ranging from 0 (very limited) to 4 (very much). We did not include any clinical interview in order to determine diagnostic criterion and comorbidity. The patients received treatment but no compensation for participating in the study.

    Table 2. Demographic and professional characteristics of the clinicians (n=9).
    Characteristics
    Age in years, mean (median, SD, range)37.78 (35, 9.2, 28-54)
    Year as licensed psychologist, mean (median, SD, range)2.44 (2, 2.5, 0-7)
    Sex, female, n (%)3 (33.3)
    Professional status, n (%)

    		Pre-licensed under supervision4 (44.4)

    		Licensed clinical psychologist5 (55,6)
    Clinical work (% per month), n (%)

    		0-25%3 (33.3)

    		26-50%1 (11.1)

    		76-100%5 (55.6)
    Table 3. Demographic and clinical characteristics of the patients (n=45).
    Characteristics
    Age in years, mean (median, SD, range)30.58 (28,10.6, 18-60)
    Work time (% per month), mean (median, SD, range)36.11 (40, 30.7, 0-100)
    Sex, female, n (%)36 (80.0)
    Having children, n (%)29 (64.4)
    Computer experience, n (%)

    		Less11 (24.4)

    		More34 (75.6)
    Educational status, n (%)

    		<High school22 (48.9)

    		>High school23 (51.1)

    The patients were either recruited from an existing wait-list at the clinic or were currently undergoing a conventional CBT at the clinic. In order to receive treatment at the psychiatric clinic, the patients had to be over 18 years of age. Eligible patients in this study were required to have some computer experience (ie, being able to handle their bank account via the Internet) and have access to a computer and mobile phone during the study period. Patients not considered eligible, or denied participation in the study, were offered conventional face-to-face CBT in line with routines at the clinic.

    All procedures contributing to this work comply with the standards of the national ethical committee and with the Helsinki Declaration of 2008.

    Cognitive Behavioral Therapy

    This study did not follow a manualized CBT protocol, nor did all the clinicians receive clinical supervision as part of the study. The clinicians tailored the CBT according to their patient’s needs (eg, based on cognitive case formulation or behavior analysis) and each clinician-patient pair individually decided how to use the Internet-based support system during the treatment.

    We evaluated the use of the support system during a period of 12 weeks. Therefore, our assessments at baseline and 12-week follow-up were not fixed at pre- and posttreatment (ie, at baseline, some patients had already started CBT, and for some patients the CBT was not terminated at the 12-week follow-up).

    Support System Usability

    For all the users (ie, clinicians and patients), we monitored the number of logins, the total time spent logged in, as well as the number of messages sent within the support system. After 12 weeks of accessing the support system, we evaluated the users’ experiences. We also asked questions targeting specific functions within the support system, for example, how often the participant read and downloaded text documents, listened to audio files from the library, set goals for the treatment, asked questions, and requested guidance via internal messages. The questions were rated on a 6-point scale ranging from never to very often. In addition, the clinicians were also asked to rate for how many of their patients the features in the support system had been, or would have been, relevant for their patients in their regular clinical practice, ranging from no one, less than 50%, more than 50%, or for most patients.

    Technology Acceptance, Perceived Usefulness, and Ease of Use

    We used 19 questions targeting usability of the Internet-based support system. The questions were adopted from questionnaires of technology acceptance [11], perceived usefulness, and perceived ease of use [12] and were translated into Swedish. We used only a sample of questions and customized them to fit the current study. All questions were rated on a 7-point Likert scale ranging from “Strongly disagree” to “Strongly agree.” All participants were asked to answer these questions (ie, both clinicians and patients).

    Clinical Outcome and Quality of Life

    The Beck Anxiety Inventory (BAI) [13] and the Generalized Anxiety Disorder Screener-7 (GAD-7) [14] were used both at baseline and as outcome measures of anxiety symptoms. Both questionnaires have been shown to have excellent internal consistency (Cronbach alpha >.90) [13,14]. The Montgomery Åsberg Depression Rating Scale‒self-rating version (MADRS-S) [15] and the Patient Health Questionnaire-9 (PHQ-9) [16] were used to measure symptoms of depression and suicidality. MADRS-S and PHQ-9 also have excellent internal consistency (alpha >.89) [15,16]. In the MADRS-S, suicidality was defined as a score of at least three points on item 9. Similarly, patients scoring one point (or above) on item 9 on the PHQ-9 were also considered suicidal in this study.

    In addition to change in symptoms of anxiety and depression, the Quality of Life Inventory (QOLI) [17] was administered both at baseline and at 12-week follow-up. QOLI has shown good to excellent internal consistency (alpha >.77) in a clinical population with both anxious and depressive disorders [18]. In agreement with our previous studies [3], all self-report questionnaires were administered via a secured Internet-based platform.

    Data Analysis

    The STATA v13.1 statistical software for Mac OS X (StataCorp) was used to analyze the data. We evaluated user experiences across patients with high versus low activity in the support system and dichotomized high versus low frequent users by performing a median split on number of times the patients accessed the support system (ie, ≥12 defined high users). Differences between users (ie, low versus high activity) groups (ie, clinicians versus patients) were analyzed using logistic regression.

    We also performed analyses on clinical outcome of anxious and depressive symptoms. Similarly, quality of life was measured at baseline and 12-week follow-up. In order to account for dependency in the data (ie, longitudinal clinical outcomes), we used generalized estimating equations (GEE) with an exchangeable correlation structure, assuming that all missing data were completely at random [19]. Outcomes are presented as coefficients or odds ratios (OR). Within-group effect sizes were calculated based on the pooled standard deviation and correlation between time points, expressed as Cohen d with 95% confidence intervals. Furthermore, we also investigated if the number of times accessing the support system was associated with change in the patient’s symptoms of anxiety and depression.

    As a way to control for multiple comparisons, we performed Bonferroni corrections within each sector of the analyses (ie, one sector corresponds to support system usability, and another was clinical outcome).

    Furthermore, we explored what time of the day the patients accessed the support system. Specifically, we were interested in the proportion of logins made after the clinic was closed (ie, before 8 a.m. and after 5 p.m.).


    Support System Usability

    Clinicians

    The mean number of times the clinicians accessed the support system during the 12-week period was 94 (SD 54, median 89), and across all the clinicians the average time logged in to the support system was 1008 minutes (16.8 h, SD 784 min, median 770 min). On average, 64 messages were sent per clinician (SD 25, median 62, range 17-100). Moreover, the mean number of sent mobile phone text messages was 32 (SD 14, median 35, range 9-51).

    As shown in Table 4, the clinicians’ ratings of usability demonstrate how often specific components were assigned to the patient, as well as the proportion of patients for whom this component was considered relevant in the therapeutic work. For example, sharing forms and studying information in the library for own professional development were on average used 2.8 times (ie, less used than “sometimes”). Yet, most of the clinicians rated these functions to be relevant for more than 50% of their patients.

    Table 4. Clinicians’ (n=9) evaluation of support system usability on a 6-point Likert scale (0=never and 5=very often), sorted by mean values.
    Questions
    		The component was relevant for n patients, n (%)
    MeanSDLess than 50%More than 50%
    Sending reminders via mobile phone text messages4.111.03 (33.3)6 (66.7)
    Shared documents, images, and audio files via the library4.001.03 (33.3)6 (66.7)
    Answered questions3.891.04 (44.4)5 (55.6)
    Providing support and encouragement3.890.94 (44.4)5 (55.6)
    Formulated homework assignments3.781.15 (55.6)4 (44.4)
    Provided psychoeducation from the library3.441.94 (44.4)5 (55.6)
    Reading the patients reports on homework assignments3.440.75 (55.6)4 (44.4)
    Asked for feedback on information in the library3.221.27 (78.0)2 (22.2)
    Examined the patient work with homework3.110.95 (55.6)4 (44.4)
    Studied information from the library for own professional development2.881.34 (44.4)5 (55.6)
    Distributed registration forms2.781.54 (44.4)5 (55.6)
    Reviewed homework assignments reported by the patient2.781.37 (78.0)2 (22.2)
    Worked with assignments from the library2.441.28 (89.0)1 (11.1)
    Corrected and revised homework assignments2.441.46 (66.7)3 (33.3)
    Formulated goals for therapy1.881.27 (78.0)2 (22.2)
    Setting an agenda1.441.37 (78.0)2 (22.2)
    Play audio during the therapy session0.110.39 (100)0 (0)
    Table 5. The patients’ (n=33) evaluation of support system usability on a 6-point Likert scale (0=never and 5=very often), sorted by mean values.
    QuestionsMeanSDMedian
    Proportion of completed homework assignments3.271.34
    Accessed your psychologists formulations of homework assignments3.181.64
    Provided information about the progress of your homework assignmentsa2.331.73
    Asked for guidance via internal messagesa2.181.83
    Answered forms2.151.73
    Downloaded and saved information on your computer1.911.72
    Reading information from the library1.881.62
    Reading and reviewed treatment goals during the therapy1.761.71
    Printed documents1.421.71
    Reading the agenda1.211.60
    Saved your own therapy-related information (text and/or images) in your personal library1.031.40
    Listened to audio files0.901.40
    Wrote notes regarding questions to discuss with your psychologist0.330.80
    Wrote memos0.300.80

    aIndicating differences between the high versus low frequent users.

    Patients

    Across 12 weeks of access, the patients’ average number of logins was 14 (SD 15.3, median 11, range 1-95), and they (n=49) spent on average 92 minutes (SD 157, median 42) on the support system. One patient was an outlier and spent more than 1000 minutes logged into the support system. After excluding this outlier, the average number of minutes was reduced to 72 (SD 72, median 40), which corresponds to an average of 6 minutes of access per week (72/12) and patient. In addition, the patients sent on average 6 messages to their therapist (SD 10, median 3, range 0-58), although there is a large variation across users.

    The patient’s usability ratings of specific components in the support system are presented in Table 5. High and low frequent users ratings differed significantly on two items: (1) providing information about the progress homework assignments (high users mean 3.1, SD 1; low users mean 1.6, SD 2; β=0.64, Z=2.40, P=.02), and (2) asked for guidance via internal messages (high users mean 3.1, SD 1; low users mean 1.2, SD 2; β=0.70, Z=2.71, P=.002). However, after controlling for multiple comparisons (ie, Bonferroni correction) the differences were not significant.

    Technology Acceptance, Perceived Usefulness, and Ease of Use

    The clinician and the patient ratings of technology acceptance, perceived usefulness, and ease of use are shown in Table 6. The clinicians and the patients rated two items significantly differently. First, the clinicians were more motivated to use the support system after the study termination (β=0.68, Z=2.10, P=.036). Second, the patients, relative to the clinicians, highlighted that the support system reminded them about tasks to complete in the support system (β=–0.50, Z=2.37, P=.018). However, by controlling for multiple comparisons, these differences were not significant.

    Clinical Outcome and Quality of Life

    Total scores on the BAI, MADRS-S, and PHQ-9 decreased from baseline to 12-week follow-up, yet the GAD-7 only showed a trend towards statistical significance. Moreover, quality of life, as measured by QOLI, increased over time (see Table 7).

    Suicidal ideations, as measured by MADRS-S item 9, decreased by 14% from baseline to follow-up (OR 0.86, Z= 2.05, P=.040). However, the scored item on suicidal ideation in PHQ-9 did not change over time (OR 0.89, Z=1.43, P=.152). With the exception of change on MADRS-S suicidality and QOLI, the other results on clinical symptoms remained statistically significant following Bonferroni correction (P<.05).

    We did not find that the number of times accessing the support system was associated with any change in clinical symptoms of anxiety or depression. We found that 30.52% (420/1376) of the patients’ logins were made after working hours at the clinic.

    Table 6. Questionnaire targeting technology acceptance and ease of use of the support system. Ratings provided on a 7-point Likert scale (1=strongly disagree to 7=strongly agree).
    QuestionsPatients (n=33)Clinicians (n=9)
    MeanSDMeanSD
    1. Using the platform improves the quality of the work I do4.091.95.221.0
    2. The platform enables me to accomplish tasks more quickly3.611.93.561.6
    3. The platform increases my productivity in my work with the therapy3.911.94.001.4
    4. The platform improves my work and the things I need to do in therapy, such as homework assignments and practical exercises4.582.04.561.0
    5. Using the platform can increase my effectiveness working with the therapy4.091.84.440.9
    6. Using the platform makes it easier to do my tasks4.702.04.111.6
    7. Learning to operate the platform is easy for me5.671.95.001.4
    8. Performing an operation in the platform always leads to a predicted result4.121.83.221.5
    9. The platform has a clear interface that helps me do what I want4.521.93.221.1
    10. The platform is flexible and easy to interact with4.331.83.111.8
    11. It is easy for me to become skillful at using the platform4.421.94.891.0
    12. Overall, I find the platform easy to use5.301.94.671.6
    13. I feel confident in finding information in the platform5.121.84.221.3
    14. I feel confident in receiving and sending messages5.241.85.331.6
    15. I feel confident in downloading files5.122.14.781.8
    16. The platform was visually appealing3.581.93.781.8
    17. The platform reminds me about tasks to complete4.792.12.671.9
    18. The organization of information in the platform is clear4.451.93.442.0
    19. I would like to use the platform on a regular basis in the future4.362.36.440.9
    Table 7. Generalized estimating equations (GEE) regarding clinical symptoms and quality of life at baseline, and 12-week follow-up (total N=33 patients in the GEE; 29 contributed with complete data from baseline to follow-up).
    MeasurePre, mean (SE)Post, mean (SE)CoefficientZPCohen da (95% CI)
    BAI20.54 (1.7)15.82 (1.8)-4.72-2.88.0040.45 (0.1 to 0.8)
    GAD-710.35 (0.8)9.15 (0.9)-1.20-1.77.0770.20 (-0.1 to 0.5)
    MADRS-S20.96 (1.3)18.18 (1.4)-2.78-2.61.0090.32 (0.0 to 0.6)
    PHQ-913.32 (0.9)9.19 (1.0)-4.13-3.68<.0010.69 (0.2 to 1.1)
    QOLI-0.37 (0.3)0.03 (0.3)0.402.12.034-0.18 (-0.4 to 0.1)

    aEffect sizes (Cohen d) were calculated on observed data.


    Principal Findings

    The aim of this feasibility study was to test an Internet-based support system in a clinical psychiatric setting with a focus on both clinician and patient experiences, and also including patient outcomes. Overall, we found that clinicians, as compared to the patients, rated some functions of the support system as more useful and that ratings by patients tended to be fairly low for some functions. We also asked clinicians to rate the proportion of patients for which the components of the support system would be useful. Less than half of the clinicians rated that the components would be useful for more than half of their patients. As there were few clinicians in the study, these estimates should be interpreted with caution but at least they signal that some functions, like sending reminders and sharing documents, may be appealing to clinicians in their work. At the same time, using the support system to formulate therapy goals, agenda setting, and playing audio files were barely used by the clinicians. Overall, usefulness ratings, ease of use, and technology acceptance varied but were fairly high for some items. Moreover, 30% of the times the patients accessed the support system were after working hours at the clinic. This indicated that this support system also has the potential to increase the availability of psychiatric care. In line with a large body of literature on the effects of CBT and ICBT, symptom ratings decreased over the study period.

    This feasibility study raises many questions. First of all, the support system tested in our first study [8] appears to work when delivered in a more regular psychiatric setting with regular clinicians and patients [20]. Yet, it is important to keep in mind that a few specific functions within the system were rarely used by the clinicians. We hesitate to refer to this study as an effectiveness study as use of the support system per se was not part of regular practice, and we introduced and tested the system simultaneously. In the first study [8], we had a smaller sample and used interviews to gather information on experiences of clinicians and patients. In this study, we investigated differences between high versus low activity users and differences between the clinicians and the patients. No difference turned out to be statistically significant after controlling for multiple comparisons but indicated that the high frequent users more often sent messages to their therapist.

    A second aspect to discuss relates to attitudes towards technology use and preferences (eg, [21,22]). There is a growing literature on these topics relating to ICBT, but far less work on the use of technology within face-to-face CBT, sometimes referred to as blended treatments [23], has been conducted. In addition, a recent stakeholder survey indicated that blended treatments are rated as more acceptable than ICBT with less therapist contact [24]. We expect more studies to appear in the field of blended interventions [23]. One recent example was a study on depression in which a mobile phone app was used [25]. Moreover, in this study we did not really focus on the technical aspects of the system, and there has been increased interest in the use of novel technologies and how they can be best incorporated and correctly described in digital health interventions [26].

    Third, what can we expect to achieve with the support system? There is emerging literature on knowledge acquisition in CBT [27], and we believe that the support system can serve as a facilitator for patients when they learn more about themselves and the treatment presented. This might not necessarily lead to better outcomes in the short run but is also unlikely to lead to worse outcomes. In the long run, it is possible that the enhanced learning and support provided by the system could help to prevent relapse.

    Fourth, this study raises questions regarding training of therapists and adherence to treatment manuals. It is possible that clinicians with less training can benefit more from blending information technology with face-to-face services. There are examples of using computerized support [7] with good outcomes, but to the best of our knowledge, there is a lack of controlled trials testing if clinicians with less training can perform as well as more skilled clinicians if they work with a support system. More experienced and well-trained clinicians may also be more effective if tasks can be delegated to the computer (eg, handling outcome measures).

    Limitations

    It is important to keep in mind that this study is limited by a number of factors. First, the within-group design limits any causal inferences, and we cannot answer whether or not the support system made any substantial, positive or negative, contributions beyond the effect of conventional CBT. However, in this feasibility study (without a predefined feasibility criteria) we focused on usability and technology acceptance at an outpatient psychiatric clinic. Second, the clinicians decided whether or not to ask a specific patient about participating in the study (ie, possible self-selection bias). Consequently, it is possible that the outcome of the study is affected by confounding by indication. Third, in terms of CBT interventions, we cannot demonstrate the specific interventions the clinicians delivered. Moreover, we did not measure the therapist’s competence in delivering CBT. Nevertheless, by the use of the current support system we were able to monitor the use of some fundamental CBT components (eg, that the clinicians provided homework assignments). Fourth, the number of patients lost to follow-up may be an important sign of dissatisfaction. Nevertheless, it is plausible that this was related to issues regarding procedure of the study (eg, the clinicians were primarily responsible for initiating the follow-up assessments). In our previous study in a university setting, we had no missing data.

    Conclusions

    In spite of the limitations, this study adds to the literature showing that modern information technology can be aligned with conventional face-to-face services. Future studies should investigate the added value of using a support system in psychiatric care. Another option is to evaluate the usability of the support system when training new therapists.

    Acknowledgments

    We thank Cecilia Olsson Lynch, Hugo Klintmalm, Mikael Salomonsson, Nahla Elkholy, Pelle Alexandersson, Ragnar Nordqvist, Rikard Sunnhed, Susanne Westh, and Thomas Hesslow who volunteered as clinical psychologists. Anders Berntsson and Dr Anna Åberg Wistedt made significant contributions initiating the current research project. Also, many thanks to Dr Sarah-Marie Vigerland for contributions with proofreading. We gratefully thank all the patients participating in the study.

    PRIMA Psychiatry Research Foundation funded this research (KM). GA is funded by a professor grant at Linköping University.

    Conflicts of Interest

    None declared.

    1. Andersson G. Using the Internet to provide cognitive behaviour therapy. Behav Res Ther 2009 Mar;47(3):175-180. [CrossRef] [Medline]
    2. Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry 2014 Oct;13(3):288-295 [FREE Full text] [CrossRef] [Medline]
    3. Andersson G. The Internet and CBT: a clinical guide. Boca Raton: CRC Press; 2014:1-144.
    4. Andersson G, Hedman E. Effectiveness of Guided Internet-Based Cognitive Behavior Therapy in Regular Clinical Settings. Verhaltenstherapie 2013;23(3):140-148. [CrossRef]
    5. Waller G. Evidence-based treatment and therapist drift. Behav Res Ther 2009 Feb;47(2):119-127. [CrossRef] [Medline]
    6. Tobin DL, Banker JD, Weisberg L, Bowers W. I know what you did last summer (and it was not CBT): a factor analytic model of international psychotherapeutic practice in the eating disorders. Int J Eat Disord 2007 Dec;40(8):754-757. [CrossRef] [Medline]
    7. Roy-Byrne P, Craske MG, Sullivan G, Rose RD, Edlund MJ, Lang AJ, et al. Delivery of evidence-based treatment for multiple anxiety disorders in primary care: a randomized controlled trial. JAMA 2010 May 19;303(19):1921-1928 [FREE Full text] [CrossRef] [Medline]
    8. Månsson KNT, Skagius RE, Gervind E, Dahlin M, Andersson G. Development and initial evaluation of an Internet-based support system for face-to-face cognitive behavior therapy: a proof of concept study. J Med Internet Res 2013 Dec 10;15(12):e280 [FREE Full text] [CrossRef] [Medline]
    9. Brännström KJ, Öberg M, Ingo E, Månsson KNT, Andersson G, Lunner T, et al. The Process of Developing an Internet-Based Support System for Audiologists and First-Time Hearing Aid Clients. Am J Audiol 2015 Sep;24(3):320-324. [CrossRef] [Medline]
    10. Andersson G, Carlbring P, Ljótsson B, Hedman E. Guided Internet-Based CBT for Common Mental Disorders. J Contemp Psychother 2013 May 21;43(4):223-233. [CrossRef]
    11. Roca JC, Chiu C, Martínez FJ. Understanding e-learning continuance intention: An extension of the Technology Acceptance Model. International Journal of Human-Computer Studies 2006 Aug;64(8):683-696. [CrossRef]
    12. Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 1989 Sep;13(3):319. [CrossRef]
    13. Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol 1988 Dec;56(6):893-897. [Medline]
    14. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
    15. Svanborg P, Asberg M. A new self-rating scale for depression and anxiety states based on the Comprehensive Psychopathological Rating Scale. Acta Psychiatr Scand 1994 Jan;89(1):21-28. [Medline]
    16. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [Medline]
    17. Frisch MB, Cornell J, Villanueva M, Retzlaff PJ. Clinical validation of the Quality of Life Inventory. A measure of life satisfaction for use in treatment planning and outcome assessment. Psychological Assessment 1992;4(1):92-101. [CrossRef]
    18. Lindner P, Andersson G, Ost L, Carlbring P. Validation of the internet-administered Quality of Life Inventory (QOLI) in different psychiatric conditions. Cogn Behav Ther 2013;42(4):315-327. [CrossRef] [Medline]
    19. Little R, Rubin D. Statistical analysis with missing data. Hoboken, NJ: Wiley; 2002.
    20. Shadish WR, Matt GE, Navarro AM, Phillips G. The effects of psychological therapies under clinically representative conditions: a meta-analysis. Psychol Bull 2000 Jul;126(4):512-529. [Medline]
    21. Gun SY, Titov N, Andrews G. Acceptability of Internet treatment of anxiety and depression. Australas Psychiatry 2011 Jun;19(3):259-264. [CrossRef] [Medline]
    22. Mohr DC, Siddique J, Ho J, Duffecy J, Jin L, Fokuo JK. Interest in behavioral and psychological treatments delivered face-to-face, by telephone, and by internet. Ann Behav Med 2010 Aug;40(1):89-98 [FREE Full text] [CrossRef] [Medline]
    23. van DVR, Witting M, Riper H, Kooistra L, Bohlmeijer ET, van GLJ. Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry 2014 Dec 14;14:355 [FREE Full text] [CrossRef] [Medline]
    24. Topooco N, Riper H, Araya R, Berking M, Brunn M, Chevreul K, et al. Attitudes towards digital treatment for depression: A European stakeholder survey. Internet Interventions 2017 Jun;8:1-9. [CrossRef]
    25. Ly KH, Topooco N, Cederlund H, Wallin A, Bergström J, Molander O, et al. Smartphone-Supported versus Full Behavioural Activation for Depression: A Randomised Controlled Trial. PLoS One 2015;10(5):e0126559 [FREE Full text] [CrossRef] [Medline]
    26. Murray E, Hekler E, Andersson G, Collins L, Doherty A, Hollis C, et al. Evaluating Digital Health Interventions. American Journal of Preventive Medicine 2016 Nov;51(5):843-851. [CrossRef]
    27. Harvey AG, Lee J, Williams J, Hollon SD, Walker MP, Thompson MA, et al. Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspect Psychol Sci 2014 Mar;9(2):161-179. [CrossRef] [Medline]

    ‎
    BAI: Beck Anxiety Inventory
    CBT: cognitive behavioral therapy
    GAD-7: Generalized Anxiety Disorder Screener‒7 items
    GEE: generalized estimation equations
    ICBT: Internet-delivered cognitive behavioral therapy
    MADRS-S: Montgomery-Åsberg Depression Rating Scale‒self-rating version
    OR: odds ratio
    PHQ-9: Patient Health Questionnaire 9 items
    QOLI: Quality of Life Inventory

    Edited by G Eysenbach; submitted 27.05.16; peer-reviewed by J Apolinário-Hagen, J Ruwaard; comments to author 22.09.16; revised version received 12.06.17; accepted 12.07.17; published 24.08.17

    Copyright

    ©Kristoffer NT Månsson, Hugo Klintmalm, Ragnar Nordqvist, Gerhard Andersson. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 24.08.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.