This paper is in the following e-collection/theme issue:

Usage Data and Process Outcomes in eHealth Trials (56)

Published on in Vol 7, No 5 (2018): May

The Development of Delta: Using Agile to Develop a Decision Aid for Pediatric Oncology Clinical Trial Enrollment

The Development of Delta: Using Agile to Develop a Decision Aid for Pediatric Oncology Clinical Trial Enrollment

The Development of Delta: Using Agile to Develop a Decision Aid for Pediatric Oncology Clinical Trial Enrollment

Authors of this article:

Eden G Robertson1, 2 Author Orcid Image ;   Claire E Wakefield1, 2 Author Orcid Image ;   Richard J Cohn1, 2, 3 Author Orcid Image ;   Tracey O'Brien2, 3 Author Orcid Image ;   David S Ziegler2, 3 Author Orcid Image ;   Joanna E Fardell1, 2 Author Orcid Image
Article Authors Cited by (12) Tweetations (60) Metrics

    Original Paper

    1Behavioural Sciences Unit, Kids Cancer Centre, Sydney Children's Hospital, Randwick, Australia

    2Discipline of Paediatrics, School of Women’s and Children’s Health, University of New South Wales, Kensington, Australia

    3Kids Cancer Centre, Sydney Children's Hospital, Randwick, Australia

    Corresponding Author:

    Eden G Robertson, BPsych (Hons)

    Behavioural Sciences Unit

    Kids Cancer Centre

    Sydney Children's Hospital

    Level 1 South

    High Street

    Randwick, 2031

    Australia

    Phone: 61 0293859868

    Email: eden.robertson@unsw.edu.au


    Background: The internet is increasingly being used to disseminate health information. Given the complexity of pediatric oncology clinical trials, we developed Delta, a Web-based decision aid to support families deciding whether or not to enroll their child with cancer in a clinical trial.

    Objective: This paper details the Agile development process of Delta and user testing results of Delta.

    Methods: Development was iterative and involved 5 main stages: a requirements analysis, planning, design, development, and user testing. For user testing, we conducted 13 eye-tracking analyses and think-aloud interviews with health care professionals (n=6) and parents (n=7).

    Results: Results suggested that there was minimal rereading of content and a high level of engagement in content. However, there were some navigational problems. Participants reported high acceptability (12/13) and high usability of the website (8/13).

    Conclusions: Delta demonstrates the utility for the use of Agile in the development of a Web-based decision aid for health purposes. Our study provides a clear step-by-step guide to develop a Web-based psychosocial tool within the health setting.

    JMIR Res Protoc 2018;7(5):e119

    doi:10.2196/resprot.9258

    Keywords

    decision support techniques; decision making; neoplasms; patients; child; patient portals 


    Decision Making Regarding Pediatric Oncology Clinical Trials

    Many parents are faced with the decision of whether to enroll their child with cancer in a clinical trial. In Australia, approximately 80% of children with cancer enroll in a clinical trial. Despite the necessity of clinical trials to test new treatments with the aim to find more effective cancer treatments, patients and parents often find the decision difficult [1,2]. Families often find the rationale, design, and long-term implications of participating in trials difficult to understand [3]. Early phase clinical trials can be particularly confusing, with parents and young people overestimating the potential benefit of the trials (called therapeutic misconception) [4,5], making it difficult for them to weigh up the benefit and burden of enrolling while hoping for a cure. Deciding whether to enroll is also complex, given the time pressures to proceed to therapy, the large amount of information to comprehend, and the emotional timing of the decision, which is often at either diagnosis or relapse. Many families experience decisional anxiety and uncertainty and psychological distress associated with the decision whether to enroll their child in a clinical trial or not [1,6].

    With clinical trial decisions, there may not necessarily be a right or wrong choice of treatment. In these scenarios, individual values and preferences of the family become crucial. Decision aids are evidence-based tools designed to assist clients to be involved in making specific and deliberated choices among health care options [7]. Decision aids support patients and caregivers to make informed decisions by helping them to balance their values with the benefits and disadvantages of their treatment options. The “International Patient Decision Aid Standards” (IPDAS) highlight the role of clear information of treatment options based on research evidence, opportunities to clarify and express values, structured guidance in deliberation and communication of choice, and developed using a systematic development process [8]. Although decision aids vary in formats, decision aids appear to improve knowledge, quality of informed consent, and decision satisfaction for adult patients making treatment or screening decisions (eg, prostate cancer screening, menopausal hormone therapy) [9]. An evaluation of a decision aid in adult cancer clinical trials (“Cancer Research Choices”) has also shown reduced decisional conflict and postdecision regret compared with standard of care, without impacting clinical trial enrollment rates [10]. Despite the fact that some decision aids exist in adult clinical trials, none are available in pediatric oncology.

    Agile Development Process

    Agile development is an overarching term to describe a software development process. Agile focuses on collaborations between developers and stakeholders, flexible methodology, and the ability to respond quickly to change through multiple iterations [11,12]. Changes even in late development are encouraged. This approach aims to deliver a product that can quickly adapt to clients’ changing needs and also deliver a high-quality and high-value project within the constraints (ie, cost, schedule, and scope) [13].

    The use of Agile approaches has an increasing worldwide use within software development [14,15] and research [16]. Use of Agile approaches has been reported to result in higher business performance, customer satisfaction, and product quality compared with more structured nonfluid approaches [17]. Agile projects were reported to be 28% more likely to succeed (defined by a project that is completed on time and budget and includes originally outlined specifications) [18]. Multiple case studies have acknowledged the challenges of implementing Agile development, such as highly intensive interactions with stakeholders [19], however many have reported an overall positive impact [20-23].

    In recent years, there has been a rapid growth in the use of, and access to, the internet for health purposes [24,25]. In Australia, studies have reported over 28% of patients access Web-based health information [26], and in the United States, more than 43% do [27]. In Swedish adult cancer patients, approximately 77% of patients seek Web-based health information [28]. Accessing health information in oncology has been reported to improve oncologist-patient communication and connectedness, patient use of medical jargon, and scientific knowledge [29].

    Given the movement toward a more patient-centered and collaborative decision making approach in the health care system [30], there is a rising need for innovative product development that is provided online and tailored to the patient and their family. The principles of Agile are an appropriate vehicle to guide the development of a Web-based patient decision aid, especially those with clear aims yet flexibility in design [11].


    Using Agile to Develop Delta

    We developed Delta, a Web-based decision aid to support families deciding whether or not to participate in a pediatric oncology clinical trial. The main of aim of Delta was to improve clinical trial knowledge and facilitate treatment discussions and shared decision making between families and health care professionals (HCPs). Families offered a clinical trial are able to log in to the Delta website, which links to the clinical trial they are considering. Delta incorporates general content about clinical trials (eg, what are clinical trials, how do clinical trials work), the specific clinical trial information sheet, and a values clarification exercise. This paper provides a detailed description of the development and early testing of Delta, acting as proof of concept for the use of Agile for the development of a Web-based decision aid.

    The development of Delta was divided into the 5 main stages involved in Agile development and was both iterative (with one iteration referring to revisions made on one particular version of the website following feedback) and incremental (see Figure 1). We expected development to take 4 months from approval of the functional specification, with an additional 2 months for prepilot user testing. Due to delays in development and complexity of using Agile in research (see Discussion section), the actual development time of Delta was approximately 4 months of full-time work and 2 months of user testing, spread over 12 months. Our development was driven by guidelines for patient websites [31], the International Patient Decision Aid Standards (IPDAS) [7], and 12 Agile principles (see Multimedia Appendix 1).

    The development of Delta was supported by a steering committee (16 members) to ensure development followed IPDAS [7]. The committee included psycho-oncology researchers (n=6), pediatric oncologists (n=5), a pediatric oncology clinical trial research manager (n=1), a clinical nurse consultant (n=1), parents (n=2), and a young person with cancer (n=1). All consumers signed a Terms of Reference Agreement specifying the purpose of the committee and their role. As Agile leans toward a leadership and collaboration approach, where a team leader acts as a facilitator, we also established a lead development team (3 members) from the steering committee to provide more regular feedback (ER, CW, and JF). The lead team met with each other face-to-face fortnightly, and all steering committee members met on an as-needed basis.

    ‎
    Figure 1. Delta development process. Estimated time is based on full days of work required, which occurred over a 12-month period.
    View this figure

    Requirements Analysis

    Requirements analysis was required to identify the needs and preferences of families in regard to information and functionality of the website. We identified needs and preferences of families by conducting a basic literature search, leading a systematic review on strategies to facilitate shared decision making in pediatric oncology clinical trials [32], and through discussions with the steering committee. Our team also conducted interviews with 25 Australian parents and 5 adolescents recently diagnosed with cancer to identify the needs and preferences of families when making treatment decisions. We continued the requirements analysis throughout the whole development process via discussions with the steering committee and literature searches.

    Planning

    Planning was required to determine appropriate timelines for the completion of sprints (ie, smaller milestones to be completed and reviewed in short time frames), content and website functions, and design. We chose Web developers based on their experience, reputation from previous clients, corresponding values in development ideologies, cost of deliverables, and workable time frames. Planned sprints, between 2 and 4 weeks, were also agreed upon based on workable time frames of the lead team and Web developers. A lead team member (ER) developed the content framework and the ideal design and functionalities of the website, which was approved after 3 iterations by the other lead team members. ER met with Web developers face-to-face on 3 occasions before beginning development to discuss the website concept, functional specifications, and website requirements. Web developers then finalized the functional specification, which the lead team approved.

    Design

    The design stage involved design of the Delta logo, color scheme, font, and graphics and website wireframes (ie, a visual display of the function and framework of the website). A graphic designer with a good working relationship with the lead team assisted with these tasks. The logo and color scheme went through 2 iterations before approval by the lead team and steering committee. ER developed wireframes following the initial planning with the Web developers. Wireframes went through 3 iterations with the lead team before beginning functional website protocol development. Web developers then developed the website prototype. We asked all steering committee members to provide feedback on the website prototype. We implemented suggested modifications based on consensus.

    Development

    The development stage involved the development of content and the functional Delta website. We developed Delta to be highly scalable, both horizontally (ie, ability to add more resources, such as content) and vertically (ie, ability to increase the functionality or capacity, such as potential functionality as an app). Delta has a responsive Web design. It has been developed to be user-friendly across multiple platforms, including computers, tablets, and mobile phone, and has in place the capability to be scaled up to be available as an app. Although currently only in English, it also has the capability to be scaled out to be available in multiple languages. Delta can also be easily scaled out by allowing the addition of as many clinical trials as necessary. The method to add in additional clinical trials has been simplified to ensure Delta is easily maintainable. We also ensured Delta was developed with a content management system, which allowed content to be easily updated without the need for Web developers.

    Content Development

    Content went through approximately 8 rounds of iterations across 6 months, before being embedded within the website. We developed 2 versions of Delta (one version for parents and one for adolescents aged 12 years and older). Our systematic review [32] identified 3 key strategies to facilitate shared decision making in pediatric oncology: (1) quality information exchange, (2) clear communication, and (3) decision making support. We developed Delta to incorporate these 3 core features of shared decision making [32]. We developed Delta to cater for low health literacy populations. To facilitate information exchange, we presented information at Grade 8 readability for parent content [33] and Grade 5 readability for adolescent content. The reader is able to access less or more detailed information based on their preference for information amount. We provided an option grid for readers to easily compare key information between their options. We also acknowledged the role of gist memory (that is, recall of bottom-line meaning rather than detailed information) and therefore included summary boxes of information. To promote communication, we incorporated a question prompt list and suggested strategies for parents to communicate with HCPs and their child (if appropriate).

    To provide more decisional support, we incorporated a values clarification exercise. A values clarification exercise allows participants to clarify and communicate the personal value of options, to ultimately make a better quality decision [34] that is most personally desirable [35], and results in less decisional regret [36]. The Delta values clarification exercise allows parents to rate their reasons to enroll or not to enroll in the clinical trial on a scale of personal importance. Delta is able to show whether the parent appears to be leaning toward enrolling in the trial or not. As per many patient decision aids [37], the algorithm for the outcome of the Delta values clarification exercise is basic, providing a summation of participant responses. Although items may be weighted differently for each participant, the purpose of the decision making exercise is not to provide a correct answer or a recommendation, but rather allow participants to weigh up their treatment options. Participants are instructed that this exercise will provide some indication as to the option they are more inclined to choose, but to discuss it further with their treating team.

    Delta also provides the specific information sheet for the clinical trial in which the family was invited to participate. The website also includes a glossary, a text-to-speech function, and the ability to save and print notes from within the website. See Table 1 for the key elements of Delta.

    Web Development

    Web developers delivered progress in fortnightly or monthly sprints. The lead team provided feedback of the sprints, and Web developers made necessary modifications as they came up. For the final sprint, of the 16 steering committee members, 12 were available to provide feedback. Modifications suggested by the steering committee were discussed with the lead team and implemented based on consensus. We received multiple iterations of the website in early stages of development from steering committee members. During later stages of development, the lead team requested less feedback and thus had fewer iterations. Fewer iterations in later development were due to significant delays in development and the minimal value that later iterations provided (eg, minor wording changes). See Multimedia Appendix 2 for a video introduction to the Delta website.

    Table 1. Key elements of Delta, categorized by function and health literacy needs.
    ElementStrategy themeaHealth literacyb
    Information about clinical trials (grade 8 readability and minimal medical jargon)Information provisionFunctional
    Specific clinical trial information sheetInformation provisionFunctional
    Interactive glossaryInformation provisionFunctional
    Text-to-speech functionalityInformation provisionFunctional
    Question prompt listCommunicationCommunicative
    Personal notes pageCommunicationCommunicative
    Communication strategiesCommunicationCommunicative
    Values clarification exerciseDecision making supportCritical

    aStrategy themes were identified via our systematic review [32].

    bHealth literacy categories are based on Nutbeam’s model of health literacy [38].

    User Testing

    We conducted eye-tracking analyses to assess the usability of the Delta website. Eye-tracking analyses have frequently been used in website development [39,40]. The analysis used a computer with an embedded webcam to track each participant’s point of gaze. We conducted eye-tracking analyses to determine where participants’ gaze lingered, their length of gaze, and readability (ie, scanning, reading, or rereading of content). Data collected through eye-tracking analyses can be presented as either a gaze plot or a heat map. Gaze plots show the location, order, and time (known as the fixation duration) spent looking across certain aspects of the website. Heat maps are a visualization of the focus of visual attention across multiple participants.

    We also conducted retrospective think-aloud interviews to establish participant satisfaction and acceptability of the website. Retrospective think-aloud interviews encourage participants to verbalize aloud their thought processes on replay of their task completion and are commonly used in website development [41,42]. The literature suggests 5 to 9 participants in think-aloud interviews, and eye-tracking analyses can detect 80% to 90% of usability problems on a website [43,44].

    Individuals were eligible to participate if they were a parent of a child treated for cancer at Sydney Children’s Hospital. ER undertook user testing in a private office at the Sydney Children’s Hospital. Participants were all experienced at using computers and the internet. Participants were reimbursed with an Aus $20 gift voucher. The local ethics committee approved the user testing.

    We instructed participants to browse the website for a maximum of 40 min. We asked participants to view the website as if they had had a consultation with their clinician regarding enrolling in a clinical trial and then provided with the Web-based decision aid. We emphasized to participants that the goal was not to assess their computer skills or their knowledge of clinical trials but rather to test the usability and acceptability of the website. If participants did not view the values clarification exercise of their own accord, ER directed the participant to complete it. Following the free browsing of the website, ER asked basic questions regarding usability, functionality, and acceptability of the website. We used a retrospective think-aloud protocol. This involved participants thinking aloud their thought process alongside a visual replay of their own eye-tracking data.


    User-Testing Results

    A total of 17 families were identified as appropriate to contact by an oncologist or nurse (see Table 2 for participant demographics). Moreover, 7 parents opted into the study (all mothers). Reasons for nonparticipation included being too busy or living too far away to come to the hospital. Eye-tracking data are missing for one parent because of technical difficulties. A total of 6 HCPs in psycho-oncology (4/6 females) also participated in user testing.

    Eye gaze was detectable, on average, 93% of the time. Validity of the gaze for both the left and right eyes was 0 on average, suggesting that the tracking quality was good. Participants left-clicked 142 times on average (range 86-237 clicks). Participants took 50.1 s to log in to the website (SD 31.0; range 25-133), 14.3 min on average to read the general content of decision aid (SD 9.5; range 2.1-33.6), and 2.5 min to complete the values clarification exercise (SD 0.9; range 1.1-4; see Table 3 for an overview of data).

    Most participants completed the values clarification exercise (5/6 HCPs; 6/7 parents), with 5 participants (4 HCPs, 1 parent) accessing the exercise without being prompted. The reasons participants did not access the exercise was because they did not see the link to access the exercise (n=4 parents) or did not feel it was relevant to them (n=1 HCP). One HCP completed the exercise twice as they wanted to see if their responses changed after reading more information. The methods of accessing content were to use the side panel menu (n=3 HCPs, n=3 parents), the “next” button (n=1 HCPs, n=2 parents), a combination of both (n=2 HCPs), or the home button (n=1 parent).

    Table 2. Summary of parent participant characteristics.
    Parental roleChild age at diagnosisTime since child’s diagnosis in monthsClinical triala
    Mother1212No
    Mother712Yes
    Mother611Yes
    Mother0.554Yes
    Mother733Yes
    Mother558Yes
    Motherb136Yes
    Average (SD)7.2 (4.2)26.6 (21.9)N/Ac

    aFamilies who were enrolled in a clinical trial as part of their child’s cancer treatment.

    bEye-tracking data are missing for this participant because of technical issues.

    cN/A: not applicable.

    Table 3. Overview of time to complete tasks.
    TasksOverall (n=12)Parent (n=6)Health care professionals (n=6)t valuea (degrees of freedom)

    		Average time (SD)RangeAverage time (SD)RangeAverage time (SD)Range
    Log in time50.1 s (31.0)25-133 s64.4 s (40.8)27-133 s38.2 s (14.9)25-65 s−1.475 (9)
    Reading general content14.3 min (9.5)2.1-33.6 min13.7 min (7.3)5.3-24.0 min14.8 min (12.0)2.1-33.6 min0.181 (10)
    Completing the values clarification exercise2.5 min (0.9)1.1-4 min2.8 min (0.7)1.8-3.7 min2.1 min (1.0)1.1-4 min0.424 (10)

    aNo comparisons were significant.

    Most participants reported high acceptability (5/6 HCPs; 7/7 parents) and high usability of the website (4/6 HCPs; 4/7 parents):

    It was nice and clear. It was simple. It had a good flow, not too many buttons to make you completely overwhelmed. Your emotions would be ridiculously high if you are looking at this, so you need to make it as simple as possible, so I like that there are three main parts to it
    [Parent of a 5-year-old boy with neuroblastoma]

    Most participants (5/6 HCPs; 6/7 parents) found the content easy to read and reported that it flowed logically. Some participants read the summary boxes before reading the main content (2/6 HCPs; 2/7 parents). These participants chose to read that section first as they felt it would be the most important information. Minor word changes were suggested by all participants (eg, change “me time” to “self-care”). None of the participants used the personal notes function, however several freely reported in the think-aloud interview that this function would be useful (n=1 HCP; n=3 parents). Moreover, 7 participants used the glossary (n=3 HCPs; n=4 parents), reporting the usefulness of the function. No participants used the speech-to-text function.

    There were few instances of rereading of content, with eye-tracking data suggesting that participants read lines to completion (see Figure 2). Most participants (5/6 HCPs; 6/6 parents; n=1 missing parent) skipped at least one page of content. Participants skipped content as they either did not want to read that information (1/5 HCPs; 1/6 parents) or did not realize they had skipped it (4/5 HCPs; 5/6 parents). Participants suggested that reducing overlap of images and phrases would improve usability. There were some particularly long fixations throughout the website, which is indicative of cognitive processing. Examples of long fixations include the terms “antitumor activity” and “dose escalation” (see Figure 3).

    Most participants reported at least some difficulty in navigating the website (3/6 parents; 4/7 parents). Problems included finding the home page and accessing more content. Eye-tracking data suggested that once participants were more familiar with the website, they were easily able to navigate to the next page of content (see Figure 4). In think-aloud interviews, many participants felt that they would have found navigation easier if font and icons were of a larger size and darker color (2/6 HCPs; 6/7 parents). Participants found the values clarification exercise useful, and the participants’ suggested leaning reflected how they felt they were leaning with regards to whether or not to enroll their child in the clinical trial (4/5 HCPs; 6/7 parents):

    I think the tool is a helpful way to make a decision. It shows you from the way you’ve answered the questions that you are tending, even if you feel undecided. It gives you options to further think about “Why am I tending that way?,” “What don’t I actually like about the trial?,” and “What more information do I need?"
    [Mother of a 10-year-old boy with neuroblastoma]
    After this tool, we would really start talking with each other. The exercise is great—it becomes a tool at that point for more people to really engage.
    [Mother of a 13-year-old boy with osteosarcoma]

    Eye-tracking analyses showed long fixations for the first few values clarification exercise items and scale. Less time was spent on the last two items of the exercise, suggesting cognitive fatigue. In think-aloud interviews, participants reported feeling unsure of how to use the values clarification exercise (2/5 HCPs; 2/7 parents). Only 5 participants (all HCPs) accessed instructions on how to complete the exercise, with minimal gaze focusing on the instructions panel (see Figure 5). However, most participants (4/6 HCPs; 6/7 parents) found that the exercise was useful and accurate in the direction they felt they were leaning:

    I was really impressed—I wasn’t expecting that. It was great. The outcome of enrolling in the clinical trial was how I was feeling. It’s still interesting to see that sentence come up and say that that though
    [Mother of a 9-year-old girl with acute lymphoblastic leukemia]

    Participants felt that the user experience of the values clarification exercise could have been improved by providing a clearer introduction and purpose of the exercise (2/6 HCPs; 4/7 parents) and providing more visible instructions (1/6 HCPs; 7/7 parents). See Table 4 for an overview of user testing findings.

    Implementation of User Testing Findings

    User testing revealed that both HCPs and parents found Delta to be acceptable and useful. On the basis of the results of the user testing, the lead team reviewed the changes suggested and implemented changes based on consensus. Main changes included replacement of the “home” icon and darker and larger font. See Table 5 for summary of main modifications made. Although some minor issues were raised, overall findings were positive.

    ‎
    Figure 2. Gaze plot of one participant. Circles represent the gaze or fixation of the participant, with larger circles indicating longer fixations. Circles are numbered based on the order of fixation. The line between each fixation point represents the saccade or rapid movement between each fixation. This participant is reading lines from start to finish and appears to understand content as there are few long fixations. Only one participant was able to be used in this gaze plot as Delta is a scrolling website, and other participants did not stop their webpage in the exact same location.
    View this figure
    ‎
    Figure 3. Heat map of one participant. Color represents the length of gaze, with deep red indicating longer fixations. Longer fixation on “antitumor activity” suggests higher levels of cognitive loading and potentially reduced comprehension.
    View this figure
    ‎
    Figure 4. Gaze plot of 2 participants. Different colored circles represent each participant. Circles represent the gaze or fixation of the participant, with larger circles indicating longer fixations. Circles are numbered based on the order of fixation. The line between each fixation point represents the saccade or rapid movement between each fixation. Both participants appeared to be able to easily see the “next” button to navigate to more content. Only 2 participants were able to be used in this gaze plot as Delta is a scrolling website, and other participants did not stop their webpage in the exact same location.
    View this figure
    ‎
    Figure 5. Gaze plot of 8 participants. Different colored circles represent each participant. Circles represent the gaze or fixation of the participant, with larger circles indicating longer fixations. Only 3 participants gazed at the instructions panel on the far left, for a minimal fixation length. Only 8 participants were able to be used in this gaze plot as Delta is a scrolling website, and other participants did not stop their web page in the exact same location.
    View this figure
    Table 4. Number of health care professionals (HCPs) and parent who used each Delta function.
    Delta functionsOverall (N=13)HCPs (N=6)Parent (N=7)
    Values clarification exercise



    		Completed values clarification exercise1257

    		Completed task without being prompted541

    		Found exercise was accurate and useful1046

    		Viewed instructions for exercise550
    Accessing content



    		Side panel633

    		Next button312

    		Combination of side panel and next button220

    		Home page101
    Acceptability and usability



    		Reported high acceptability1257

    		Reported high usability844

    		At least some minor navigational issues734

    		Suggestion for larger and darker font826
    Content



    		Content easy to read1156

    		Minor word changes suggested1367

    		Read summary boxes first422

    		Used notes function000

    		Used glossary function734

    		Used text-to-speech function000

    		Skipped at least one pagea1156

    aData missing from 1 parent.

    Table 5. Main modifications based on eye-tracking analyses and think-aloud interviews.
    AreaaModification
    ContentMinor wording changes made throughout website
    ContentReduction of overlap of content throughout
    ContentClear introduction of purpose of values clarification exercise
    DesignHome page more obvious to navigate
    DesignDarker color and large-sized font
    DesignValues clarification exercise instructions more obvious

    aModifications were related to either content or design.

    Some of the features within the website were not used (eg, notes page) or were used by approximately half of the sample. We expected this level of use, given that approximately half of the features and functions are used in a typical website [13].

    User testing findings should be considered in light of several limitations. We were only able to recruit mothers for this study. However, this may be appropriate feedback, given that mothers tend to take on a greater decisional role than fathers in pediatric oncology [45]. Our findings may also be biased because of our sample having more interest in participating in such research. Finally, as Delta involves pages that scroll, we cannot guarantee the precise accuracy of the eye-tracking analyses in regard to the gaze plots and heat map outputs. Although feedback regarding Delta was positive, we cannot assure ecological validity in that parents would actively engage with the content outside of the user testing given the potential of the Hawthorne effect.


    Study Overview

    This paper details the development of Delta, a Web-based decision aid to support families to decide whether or not to enroll in a pediatric oncology clinical trial. We developed Delta using the Agile approach, which included a requirements analysis, planning, design, development, and user testing. We developed Delta iteratively, focusing on short sprints of work. Delta is the first a Web-based decision aid, to the authors’ knowledge, developed using the Agile approach.

    Benefit of Using Agile in Developing a Web-Based Psychosocial Tool

    The benefit of using Agile in the development of Delta has allowed for a high level of collaboration between the lead development team, HCPs and researchers, and consumers. In research projects, such as Delta, collaborators are often chosen based on their individual expertise. This aligns well with Agile, as the success of Agile projects is largely dependent on the expertise of the team involved [46].

    Involving consumers throughout development and in user testing has been especially important for Delta. The World Health Organization Declaration of Alma Ata states that “people have the right and duty to participate individually and collectively in the planning and implementation of their health care” [47]. When developing patient tools, especially for vulnerable populations as the case for Delta, patients’ and families’ roles should play an active role [48]. The Agile process caters for consumer involvement. Involving patients and families in the development of patient information material may result in more appropriate, sensitive, readable, and understandable information [49]. For Delta and similar research projects, this means that the limited funds available are available to be spent on what is considered most important for families and clinicians.

    The benefit of using Agile in the development of Delta has also been in working in short sprints and iteratively. This means that smaller aspects of the website were presented to the developmental team, followed by iterative refinement. Short sprints mean that required changes were usually minor and thus more financially manageable. For Delta, minor issues such as reorganization of content were able to be resolved early into development. Breaking Delta into more manageable units also meant that we were able to focus on detailed yet high-quality development. Quick releases are a major benefit of using Agile [50]. Being able to deliver short sprints of work has encouraged collaborators to become more engaged, provide quality feedback, and ensure deadlines are being met.

    Barriers to Using Agile in Developing a Psychosocial Tool

    One of the major limitations of using Agile to develop Delta is because of the slow-moving nature of research. This has limited the ability for short and fast sprints of work. To ensure high quality of product, involvement of clinicians and consumers has been imperative. We constantly adapted our time frames to work within the constraints of the experts involved. The Delta steering committee, however, found difficulties in maintaining constant face-to-face meetings. Agile may be difficult when working with larger teams, especially with more than 20 to 30 members [50]. Even with a team of 13 members such as with Delta, issues arose in regard to meeting deadlines for feedback and having too many meetings. To overcome the difficulty in obtaining feedback at each iteration, we allocated certain tasks to team members based on their expertise and interests. The very hierarchical approach that Agile takes to development may also not work within some workplace cultures [46]. Organizations that have greater bureaucracy and formality may also experience difficulty in fast sprints of work.

    With multiple iterations, the project scope is subject to change. Creating accurate budgets and schedules at the start of the project is difficult. For Delta, over the course of development, the scope of the website has almost doubled, from originally being a purely parent website to now including both parent and adolescent versions. Although Agile has allowed for scope creep in a more controlled and manageable manner, the change in scope has been costly and needed to be better budgeted. Some functions that were implemented during early cycles of development can also become redundant as requirements and scope change. The redesign and recoding can add significant costs.

    For Delta, we were required to obtain ethics approval to conduct user testing, which pushed back the schedule by several weeks. Difficulties with recruitment, such as booking in times for families to complete the testing at the designated location, also delayed user testing. When working with patients and families, researchers need to be aware of the time to recruit and to conduct user testing.

    Although discussions with experts and the literature have guided the development of Delta, sometimes this does not always turn out to be what is logistically possible or what families want. Agile focuses on consumers’ and development teams’ preferences, which meant there was an emphasis on the functional requirements of Delta (eg, text-to-speech function, glossary, and wording). The Research-Based Web Design and Usability Guidelines also encouraged us to consider nonfunctional requirements. Nonfunctional requirements include horizontal (ie, ability to build out and be produced in a variety of capabilities, such as Delta on a computer and also mobile phone compatible) and vertical scalability (ie, increasing resources or capabilities on a single component, such as Delta), security, maintainability, and longevity of Delta. However, when working with Agile, researchers should balance consumer preferences with usability and performance.

    Future Directions

    We recommend that researchers can develop a Web-based decision aid using Agile. We suggest that researchers set deadlines for iterations of sprints of work even in the case that the planned work may not be completely finished [51]. We strongly recommend that the 10th Agile principle of “simplicity” (ie, getting “just” enough done as needed for right now) be incorporated. We suggest that if only minimal feedback is provided by a set deadline, development continues as per these deadlines. Given the increase in value after a certain number of iterations reaches saturation [13], this may ultimately reduce the “time-to-market” without impacting quality of work. However, it must be noted that although these deadlines are technically “set,” they do require constant updating in regard to priorities and expected time for completion as the project progresses. We suggest having a product backlog of what has been completed and what needs to be done. The product backlog should acknowledge the value (ie, how useful or important is this as part of the project?) and the size (ie, how difficult or long development is expected to take?) of developing the feature or function. Better understanding the value and size can allow researchers to predict the return on investment and potentially reduce costs where possible. The product backlog should also note the velocity (ie, the rate of progress or time to complete an iteration) to ensure progress continues at an appropriate pace. We have included a suggested template for reporting the product backlog (see Multimedia Appendix 3). We also suggest future researchers’ budget for multiple iterations throughout development, both in regard to time and cost.

    Conclusions

    This paper provides an overview of the development and early testing of Delta, a patient Web-based decision aid for pediatric oncology clinical trial enrollment. We developed Delta using the gold standards of patient decision aids and Web development. In development of decision aids specifically, there is a definite need for guidelines or examples of development processes [52]. This paper has begun to fill this gap in the literature, providing more guidance for researchers looking to develop a Web-based decision aid. Delta acts as proof of concept of the use of Agile for the development of a Web-based decision aid. Aspects of Agile development may be useful to incorporate to ensure the development of a high-quality and high-value project within the constraints of cost, schedule, and scope. The development process detailed in this paper provides a suggested template from which future tools can be developed.

    Acknowledgments

    The authors would like to thank Luke Fry, Ursula Sansom-Daly, and Brad Zebrack for their assistance on this project; Objective Experience for the eye-tracking services; Feel Creative for the Web development services; and Emily Boulton for her graphic design services. They would also like to thank the Delta study steering committee: Dr Richard Mitchell, Prof Glenn Marshall, Ms Donna Drew, Ms Grace Wong, Prof Andrea Patenaude, Prof Phyllis Butow, Dr Joanne Shaw, Ms Nell Hanbury, Ms Donna Ludwinski, and Ms Sue Walter. The authors would also like to thank all participants involved in user testing. This project and EGR are supported by the Cancer Institute NSW Translational Program Grant (Experimental therapeutics for Myc-driven childhood cancer, 10/TPG/1-13). CW is supported by a Career Development Fellowship from the National Health and Medical Research Council of Australia (APP1067501). JF is supported by The Kids Cancer Project. The Behavioural Sciences Unit is supported by the Kids with Cancer Foundation, the Kids Cancer Alliance, as well as a Cancer Council New South Wales Program Grant PG16-02 with the support of the Estate of the Late Harry McPaul.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Table of Agile principles and translation to the development of Delta.

    PDF File (Adobe PDF File), 27KB
    ‎

    Multimedia Appendix 2

    Video providing an overview of the Delta website and user-testing.

    MP4 File (MP4 Video), 32MB
    ‎

    Multimedia Appendix 3

    Table providing recommended backlog template for future researchers.

    PDF File (Adobe PDF File), 23KB
    1. Woodgate RL, Yanofsky RA. Parents' experiences in decision making with childhood cancer clinical trials. Cancer Nurs 2010;33(1):11-18. [CrossRef] [Medline]
    2. Hinds PS, Oakes L, Furman W, Quargnenti A, Olson MS, Foppiano P, et al. End-of-life decision making by adolescents, parents, and healthcare providers in pediatric oncology: research to evidence-based practice guidelines. Cancer Nurs 2001 Apr;24(2):122-134. [Medline]
    3. Cousino MK, Zyzanski SJ, Yamokoski AD, Hazen RA, Baker JN, Noll RB, et al. Communicating and understanding the purpose of pediatric phase I cancer trials. J Clin Oncol 2012 Dec 10;30(35):4367-4372 [FREE Full text] [CrossRef] [Medline]
    4. Miller VA, Baker JN, Leek AC, Hizlan S, Rheingold SR, Yamokoski AD, et al. Adolescent perspectives on phase I cancer research. Pediatr Blood Cancer 2013 May;60(5):873-878 [FREE Full text] [CrossRef] [Medline]
    5. Yap TY, Yamokoski AD, Hizlan S, Zyzanski SJ, Angiolillo AL, Rheingold SR, Phase I Informed Consent (POIC) Research Team. Informed consent for pediatric phase 1 cancer trials: physicians' perspectives. Cancer 2010 Jul 01;116(13):3244-3250 [FREE Full text] [CrossRef] [Medline]
    6. Mack JW, Cronin AM, Kang TI. Decisional regret among parents of children with cancer. J Clin Oncol 2016 Nov 20;34(33):4023-4029. [CrossRef] [Medline]
    7. Elwyn G, O'Connor A, Stacey D, Volk R, Edwards A, Coulter A, International Patient Decision Aids Standards (IPDAS) Collaboration. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. Br Med J 2006 Aug 26;333(7565):417 [FREE Full text] [CrossRef] [Medline]
    8. Elwyn G, O'Connor AM, Bennett C, Newcombe RG, Politi M, Durand MA, et al. Assessing the quality of decision support technologies using the International Patient Decision Aid Standards instrument (IPDASi). PLoS One 2009;4(3):e4705 [FREE Full text] [CrossRef] [Medline]
    9. Stacey D, Légaré F, Lewis K, Barry MJ, Bennett CL, Eden KB, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev 2017 Dec 12;4:CD001431. [CrossRef] [Medline]
    10. Politi MC, Kuzemchak MD, Kaphingst KA, Perkins H, Liu J, Byrne MM. Decision aids can support cancer clinical trials decisions: results of a randomized trial. Oncologist 2016 Dec;21(12):1461-1470 [FREE Full text] [CrossRef] [Medline]
    11. Highsmith J, Cockburn A. Agile software development: the business of innovation. Computer 2001;34(9):120-122 [FREE Full text]
    12. Beck K, Beedle M, Van Bennekum A, Cockburn A, Cunningham W, Fowler M, et al. Agilemanifesto. 2001. Manifesto for agile software development   URL: http://agilemanifesto.org/ [accessed 2018-04-12] [WebCite Cache]
    13. Torrecilla-Salinas C, Sedeño J, Escalona M, Mejías M. Estimating, planning and managing Agile Web development projects under a value-based perspective. Inf Softw Technol 2015 May;61:124-144. [CrossRef]
    14. Charette R. Cutter.: Cutter Consortium e-Project Management Advisory Service; 2001. The Decision is in: Agile versus heavy Methodologies   URL: https://www.cutter.com/article/decision-agile-versus-heavy-methodologies-425981 [accessed 2018-04-12] [WebCite Cache]
    15. Dingsøyr T, Nerur S, Balijepally V, Moe NB. A decade of agile methodologies: towards explaining agile software development. J Syst Softw 2012 Jun;85(6):1213-1221. [CrossRef]
    16. Hazzan O, Tozik S. InfoQ. 2014. Agile Research   URL: https://www.infoq.com/articles/agile-academic-research [accessed 2018-04-12] [WebCite Cache]
    17. Cockburn A, Highsmith J. Agile software development: the people factor. Computer 2001;34(11):131-133 [FREE Full text]
    18. The Standish Group. 2012. CHAOS Manifesto   URL: https://cs.calvin.edu/courses/cs/262/kvlinden/resources/CHAOSManifesto2012.pdf [accessed 2018-04-12] [WebCite Cache]
    19. Abrahamsson P, Salo O, Ronkainen J, Warsta J. VTT.: Espoo, VTT Publications; 2002. Agile software development methods: review and analysis   URL: http://www.vtt.fi/inf/pdf/publications/2002/P478.pdf [accessed 2018-04-12] [WebCite Cache]
    20. Paasivaara M, Duraziewicz S, Lassenius C. Using scrum in distributed agile development: a multiple case study. 2009 Jul 13 Presented at: Fourth IEEE International Conference on Global Software Engineering; 2009; Limerick, Ireland.
    21. Olsson HH, Alahyari H, Bosch J. Climbing the Stairway to Heaven--A Mulitiple-Case Study Exploring Barriers in the Transition from Agile Development towards Continuous Deployment of Software. 2012 Sep 05 Presented at: 38th Euromicro Conference on Software Engineering and Advanced Applications; September 5-8, 2012; Cesme, Izmir, Turkey.
    22. Moe NB, Dingsøyr T, Dybå T. A teamwork model for understanding an agile team: a case study of a Scrum project. Inf Softw Technol 2010 May;52(5):480-491. [CrossRef]
    23. Dybå T, Dingsøyr T. Empirical studies of agile software development: a systematic review. Inf Softw Technol 2008 Aug;50(9-10):833-859. [CrossRef]
    24. Murero M, Rice RE, editors. The Internet and Health Care: Theory, Research, and Practice. Abingdon: Routledge; 2013.
    25. Diaz JA, Griffith RA, Ng JJ, Reinert SE, Friedmann PD, Moulton AW. Patients' use of the Internet for medical information. J Gen Intern Med 2002 Mar;17(3):180-185 [FREE Full text] [Medline]
    26. Wong C, Harrison C, Britt H, Henderson J. Patient use of the internet for health information. Aust Fam Physician 2014 Dec;43(12):875-877. [Medline]
    27. Amante DJ, Hogan TP, Pagoto SL, English TM, Lapane KL. Access to care and use of the Internet to search for health information: results from the US National Health Interview Survey. J Med Internet Res 2015 Apr 29;17(4):e106 [FREE Full text] [CrossRef] [Medline]
    28. Mattsson S, Olsson EM, Johansson B, Carlsson M. Health-related internet use in people with cancer: results from a cross-sectional study in two outpatient clinics in Sweden. J Med Internet Res 2017 May 15;19(5):e163 [FREE Full text] [CrossRef] [Medline]
    29. Welch Cline RJ, Penner LA, Harper FW, Foster TS, Ruckdeschel JC, Albrecht TL. The roles of patients' internet use for cancer information and socioeconomic status in oncologist-patient communication. J Oncol Pract 2007 May;3(3):167-171 [FREE Full text] [CrossRef] [Medline]
    30. National Health and Medical Research Council. 2016. Statement on Consumer and Community Involvement in Health and Medical Research   URL: https://www.nhmrc.gov.au/guidelines-publications/s01 [accessed 2018-04-12] [WebCite Cache]
    31. Leavitt MO, Shneiderman B. US Department of Health and Human Services. 2006. Research-based web design & usability guidelines   URL: https://www.usability.gov/sites/default/files/documents/guidelines_book.pdf [accessed 2018-04-12] [WebCite Cache]
    32. Robertson EG, Wakefield CE, Signorelli C, Cohn RJ, Patenaude A, Foster C, et al. Strategies to facilitate shared decision-making about pediatric oncology clinical trial enrollment: a systematic review. Patient Educ Couns 2018 Feb 11. [CrossRef] [Medline]
    33. Cheng C, Dunn M. Health literacy and the Internet: a study on the readability of Australian online health information. Aust N Z J Public Health 2015 Aug;39(4):309-314. [CrossRef] [Medline]
    34. Trevena LJ, Davey HM, Barratt A, Butow P, Caldwell P. A systematic review on communicating with patients about evidence. J Eval Clin Pract 2006 Feb;12(1):13-23. [CrossRef] [Medline]
    35. Pignone M, Fagerlin A, Abhyankar P, Col N, Feldman-Stewart D, Kryworuchko J, et al. International Patient Decision Aid Standards Collaboration. 2012. Clarifying and expressing values (2012 Update of the International Patient Decision Aids Standards (IPDAS) Collaboration's Background Document)   URL: http://ipdas.ohri.ca/ipdas-chapter-d.pdf [accessed 2018-04-12] [WebCite Cache]
    36. Feldman-Stewart D, Tong C, Siemens R, Alibhai S, Pickles T, Robinson J, et al. The impact of explicit values clarification exercises in a patient decision aid emerges after the decision is actually made: evidence from a randomized controlled trial. Med Decis Making 2012;32(4):616-626. [CrossRef] [Medline]
    37. Witteman HO, Scherer LD, Gavaruzzi T, Pieterse AH, Fuhrel-Forbis A, Chipenda Dansokh S, et al. Design features of explicit values clarification methods: a systematic review. Med Decis Making 2016 Dec;36(4):453-471. [CrossRef] [Medline]
    38. Nutbeam D. Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promot Int 2000;15(3):259-267. [CrossRef]
    39. Roth SP, Tuch AN, Mekler ED, Bargas-Avila JA, Opwis K. Location matters, especially for non-salient features–an eye-tracking study on the effects of web object placement on different types of websites. Int J Hum Comput Stud 2013 Mar;71(3):228-235. [CrossRef]
    40. Djamasbi S. Eye tracking and web experience. ACM Trans Comput Hum Interact 2014;6(2):37-54.
    41. Eveland Jr WP, Dunwoody S. Examining information processing on the World Wide Web using think aloud protocols. Media Psychol 2000 May;2(3):219-244. [CrossRef]
    42. Jaspers MW, Steen T, van den Bos C, Geenen M. The think aloud method: a guide to user interface design. Int J Med Inform 2004 Nov;73(11-12):781-795. [CrossRef] [Medline]
    43. Nielsen J. Estimating the number of subjects needed for a thinking aloud test. Int J Hum Comput Stud 1994 Sep;41(3):385-397. [CrossRef]
    44. Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors 2016 Nov 23;34(4):457-468. [CrossRef]
    45. Wakefield CE, Fardell JE, Doolan EL, Aaronson NK, Jacobsen PB, Cohn RJ, et al. Participation in psychosocial oncology and quality-of-life research: a systematic review. Lancet Oncol 2017 Mar;18(3):e153-e165. [CrossRef] [Medline]
    46. Nerur S, Mahapatra R, Mangalaraj G. Challenges of migrating to agile methodologies. Commun ACM 2005;48(5):72-78.
    47. World Health Organization. 1978. Declaration of Alma-Ata   URL: http://www.who.int/publications/almaata_declaration_en.pdf [accessed 2018-04-12] [WebCite Cache]
    48. Coulter A. Engaging Patients in Healthcare. Berkshire: McGraw-Hill Education; 2006.
    49. Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst Rev 2006 Jul 19(3):CD004563. [CrossRef] [Medline]
    50. Begel A, Nagappan N. Usage and perceptions of agile software development in an industrial context: An exploratory study. In: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement. 2007 Presented at: International Symposium on Empirical Software Engineering and Measurement; September 20-21, 2007; Madrid.
    51. Schwaber K, Sutherland J. Scrumguides. 2017 Nov. The Scrum Guide - The Definitive Guide to Scrum: The Rules of the Game   URL: https://www.scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf [accessed 2018-04-12] [WebCite Cache]
    52. Coulter A, Stilwell D, Kryworuchko J, Mullen PD, Ng CJ, van der Weijden T. A systematic development process for patient decision aids. BMC Med Inform Decis Mak 2013;13 Suppl 2:S2 [FREE Full text] [CrossRef] [Medline]

    ‎
    HCP: health care professional
    IPDAS: International Patient Decision Aid Standards

    Edited by G Eysenbach; submitted 25.10.17; peer-reviewed by L Warrington, R El Shafie; comments to author 29.01.18; revised version received 30.01.18; accepted 16.03.18; published 04.05.18

    Copyright

    ©Eden G Robertson, Claire E Wakefield, Richard J Cohn, Tracey O'Brien, David S Ziegler, Joanna E Fardell. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 04.05.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.