Published on in Vol 11, No 12 (2022): December

Preprints (earlier versions) of this paper are available at, first published .
Identification of Clinical Measures to Use in a Virtual Concussion Assessment: Protocol for a Mixed Methods Study

Identification of Clinical Measures to Use in a Virtual Concussion Assessment: Protocol for a Mixed Methods Study

Identification of Clinical Measures to Use in a Virtual Concussion Assessment: Protocol for a Mixed Methods Study


1School of Rehabilitation Sciences, Faculty of Health Sciences, University of Ottawa, Ottawa, ON, Canada

2Bruyere Research Institute, Ottawa, ON, Canada

3Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada

4Kite Research Institute, Toronto Rehabilitation Institute, University Health Network, Toronto, ON, Canada

5Division of Physical Medicine and Rehabilitation, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

6Division of Neurology, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada

7School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada

8Department of Medicine, University of Ottawa, Ottawa, ON, Canada

*these authors contributed equally

Corresponding Author:

Keely Barnes, MHK

School of Rehabilitation Sciences

Faculty of Health Sciences

University of Ottawa

75 Laurier Ave. E

Ottawa, ON, K1N 6N5


Phone: 1 6136126127


Background: Workplace concussions can have a significant impact on workers. The impact of concussion symptoms, combined with challenges associated with clinical environments that are loud, bright, and busy, create barriers to conducting effective in-person assessments. Although the opportunity for remote care in rural communities has long been recognized, the COVID-19 pandemic has catalyzed the transition to virtual assessments and care into the mainstream. With this rapid shift, many clinicians have been completing remote assessments. However, the approaches and measures used in these assessments have not yet been standardized. Furthermore, the psychometric properties of the assessments when completed remotely using videoconference have not yet been documented.

Objective: Through this mixed methods study, we aim to (1) identify the concussion assessment measures clinicians are currently using in person and are most relevant to the following 5 physical domains: neurological examination (ie, cranial nerve, coordination, motor, and sensory skills), cervical spine, vestibular, oculomotor, and effort assessment; (2) document the psychometric properties of the measures identified; (3) identify measures that appear feasible in a virtual context; and (4) identify practical and technical barriers or challenges, facilitators, and benefits to conducting or engaging in virtual concussion assessments.

Methods: This study will follow a sequential mixed methods design using a survey and Delphi approach, working groups with expert clinicians, and focus groups with experienced clinicians and people living with concussions. Our target sample sizes are 50 clinicians for the Delphi surveys, 4 clinician-participants for the working group, and 5-7 participants for each focus group (roughly 6-10 total groups being planned with at least two groups consisting of people living with concussions). The results from this study will inform the decision regarding the measures that should be included in a virtual assessment tool kit to be tested in a future planned prospective evaluation study.

Results: The study is expected to be completed by January 2023.

Conclusions: This mixed methods study will document the clinical measures that are currently used in person and will identify those that are most relevant to assessing the physical domains impacted by concussions. Potential feasibility of using these measures in a virtual context will be explored.

International Registered Report Identifier (IRRID): DERR1-10.2196/40446

JMIR Res Protoc 2022;11(12):e40446




Concussions or mild traumatic brain injuries affect thousands of Canadians every year [1]. Workplace injuries account for a notable portion of nonsport concussions [2]. Terry et al [3] reported that approximately 25% of adult concussions occur at work. Workplace injuries resulting in concussions pose a significant challenge for employers and insurers while disrupting the lives of the workers due to greater recovery times compared to non–work-related concussions [4]. In general, roughly 10%-20% of individuals who sustain a concussion experience symptoms lasting beyond the one- to two-week typical recovery time frame [5,6]. In the workplace context specifically, prolonged symptoms lead to reduction in productivity at work or even disability, which in turn has an economic impact on people, companies, and government agencies [3]. Although it is apparent that there is a need for further support and research in the workplace concussion rehabilitation field, a focus on the evaluation of concussion will be an important starting point to individualizing care.

Assessing symptoms and function after concussion presents clinicians with challenges due to the complex and diverse symptom presentation and a lack of sensitive and reliable clinical assessment measures [7]. Symptoms may be physical (such as dizziness, balance issues, headaches, neck pain, and vision difficulties), emotional (such as irritability and disinhibition) or cognitive (such as memory and concentration difficulties) [7-11]. All the domains of concussion symptoms must be considered [8]. Currently, however, a gold standard test to evaluate all concussion symptoms does not exist. Commonly, a battery of tests and symptom self-reports are relied upon [7,12,13]. For example, the Balance Error Scoring System and the Sensory Organization Test are used to assess balance deficits following a concussion injury [14]. The King-Devick test and the Vestibular Ocular Motor Screen may be used to evaluate oculomotor deficits [14,15]. Furthermore, the psychometric properties of the tools involved are often underdeveloped, and clinical utility varies. There is an additional challenge with all assessments of injured workers because of access to compensation that may interfere with the validity of effort provided during the assessment [3]. Measures that allow the examiner to ensure that a valid effort was made to complete the assessment should be considered.

With the shift in clinical service delivery to virtual health care driven by the COVID-19 pandemic, the need for valid and reliable approaches to virtual concussion assessments has become more pressing. In addition, people living in remote areas or for whom travel is difficult due to comorbidities continue to have a great need for assessment that can be carried out at distance [16,17]. Such assessment would also be helpful for people whose symptoms are aggravated by travel and environmental factors such as noise and light [18].

Clinicians are currently completing virtual concussion assessments using a variety of clinical assessment measures; however, as with in-person assessments, there is no standardized approach to assessing adult concussions virtually. Resources have been developed to support completion of the virtual concussion examination, including a training manual for the examination and a living guideline that outlines considerations for the pediatric examination [19,20]. Comparison of the measures identified in these resources when administered in an in-person and virtual context has not yet occurred.

The psychometric properties of many of the assessment measures being used have not been established for use in person or virtually. Accelerated adoption of information and communication technology has been occurring globally to enhance service delivery during the COVID-19 pandemic; however, it is unclear whether the outcomes of assessments completed virtually are consistent with assessments performed in person [21,22]. It is, therefore, important to understand if the measures used to assess concussions in person could provide equivalent results when used virtually [23].

Overall, there are multiple clinical assessment measures being used by clinicians to assess the physical domains impacted by a concussion injury. There is no standardized tool kit of measures available with established psychometric properties that are relevant and feasible in a virtual context, nor are there guidelines outlining specific measures to use in the adult population [12-14].


The objective of this mixed methods study is to produce a tool kit of measures that can be used in virtual assessments of concussion’s physical symptoms.

This study will follow a sequential mixed methods design [24] that will include a Delphi survey, a working group consultation, and focus groups. The methodological approach is outlined in Figure 1.

Figure 1. Sequential explanatory approach commencing with survey administration guided by the Delphi methods and working group completion (quantitative) followed by focus group completion (qualitative) to expand on and explain survey and working group results.
View this figure


Delphi Survey

Expert clinicians (ie, physiatrists, neurologists, sports medicine physicians, and physiotherapists) from across Canada will be identified through both regional and national professional and brain injury or concussion organizations and networks. Members of these networks represent an accessible and meaningful sample that will include many practicing clinicians with concussion assessment experience. The survey will be sent through these networks and associations by including the survey link in a newsletter or by sending the survey link through email to all members of the networks and associations. To ensure ‘experts’ are completing the survey, respondents will be asked to rate their concussion expertise using a 5-point Likert scale (“strongly not competent” to “strongly competent”). Any responses from participants self-reporting their competency as below 3 or “neutral” will be excluded. When needed, targeted emails will be sent to expert clinicians with publicly available contact information to ensure representation (minimum of 2 responses) of each profession.

Sample Size for Delphi Survey

Our target sample size is 50 clinicians, which we consider adequate to reach saturation on the types of clinical measures used in practice. This sample size is feasible based on an anticipated 25% response rate and assuming that approximately 200 clinicians will view the survey [25,26]. Clinicians who complete the first-round survey will be sent the second-round survey. It is expected that the second round of surveys will elicit a response rate of approximately 60% (~30 clinicians) [27].

Working Group Membership

A working group consisting of expert clinicians, including at least one neurologist, physiatrist, sports medicine physician, and physiotherapist, will meet to discuss the feasibility of virtual use of each measure identified in the survey. Members of the research team will be offered the opportunity to participate. Targeted emails to Canadian practicing clinicians will also be used to ensure representation from each clinical field. A final list of potentially feasible outcome measures will be identified in the working group, which will then be further explored in the focus groups.

Focus Group Membership

Focus groups consisting of 5 to 7 participants [28,29] will be conducted following the methodological framework outlined by Breen [30]. It is hypothesized that 6 to 10 focus groups will be adequate to reach saturation (2-3 groups consisting of people living with concussions who have experience with virtual assessment and 4-8 focus groups consisting of clinician-experts) [28,29]. Patient-participants who have attended a virtual concussion assessment will be identified and recruited from the Ottawa Hospital Rehabilitation Centre. The remaining focus groups will contain mixes of neurologists, physiatrists, sports medicine physicians, and physiotherapists. Participants who complete the Delphi surveys will have the option to express interest in participating in a focus group. Additional recruitment strategies include face-to-face recruitment at the Ottawa Hospital Rehabilitation Centre and through Ontario Workers Network clinics. Targeted emails to Canadian practicing clinicians will also be used.


Quantitative Delphi Survey Data Collection

The survey process will follow a Delphi method, which aims to seek consensus among the participants [31]. In round one, clinicians will receive a link for participation, which includes the implied consent form and a demographic questionnaire with open-ended questions; the questionnaire will have free-text boxes asking participants to identify multiple outcome measures they currently use to assess each 5 domains of interest (Figure 2). The invitation to participate in the surveys will be undersigned by a physiatrist (SM), including the affiliation, to secure buy-in and encourage responses. Measures identified by at least 15% of all participants (eg, 8/50 participants) or at least 60% of participants from one profession will be retained for the survey in round two.

Figure 2. Clinical domains of interest for identification of assessment measures and sample questions for the vestibular domain for round one and two surveys. Similar questions were provided for each domain.
View this figure

In round two, all participants from round one will be asked to rank order, according to perceived usefulness for their practice, the concussion assessment measures from each domain (ie, neurological exam, cervical spine, oculomotor, vestibular, and effort) retained from the first round (Figure 2). Prior to completing the second-round survey, participants will be invited to review reading materials with information on the retained measures, including information on sensitivity, specificity, and feasibility from the literature, when available.

Delphi Survey Questionnaire Administration

To increase response rates, reminder emails will be sent at 7-day and 14-day time points following the initial emails in the first and second rounds [25,30]. A universal level of consensus does not yet exist for the Delphi survey approach; however, various works have suggested a range of 51%-80% agreement [27]. For the purposes of this study, an above 51% agreement level for rankings of assessments has been set for the second round. Assessments that reach 15% in round one will be considered for clinical feasibility by the expert clinician working group.

Working Group to Derive Consensus on the List of Feasible Measures

A working group consisting of expert clinicians will meet over a videoconferencing platform, which involves bidirectional visual and audio communication technology, to discuss the current feasibility of the identified measures in a virtual context. Any measures deemed not feasible in the working group will be eliminated from further discussion in the focus groups. The final list of measures will be further explored in the clinician focus groups.

Qualitative Focus Group Data Collection

Focus groups with both clinician and patient-participants will be conducted. An interview guide with broad open-ended questions will be used as a prompt for the conduct of the focus groups. All focus groups will occur over a videoconferencing platform and will be audio and video recorded for later transcription. Multimedia Appendix 1 includes the semistructured interview guide for both patient-participants and clinician-participants.

Clinician-Participant Focus Groups

Prior to conducting the focus group, clinician-participants will be provided with available psychometric properties documented in the literature of the in-person measures that were identified from the second round of the surveys as well as descriptions of the assessments and instructions on how to complete the assessments. Clinician-participants in the focus groups will be prompted about the practical and technical issues associated with using each of the final measures. Barriers and facilitators associated with the assessments, including adverse events experienced or observed, will also be explored.

Patient-Participant Focus Groups

Patient-participants will be prompted to discuss the benefits and challenges associated with virtual concussion assessments based on their experiences.

Data Analysis

Quantitative Analysis of Delphi Surveys

Clinical measures identified in round one of the Delphi surveys will be categorized into the preestablished domains (ie, neurological examination, cervical spine, vestibular, oculomotor, and effort assessment). Frequency counts of the measures will be used to identify most commonly identified measures [32], and consistency of wording for the description of measures will be documented. For example, some clinicians may describe the measure rather than providing the name of the measure.

The quantitative data obtained in the second round of surveys (ie, rank order of measures) will be analyzed descriptively (ie, summary statistics to demonstrate patterns in the data). Measures of frequency and agreement percentages will be calculated.

Qualitative Analysis of Focus Groups

NVivo will be used to organize the qualitative data analysis. Recordings from the focus groups will be transcribed verbatim. Content analysis will be used to analyze the focus group data [33]. Two research assistants will independently identify codes related to barriers or challenges, benefits, and facilitators associated with using each of the identified measures. Codes will be sorted into 2 levels of categories: categories related to assessment of specific symptoms or domains and categories related to virtual assessment in general.

Ethics Approval

Ethics approval was obtained by the Ottawa Health Sciences Network Research Ethics Board (20210575-01H) in September 2021 followed by the Bruyère Research Institute Research Ethics Board (M16-22-006) and the University of Ottawa Board of Ethics (H-02-22-7611) in February 2022.

Survey administration and the working group have been completed. Focus group recruitment is underway. The final results of the surveys, working group, and focus groups will lead to the identification of clinical measures to use in a virtual assessment tool kit, which will be tested in a future planned evaluation study.

Expected Outcomes

We presented a protocol for a mixed methods study to identify the most appropriate clinical measures to include in a virtual assessment. We hypothesize that the measures identified in the surveys will vary based on clinical profession. We anticipate reliability properties of the identified measures to range from moderate to strong (intraclass correlation coefficients=0.41 to above 0.81). It is anticipated that the working group and focus group discussions will lead to an understanding of some of the real and perceived barriers and facilitators related to participating in or completing a virtual concussion assessment. We expect patient-perceived challenges to relate to the technical issues with technology and clinician-perceived challenges to relate to the challenges associated with engaging in hands-on approaches in virtual care.

A toolbox of concussion physical assessments has been proposed by Matuszak et al [14], which includes measures to assess domains that are frequently impacted by a concussion injury; however, many of the physical assessment measures used in the concussion population have limited psychometric data. The proposed toolbox includes evaluation of vital signs, mental status, neurological examination (ie, cranial nerves, manual muscle testing, and reflexes), head and cervicothoracic evaluation, balance or coordination assessment, and vestibulo-ocular evaluation [14]. This toolbox is proposed for the in-person examination. However, many of the virtual resources and guidelines include similar evaluations [19,20].

Although telehealth has existed for a long time, the COVID-19 global pandemic has increased both its need and its use in delivering health care services [34]. The COVID-19 pandemic has accelerated the need for telemedicine-supported remote assessments and has pushed both clinicians and researchers to determine what aspects of medical care could be feasible in a telehealth context [34,35]. A scoping review by O’Neil et al [36] noted that videoconferencing could be a valid means to remotely assess patients with moderate to severe traumatic brain injury. In addition, according to Fjeldstad-Pardo et al [37], no adverse events have been identified during participation in telerehabilitation, indicating that it is a safe approach to deliver services. Specifically, telerehabilitation has been reported to be feasible and effective for the intervention and management of neurological patients [38]. There has been a significant increase in the use of virtual assessments, with a rapid transition due to the pandemic. This transition limited the ability to implement virtual care using a planned and organized approach. The transition was a reaction to the pandemic rather than a planned response [39]. Due to this reactionary response, clinicians have been forced to complete remote assessments with limited information on the accuracy and reliability of the measures used in these assessments.

It is, therefore, important to understand which assessments clinicians are completing both in person and virtually in the context of the concussion examination. A study by Tobler-Amman et al [40] found that correlation values were low when measures assessing similar constructs in people living with stroke were administered in person and virtually. Although clinicians thought the measures were assessing the same construct, the low correlation values indicate that administering the measures in the two different contexts may not provide the same information. Similarly, a study by Wang et al [41] on people living with Parkinson disease found that participants elicited a weaker reach when an assessment was completed virtually compared to in person. In the context of the concussion examination, understanding which assessments are being conducted and how the assessments are being conducted both in person and virtually is necessary to determine the equivalence of the assessments when administered in both contexts. Psychometric properties should be established in both in-person and virtual contexts of use to ensure accurate and reliable implementation in both contexts [23]. If the outcome measures used in person have poor documented psychometric properties or no documented psychometric properties, there may be additional challenges to validity and reliability when using the measure in a virtual context. It is important to gain an understanding of the properties of the clinical measures and their ability to produce equivalent results to the in-person assessment to inform decision-making in terms of their use in practice and their potential use in the virtual context [42].

With the increase in need for the uptake of telehealth in rural and remote areas, and now globally due to the COVID-19 pandemic, standardization of a feasible virtual concussion assessment is needed. The identification of the clinical measures that are most relevant and contribute the most reliable information in an adult concussion examination and the identification of the barriers and facilitators associated with using these specific measures in a telehealth context is an important first step to lead to this standardization. The results of this study and the future planned prospective evaluation study will be published and disseminated to targeted end users and networks.

Strengths and Limitations of Methodological Approaches

An important strength of the Delphi method is that it is a useful approach when lack of clarity exists, which is likely the case in the concussion assessment field [43]. It is apparent that although the Delphi approach has many strengths, it also has some limitations. Generally, the Delphi approach lacks agreed-upon standards or consensus [43].

Focus groups have been reported to be beneficial when participants are in different geographical locations [30], which will occur in this project due to the need to gain input from various clinicians and from people living with concussions to inform the follow-up studies. Some important limitations of focus groups include the following: the group setting nature of the focus groups may discourage participants from sharing their input; similarly, one single participant may dominate the focus group discussion (ie, overpower other participants) [30].


This mixed methods study will identify the assessment measures that are currently being used in person to evaluate people living with concussions. This study will further identify the barriers and facilitators as well as the challenges and benefits associated with using the identified measures, virtually. Psychometric properties documented in the literature of the in-person measures that will be discussed in the focus groups will be described. Results from this study will inform the selection of measures to include in a virtual assessment tool kit, which will be tested in follow-up studies. It is apparent that there are multiple gaps in the clinical assessment of concussion. There appears to be a need to further explore and expand on the understanding of the measures used to assess individuals experiencing prolonged symptoms in person; however, there is currently a need to explore the use of these measures in a virtual context. The limitations of the COVID-19 pandemic have highlighted the need for a rapid change in service delivery to virtual means, and although limited information exists regarding concussion assessment in a virtual context, remote assessments are currently being conducted in clinical practice, and therefore, need to be better understood.


KB is supported by an admission scholarship from the University of Ottawa. This Project was funded by a grant (MARS2021) provided by the Workplace Safety and Insurance Board (WSIB) of Ontario. The provision of grant support by WSIB does not in any way infer or imply endorsement of the content by the WSIB.

Data Availability

Data sharing is not applicable to this study, as no data sets were analyzed during the study.

Authors' Contributions

All authors have made substantial contribution to the work presented in this paper.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Interview guide.

DOCX File , 14 KB

Multimedia Appendix 2

Peer reviewed by Workplace Safety & Insurance Board (WSIB) Grants Program (Toronto, Ontario, Canada).

PDF File (Adobe PDF File), 164 KB

  1. Langer L, Levy C, Bayley M. Increasing incidence of concussion: true epidemic or better recognition? J Head Trauma Rehabil 2020;35(1):E60-E66. [CrossRef] [Medline]
  2. Matthews A, Iboaya A, Smith J, Bell K. Acute management of concussion diagnostic criteria. In: Concussion: Assessment, Management and Rehabilitation. Amsterdam, Netherlands: Elsevier; 2020:1-12.
  3. Terry D, Iverson G, Panenka W, Colantonio A, Silverberg N. Workplace and non-workplace mild traumatic brain injuries in an outpatient clinic sample: a case-control study. PLoS One 2018;13(6):e0198128 [FREE Full text] [CrossRef] [Medline]
  4. Colantonio A, Mroczek D, Patel J, Lewko J, Fergenbaum J, Brison R. Examining occupational traumatic brain injury in Ontario. Can J Public Health 2010 Mar 1;101(S1):S58-S62. [CrossRef]
  5. Hill S, Barr C, Killington M, McLoughlin J, Daniels R, van DBM. The Design and Development of MOVE-IT: A System for Remote Vestibular and Oculomotor Assessment in People with Concussion. Amsterdam, Netherlands: IOS Press Ebooks; 2021:27-36.
  6. Mollayeva T, Shapiro CM, Mollayeva S, Cassidy JD, Colantonio A. Modeling community integration in workers with delayed recovery from mild traumatic brain injury. BMC Neurol 2015 Oct 09;15(1):194 [FREE Full text] [CrossRef] [Medline]
  7. Broglio S, Macciocchi S, Ferrara M. Sensitivity of the concussion assessment battery. Neurosurgery 2007;60(6):1050-1058. [CrossRef]
  8. McKay C, Velikonja D, Davis J, Hamilton J, Lubinsky T, Lopo V, et al. OPA guidelines for best practices in the assessment of concussion and related symptoms. Ontario Psychological Association. 2016.   URL: https:/​/www.​​getmedia/​b7ada02e-76ca-4a5c-891a-bc610c81a213/​OPAConcussionGuidelinesFINAL2018.​pdf [accessed 2022-11-07]
  9. Clugston J, Houck Z, Asken B, Boone J, Kontos A, Buckley T, et al. Relationship between the King-Devick test and commonly used concussion tests at baseline. J Athl Train 2019 Dec;54(12):1247-1253 [FREE Full text] [CrossRef] [Medline]
  10. Murray N, Salvatore A, Powell D, Reed-Jones R. Reliability and validity evidence of multiple balance assessments in athletes with a concussion. J Athl Train 2014;49(4):540-549 [FREE Full text] [CrossRef] [Medline]
  11. Scorza KA, Raleigh MF, O'Connor FG. Current concepts in concussion: evaluation and management. Am Fam Physician 2012 Jan 15;85(2):123-132 [FREE Full text] [Medline]
  12. Randolph C, Millis S, Barr W, McCrea M, Guskiewicz KM, Hammeke TA, et al. Concussion symptom inventory: an empirically derived scale for monitoring resolution of symptoms following sport-related concussion. Arch Clin Neuropsychol 2009 May;24(3):219-229 [FREE Full text] [CrossRef] [Medline]
  13. Kontos A, Collins M. Psychological issues associated with concussion. APA 2018:59-71. [CrossRef]
  14. Matuszak JM, McVige J, McPherson J, Willer B, Leddy J. A practical concussion physical examination toolbox. Sports Health 2016 Mar 28;8(3):260-269 [FREE Full text] [CrossRef] [Medline]
  15. Mucha A, Collins M, Elbin R, Furman JM, Troutman-Enseki C, DeWolf RM, et al. A brief Vestibular/Ocular Motor Screening (VOMS) assessment to evaluate concussions: preliminary findings. Am J Sports Med 2014 Oct;42(10):2479-2486 [FREE Full text] [CrossRef] [Medline]
  16. Vargas BB, Shepard M, Hentz JG, Kutyreff C, Hershey LG, Starling AJ. Feasibility and accuracy of teleconcussion for acute evaluation of suspected concussion. Neurology 2017 Mar 24;88(16):1580-1583. [CrossRef]
  17. Ownsworth T, Theodoros D, Cahill L, Vaezipour A, Quinn R, Kendall M, et al. Perceived usability and acceptability of videoconferencing for delivering community-based rehabilitation to individuals with acquired brain injury: a qualitative investigation. J Int Neuropsychol Soc 2020 Jan 27;26(1):47-57. [CrossRef]
  18. Ryan LM, Warden DL. Post concussion syndrome. Int Rev Psychiatry 2003 Nov 11;15(4):310-316. [CrossRef] [Medline]
  19. Johnston S, Leddy J, Reed N, Cortel-LeBlanc A, Hafizi R, Laing S, et al. Virtual Concussion Exam Training Manual. 2022.   URL: [accessed 2022-11-07]
  20. Ellis MJ, Mendez I, Russell K. Preliminary clinical algorithm to optimise remote delivery of paediatric concussion care in Canada's North. Int J Circumpolar Health 2020 Dec 22;79(1):1832390 [FREE Full text] [CrossRef] [Medline]
  21. Almathami HKY, Win KT, Vlahu-Gjorgievska E. Barriers and facilitators that influence telemedicine-based, real-time, online consultation at patients' homes: systematic literature review. J Med Internet Res 2020 Feb 20;22(2):e16407 [FREE Full text] [CrossRef] [Medline]
  22. Mani S, Sharma S, Singh DK. Concurrent validity and reliability of telerehabilitation-based physiotherapy assessment of cervical spine in adults with non-specific neck pain. J Telemed Telecare 2019 Jul 04;27(2):88-97. [CrossRef]
  23. Russell TG, Martin-Khan M, Khan A, Wade V. Method-comparison studies in telehealth: Study design and analysis considerations. J Telemed Telecare 2017 Sep 11;23(9):797-802. [CrossRef]
  24. Creswell J, Plano CV. Designing and Conducting Mixed Methods Research. New York, NY: SAGE Publications, Inc; 2018.
  25. Burns KE, Duffett M, Kho ME, Meade MO, Adhikari NK, Sinuff T, ACCADEMY Group. A guide for the design and conduct of self-administered surveys of clinicians. CMAJ 2008 Jul 29;179(3):245-252 [FREE Full text] [CrossRef] [Medline]
  26. Cunningham CT, Quan H, Hemmelgarn B, Noseworthy T, Beck CA, Dixon E, et al. Exploring physician specialist response rates to web-based surveys. BMC Med Res Methodol 2015 Apr;15:32 [FREE Full text] [CrossRef] [Medline]
  27. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000 Oct;32(4):1008-1015. [Medline]
  28. Guest G, Namey E, McKenna K. How many focus groups are enough? Building an evidence base for nonprobability sample sizes. Field Methods 2016 Jul 24;29(1):3-22. [CrossRef]
  29. Hennink MM, Kaiser BN, Weber MB. What influences saturation? Estimating sample sizes in focus group research. Qual Health Res 2019 Aug 10;29(10):1483-1496 [FREE Full text] [CrossRef] [Medline]
  30. Breen RL. A practical guide to focus-group research. J Geogr High Educ 2006 Nov;30(3):463-475. [CrossRef]
  31. Hardin K, Black C, Caldbick K, Kelly M, Malhotra A, Tidd C, et al. Current practices among speech-language pathologists for mild traumatic brain injury: a mixed-methods modified Delphi approach. Am J Speech Lang Pathol 2021 Jul 14;30(4):1625-1655. [CrossRef] [Medline]
  32. Winser S, Smith C, Hale L, Claydon L, Whitney S. Balance outcome measures in cerebellar ataxia: a Delphi survey. Disabil Rehabil 2015;37(2):165-170 [FREE Full text] [CrossRef] [Medline]
  33. Stemler S. An overview of content analysis. Pract Assess Res 2000;7(17):1-5. [CrossRef]
  34. Toresdahl B, Young W, Quijano B, Scott D. A systematic review of telehealth and sport-related concussion: baseline testing, diagnosis, and management. HSS J 2021 Feb;17(1):18-24 [FREE Full text] [CrossRef] [Medline]
  35. Goyal D, Divi S, Schroeder G, Pfeifer R, Canseco JA, Bowles D, et al. Development of a telemedicine neurological examination for spine surgery: a pilot trial. Clin Spine Surg 2020 Nov;33(9):355-369. [CrossRef] [Medline]
  36. O’Neil J, van Ierssel J, Sveistrup H. Remote supervision of rehabilitation interventions for survivors of moderate or severe traumatic brain injury: A scoping review. J Telemed Telecare 2019 May 31;26(9):520-535. [CrossRef]
  37. Fjeldstad-Pardo C, Thiessen A, Pardo G. Telerehabilitation in multiple sclerosis: results of a randomized feasibility and efficacy pilot study. Int J Telerehabil 2018;10(2):55-64 [FREE Full text] [CrossRef] [Medline]
  38. Vargas B, Channer D, Dodick D, Demaerschalk B. Teleconcussion: an innovative approach to screening, diagnosis, and management of mild traumatic brain injury. Telemed J E Health 2012 Dec;18(10):803-806. [CrossRef] [Medline]
  39. Sosnowski R, Kamecki H, Joniau S, Walz J, Klaassen Z, Palou J. Introduction of telemedicine during the COVID-19 pandemic: a challenge for now, an opportunity for the future. European Urology 2020 Dec;78(6):820-821. [CrossRef]
  40. Tobler-Ammann BC, de Bruin ED, Fluet M, Lambercy O, de Bie RA, Knols RH. Concurrent validity and test-retest reliability of the Virtual Peg Insertion Test to quantify upper limb function in patients with chronic stroke. J Neuroeng Rehabil 2016 Jan 22;13(1):8 [FREE Full text] [CrossRef] [Medline]
  41. Wang C, Hwang W, Fang J, Sheu C, Leong I, Ma H. Comparison of virtual reality versus physical reality on movement characteristics of persons with Parkinson's disease: effects of moving targets. Arch Phys Med Rehabil 2011 Aug;92(8):1238-1245. [CrossRef] [Medline]
  42. Valovich McLeod TC, Barr W, McCrea M, Guskiewicz K. Psychometric and measurement properties of concussion assessment tools in youth sports. J Athl Train 2006;41(4):399-408 [FREE Full text] [Medline]
  43. Fink-Hafner D, Dagen T, Doušak M, Novak M, Hafner-Fink M. Delphi method. Adv Meth Stat 2019 Jul 01;16(2):1-16. [CrossRef]

Edited by T Leung; This paper was peer reviewed by the Workplace Safety & Insurance Board (WSIB) Grants Program (Toronto, Ontario, Canada). See the Multimedia Appendix for the peer-review report; submitted 16.08.22; accepted 28.10.22; published 22.12.22


©Keely Barnes, Heidi Sveistrup, Mark Bayley, Mary Egan, Michel Rathbone, Monica Taljaard, Shawn Marshall. Originally published in JMIR Research Protocols (, 22.12.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.