Background: Health technology assessment (HTA) is one of the main tools that health systems have to appraise evidence and determine the value of a given health technology. Although the existing HTA frameworks are useful tools for the evaluation of a wide range of health technologies, more and more experts, organizations across the world, and HTA agencies are highlighting the need to update or develop specific methodological frameworks for the evaluation of digital health technologies in order to take into account additional domains that cover these technologies’ intrinsic characteristics.
Objective: The purpose of our scoping review is to identify the methodological frameworks that are used worldwide for the assessment of digital health technologies; determine what dimensions and aspects are being considered; and generate, through a thematic analysis, a proposal for a methodological framework that is based on the most frequently described dimensions in the literature.
Methods: The scoping review will be performed in accordance with the guidelines established in the updated statement of the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews). We will search for peer-reviewed and grey literature published between 2011 and the date of the search execution. The retrieved references will be reviewed in a single-blind manner by 2 independent authors, and their quality will be assessed by using the Critical Appraisal Skills Program tool. The ATLAS.ti software (Scientific Software Development GmbH) will be used for data extraction and to perform the thematic analysis.
Results: The scoping review is currently (May 2022) in progress. It is expected to be completed in October 2022, and the final results of the research will be presented and published by November 2022.
Conclusions: To our knowledge, no studies have been published to date that identify the existing methodological frameworks for digital HTA, determine which dimensions must be evaluated for correct decision-making, and serve as a basis for the development of a methodological framework of reference that health care systems can use to carry out this kind of assessment. This work is intended to address this knowledge gap of key relevance for the field of HTA.
International Registered Report Identifier (IRRID): DERR1-10.2196/39905
European health systems, including the Spanish National Health System, face different challenges associated mainly with the progressive aging of the population [- ]; the increasing prevalence of chronic conditions [ ]; the growing need to medicalize citizens [ ]; the rapid growth of health care expenditures, which are exceeding national incomes [ , ]; or the unequal distribution of health services throughout the territories [ , ]. Likewise, the health crisis caused by the SARS-CoV-2 (COVID-19) pandemic has increased the stress on health systems, challenging their sustainability and the values of universality, equity, and quality on which they are based [ , - ]. Further, the COVID-19 pandemic has forced a hasty change from the face-to-face care model to a non–face-to-face model [ ].
In this context, digital health, which is defined by the World Health Organization (WHO) as “the field of knowledge and practice associated with the development and use of digital technologies to improve health”  and by the European Commission as “the set of tools and services that use information and communication technologies to improve prevention, diagnosis, treatment, monitoring and management of health-related issues and to monitor and manage lifestyle-habits that impact health” [ ], offers a unique opportunity to face these challenges and improve the accessibility, efficiency, sustainability, and quality of health systems [ , ].
The integration of digital health technologies (eg, mobile health [mHealth] apps, artificial intelligence [AI]–based solutions, etc) in health systems, however, entails certain challenges that hinder its implementation. Generally, these challenges are related to the rights of patients, the ownership of data, acceptance by users, the absence of adequate technological infrastructures, the literacy of professionals and patients, or the lack of robust evidence that makes the decision-making process difficult and can result in the development and reproduction of low-value technologies with a short, useful life span [, ].
With regard to this last aspect, one of the main tools that the Spanish National Health System uses to generate evidence and determine the value of a given health technology is the health technology assessment (HTA) . HTA, as well as its definition, has evolved since the 1980s, incorporating different dimensions in addition to safety, efficacy, effectiveness, and efficiency, such as the inclusion of the patient perspective, organizational aspects, or social impacts [ ]. Currently, health technology assessment is defined as a “multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its life cycle,” and it is intended to inform decision-making processes in order to promote an equitable, efficient, and high-quality health system [ ].
HTA is generally carried out by using specific methodological frameworks, such as the HTA Core Model of the European Network for Health Technology Assessment (EUnetHTA; known since 2022 as the EUnetHTA 21 Consortium) . In Spain, HTA is performed by using the Guideline for the Development and Adaptation of Rapid Health Technology Assessment Reports of the Spanish Network of Agencies for Assessing National Health System Technologies and Performance, which was developed based on the HTA Core Model and other methodological frameworks [ ]. Generally, these frameworks specify and standardize methods for evaluating the quality and value of health technologies, as well as the relevant information or elements that must be reported for a complete HTA. In this sense, the HTA Core Model 3.0 describes the following nine domains to be evaluated [ ]: health problem and current use of technology, description and technical characteristics, safety, clinical effectiveness, costs and economic effectiveness, ethical analysis, organizational aspects, patient and social aspects, and legal aspects.
Although these frameworks are useful tools for the evaluation of a wide range of health technologies, more and more experts, organizations across the world (eg, the WHO), and HTA agencies (eg, National Institute for Health and Care Excellence [NICE], Canada’s Drug and Health Technology Agency, Finnish Coordinating Center for Health Technology Assessment [FinCCHTA], etc) are highlighting the need to update or develop specific methodological frameworks for the evaluation of digital health technologies that take into account additional domains (eg, interoperability, usability, etc) that cover these technologies’ intrinsic characteristics [, , ]. For this reason, some initiatives have emerged, such as the Evidence Standard Framework of the NICE [ ] or the Digi-HTA Framework of the FinCCHTA [ ]. However, most of these initiatives have some limitations, such as the development being conducted according to a specific socioeconomic or national context that hinders the transferability or applicability of the tool or framework to other countries, the specificity or exclusion of certain digital health technologies with limitations in their use, or the low evidence available in relation to the real usefulness of the methodological frameworks.
In this context, we intend to develop a scoping review with the aim of identifying the methodological frameworks that are used worldwide for the evaluation of digital health technologies; determining what dimensions and aspects are being considered; and generating, through a thematic analysis, a proposal for a methodological framework that is based on the most frequently described dimensions in the literature.
Identifying the Research Questions
The scoping review will answer the following research questions:
- What methodological frameworks currently exist for digital HTA?
- What dimensions are being considered for the digital HTA?
- What dimensions are being described in more frequency in existing methodological frameworks?
- Are different dimensions being considered depending on whether the HTA is for a non–face-to-face care model of health care provision, a mobile device (mHealth), or a device that incorporates AI?
Overview of Methods for Conducting the Scoping Review
The scoping review of the available scientific literature will be carried out in accordance with the guidelines established in the updated statement of the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews, see) [ ], with the aim of guaranteeing the transparency and reproducibility of the results.
Different experts in the fields of HTA and digital health technology participated in the planning of the study. Furthermore, they will be involved in its execution.
Identifying Relevant Studies
The search strategy will be designed by an information specialist (RPP) and be based on the validated filter of Ayiku et al  for health apps; we will add the terms for concepts related to mHealth, remote care models, AI, digital health, methodological frameworks, and HTA. Taking into account the research questions, the initial search strategy for MEDLINE Ovid was designed by the information specialist (RPP) and peer-reviewed according to the Peer Review of Electronic Search Strategies Statement by JSF and CMP. The initial search strategy ( ) will be exported to the following electronic databases: Ovid via MEDLINE, CINAHL Plus, Embase, Cochrane Library, Scopus, Web of Science, and TripDatabase. The characteristics of each database related to syntax, controlled vocabulary, and proximity operators will be taken into account. No time, language, or other filters will be used.
The identification of the studies will be complemented with a manual search that will be based on the references of the included studies, as well as the websites of the HTA agencies detected through the web pages of the EUnetHTA, the International Network for Agencies for Health Technology Assessment, and Health Technology Assessment International. Finally, a search will be carried out in Google Scholar, which will include the first 250 items to ensure that no relevant results are missed .
The criteria for the selection of studies in the reference screening process will be based on the previously detailed research questions, and these criteria are described in, using the PICo-D (Population, Phenomenon of Interest, Context, and Design) format [ ]. It should be noted that the PICo-D format has been used instead of the traditional PICO-D (Population, Intervention, Comparator, Outcomes, Design) format due to the qualitative nature of the research questions and the characteristics of the phenomenon of interest.
Research questions in the PICo-D (Population, Phenomenon of Interest, Context, and Design) format (inclusion criteria).
- Digital health technology assessment
Phenomenon of interest
- Specific methodological frameworks for the evaluation of digital health (with a special focus on mobile health, non–face-to-face models, and devices that incorporate artificial intelligence) that describe the domains that must be taken into account in this type of process, as well as the levels of evidence that should be considered for this process
- Health technology assessment
- Methodological guidelines and frameworks, scoping reviews, systematic reviews, consensus documents, and qualitative studies
In the study selection process, studies published before 2011, studies that do not describe dimensions or evaluation criteria, studies that are based on methodological frameworks that are not intended for this purpose (eg, EUnetHTA Core Model 3.0), comments, editorials, letters, and conference abstracts will be excluded. Likewise, methodological frameworks or tools that focus on the evaluation of digital health technologies by users (eg, user version of the Mobile App Rating Scale) and documents in languages other than English, Spanish, or Catalan will also be excluded. Nevertheless, in the case that we identify any methodological frameworks written in languages other than those mentioned above, the authors of those documents will be contacted to confirm the absence of an English version. Additionally, the translation of the documents will be considered.
All identified references will be imported into the EndNote bibliographic citation manager (version 20.2.1; Clarivate) , and duplicates will be removed according to the guidelines of Bramer et al [ ].
The selection of studies will be carried out in 2 different phases. The first one will be the selection of studies via a single-blind peer review of the titles and abstracts of the references identified in the bibliographic search. This will be conducted by authors CMP and JSF. The second one will be a full-text, single-blind review of the studies included in the first phase, and this will be carried out by the same authors (CMP and JSF) in accordance with the selection criteria detailed above.
The quality of the evidence will be assessed by CMP and JSF using the Critical Appraisal Skills Program tool . However, it should be noted that this tool cannot be used to obtain an overall score of the quality of the studies. Therefore, no references will be excluded due to their quality.
Charting the Data
After the selection of the articles, the data of the included studies will be extracted. This task will be carried out by 3 reviewers (CMP, RPP, and JSF) using the web and desktop versions of the ATLAS.ti software (version 22.0; Scientific Software Development GmbH)  and the data extraction sheets that were designed ad hoc for this purpose according to the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions [ ]. The data to be extracted through the tables will be authors, publication dates, methodological framework/tool names, HTA agencies, countries, study designs, technology characteristics, the number of dimensions and criteria, dimensions, and detailed criteria.
For cases of discrepancies in either of the two processes (selection of studies or data extraction), a consensus will be reached among 3 reviewers (CMP, RPP, and JSF). If a discrepancy remains, a fourth reviewer (RVH) will be consulted.
Collecting, Summarizing, and Reporting the Results
The evidence will be analyzed by using 2 approaches. First, a descriptive analysis will be carried out to evaluate and report the existing methodological frameworks and their characteristics. Second, a thematic analysis will be carried out according to the following three phases, which are described by Thomas and Harden , to identify HTA dimensions for digital health technologies: (1) line-by-line text coding, (2) the development of descriptive topics, and (3) the generation of analytical themes. Both analyses will be executed by three of the authors (CMP, RPP, and JSF) using the web and desktop versions of the ATLAS.ti software (version 22.0) [ ].
The synthesis of the evidence will be carried out in a narrative manner, taking into account the selection criteria and the research questions detailed above.
Dimensions identified from systematic reviews that derived data from primary studies that are also identified in our systematic search will only be counted once in order to avoid the duplication of data and the risk of bias.
No ethical board approval is necessary to conduct this scoping review.
The scoping review is currently (May 2022) in progress. It is expected to be completed in October 2022, and the final results of the research will be presented and published by November 2022.
A dissemination plan has been developed to share the knowledge generated from the scoping review. Specifically, the results obtained from our work will be openly published in a scientific paper by March 2023, and they will also be presented at a national congress and an international congress. Furthermore, the results will be shared with the Spanish Ministry of Health, other relevant Spanish health care stakeholders, and national and international HTA agencies via direct emails and webinars.
Our scoping review is expected to identify and evaluate the existing frameworks for digital HTA and generate, through a thematic analysis, a proposal for a methodological framework that is based on the most frequently described dimensions in the literature. Although there are not many published frameworks that specifically address the assessment of digital health technologies, it is expected that considerable differences will be found among them (eg, in terms of the dimensions considered or the kinds of digital health technology addressed). Besides, additional domains that can be compared to those of conventional HTA methodological frameworks (eg, HTA Core Model) will probably be found. Finally, this work can be useful for the HTA field, as it will outline the main additional dimensions that should be considered for digital HTA and propose a framework that covers the intrinsic characteristics of digital health technologies [, , ].
Comparison With Previous Works
There are some publications that focus on analyzing—through qualitative studies, narrative reviews, or systematic reviews—what information related to the dimensions of the EUnetHTA HTA Core Model are reported by studies on digital health, what methodological frameworks and tools exist for the evaluation of digital health technologies, and what dimensions are considered by these frameworks. However, none of the articles identified through the preliminary literature search, which was done before the development of this protocol, addresses the same research questions from the perspective of HTA. For example, a review by Moshi et al  analyzed 45 tools for the evaluation of mobile apps, regardless of their intended audience, and a review by von Huben et al [ ] analyzed the degree to which such tools cover the evaluation domains of the EUnetHTA HTA Core Model 3.0.
Strengths and Limitations
There are 2 main strengths to our study. First, different experts in the fields of HTA and digital health technology will participate in the planning and development of the study. Second, the validated filter of Ayiku et al  has been used to develop the search strategy. The main limitation of our study is the exclusion of frameworks published in languages other than English, Spanish, or Catalan. Another limitation is the use of controlled vocabulary that is not suited to the current state of knowledge in the digital health field.
According to the WHO , there is low evidence available in relation to the real usefulness of the existing methodological frameworks for digital HTA. Future work will be conducted to explore the utility of the methodological framework that will be developed based on our scoping review and compare it to existing frameworks.
To our knowledge, no study has been published so far with the aim of identifying the existing methodological frameworks for digital HTA, determining which dimensions must be evaluated for correct decision-making from the HTA perspective, and serving as a basis for the development of a methodological framework of reference that health care systems can use to carry out this kind of assessment. This work is intended to address this knowledge gap and may be useful in the field of HTA.
We acknowledge Liliana Arroyo Moliner, Juan Antonio Blasco, Mario Cárdaba, Carme Carrion Ribas, Montserrat Daban, Maria Dolors Estrada Sabadell, Iñaki Gutierrez Ibarluzea, Ma Fé Lapeña Gutiérrez, Felip Miralles, Montserrat Moharra Frances, Celia Muñoz Fernández, Lilisbeth Perestelo Pérez, Jordi Piera Jiménez, Carme Pratdepàdua Bufill, Lucía Prieto, Janet Puñal Riobóo, and Benigno Rosón Calvo for reviewing the protocol.
This research is framed within the budget of the work plan of the Spanish Network of Health Technology Assessment Agencies and Benefits of the National Health System.
JSF, CMP, RPP, and RMVH contributed to the development of the protocol. RPP was in charge of developing the search strategy. JSF, CMP, and RPP will develop the rest of the tasks of the scoping review.
Conflicts of Interest
PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) checklist.DOC File , 66 KB
Search strategy.DOC File , 61 KB
- Avanzas P, Pascual I, Moris C. The great challenge of the public health system in Spain. J Thorac Dis 2017 May;9(Suppl 6):S430-S433 [FREE Full text] [CrossRef] [Medline]
- European semester thematic factsheet: Health systems. European Commission. 2016. URL: https://ec.europa.eu/info/sites/default/files/file_import/european-semester_thematic-factsheet_health-systems_en_0.pdf [accessed 2022-09-26]
- Pla Director de Sistemes d’Informació del SISCAT: Construint junts una estratègia de salut digital per a Catalunya. Generalitat de Catalunya, Departament de Salut. 2017. URL: https://salutweb.gencat.cat/web/.content/_ambits-actuacio/Linies-dactuacio/Plans-sectorials/pd_sistemes_informacio/pla_director_final_v27.pdf [accessed 2022-09-26]
- Health 2020: A European policy framework and strategy for the 21st century. World Health Organization. 2020. URL: https://www.euro.who.int/__data/assets/pdf_file/0011/199532/Health2020-Long.pdf [accessed 2022-09-26]
- World Health Organization. WHO Guideline: Recommendations on Digital Interventions for Health System Strengthening. Geneva, Switzerland: World Health Organization; Jun 06, 2019.
- The NICE methods of health technology evaluation: the case for change. National Institute for Health and Care Excellence. 2020. URL: https://www.bioindustry.org/static/e9c9093d-8a4b-4e1e-b01590df056d7f6a/BIA-response-to-the-NICE-methods-of-health-technology-evaluation-the-case-for-change.pdf [accessed 2022-09-26]
- Kickbusch I, Piselli D, Agrawal A, Balicer R, Banner O, Adelhardt M, Secretariat of the Lancet and Financial Times Commission. The Lancet and Financial Times Commission on governing health futures 2030: growing up in a digital world. Lancet 2021 Nov 06;398(10312):1727-1776. [CrossRef] [Medline]
- Catalonia, Spain. World Health Organization. 2018. URL: https://cdn.who.int/media/docs/librariesprovider2/country-sites/catalonia.pdf?sfvrsn=d000806_3&download=true [accessed 2022-09-26]
- Spain: Health care and long-term care systems. European Commission. 2016. URL: https://ec.europa.eu/info/sites/default/files/file_import/joint-report_es_en_2.pdf [accessed 2022-09-26]
- Global strategy on digital health 2020-2025. World Health Organization. 2021. URL: https://www.who.int/docs/default-source/documents/gs4dhdaa2a9f352b0445bafbc79ca799dce4d.pdf [accessed 2022-09-26]
- eHealth: Digital health and care. European Commission. 2019. URL: https://pbu2020.eu/files/uploads/annual%20conference_2019/EB0219277ENN.en.pdf [accessed 2022-09-26]
- Reeves JJ, Ayers JW, Longhurst CA. Telehealth in the COVID-19 era: A balancing act to avoid harm. J Med Internet Res 2021 Feb 01;23(2):e24785 [FREE Full text] [CrossRef] [Medline]
- O'Rourke B, Oortwijn W, Schuller T, International Joint Task Group. The new definition of health technology assessment: A milestone in international collaboration. Int J Technol Assess Health Care 2020 Jun;36(3):187-190. [CrossRef] [Medline]
- EUnetHTA JA2 WP8 deliverable: HTA Core Model version 3. European Network for Health Technology Assessment. 2016. URL: https://www.eunethta.eu/wp-content/uploads/2018/03/HTACoreModel3.0-1.pdf [accessed 2022-04-07]
- Puñal-Riobóo J, Álvarez EB, Varela LL, Muñoz MAC, Merino GA, Pérez RU, et al. Guía para la elaboración y adaptación de informes rápidos de evaluación de tecnologías sanitarias. Ministerio de Sanidad, Servicios Sociales e Igualdad. 2016. URL: https://extranet.sergas.es/catpb/Docs/gal/Publicaciones/Docs/avalia-t/PDF-2496-ga.pdf [accessed 2022-09-26]
- Hussain MS, Silvera-Tawil D, Farr-Wharton G. Technology assessment framework for precision health applications. Int J Technol Assess Health Care 2021 May 26;37(1):e67. [CrossRef] [Medline]
- Evidence standards framework for digital health technologies. National Institute for Health and Care Excellence. 2018. URL: https://www.nice.org.uk/corporate/ecd7/resources/evidence-standards-framework-for-digital-health-technologies-pdf-1124017457605 [accessed 2022-09-26]
- Haverinen J, Keränen N, Falkenbach P, Maijala A, Kolehmainen T, Reponen J. Digi-HTA: Health technology assessment framework for digital healthcare services. Finnish Journal of eHealth and eWelfare 2019 Nov 02;11(4):326-341. [CrossRef]
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med 2018 Oct 02;169(7):467-473 [FREE Full text] [CrossRef] [Medline]
- Ayiku L, Hudson T, Glover S, Walsh N, Adams R, Deane J, et al. The NICE MEDLINE and Embase (Ovid) health apps search filters: development of validated filters to retrieve evidence about health apps. Int J Technol Assess Health Care 2020 Oct 27;37:e16. [CrossRef] [Medline]
- Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev 2017 Dec 06;6(1):245 [FREE Full text] [CrossRef] [Medline]
- Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Healthc 2015 Sep;13(3):179-187. [CrossRef] [Medline]
- EndNote 20. Clarivate. URL: https://endnote.com [accessed 2022-09-26]
- Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc 2016 Jul;104(3):240-243 [FREE Full text] [CrossRef] [Medline]
- CASP Qualitative Checklist 2018. Critical Appraisal Skills Programme. URL: https://casp-uk.net/casp-tools-checklists/ [accessed 2022-04-07]
- ATLAS.ti 22. Scientific Software Development GmbH. URL: https://atlasti.com [accessed 2022-09-26]
- Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions (v6.3). London, United Kingdom: The Cochrane Collaboration; 2022.
- Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 2008 Jul 10;8:45 [FREE Full text] [CrossRef] [Medline]
- Moshi MR, Tooher R, Merlin T. Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review. Int J Technol Assess Health Care 2018 Jan;34(5):464-475. [CrossRef] [Medline]
- von Huben A, Howell M, Howard K, Carrello J, Norris S. Health technology assessment for digital technologies that manage chronic disease: a systematic review. Int J Technol Assess Health Care 2021 May 26;37(1):e66. [CrossRef] [Medline]
|AI: artificial intelligence|
|EUnetHTA: European Network for Health Technology Assessment|
|FinCCHTA: Finnish Coordinating Center for Health Technology Assessment|
|HTA: health technology assessment|
|mHealth: mobile health|
|NICE: National Institute for Health and Care Excellence|
|PICO-D: Population, Intervention, Comparator, Outcomes, Design|
|PICo-D: Population, Phenomenon of Interest, Context, and Design|
|PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews|
|WHO: World Health Organization|
Edited by T Leung; submitted 27.05.22; peer-reviewed by M Aymerich, H Unsworth; comments to author 10.07.22; revised version received 15.07.22; accepted 31.07.22; published 11.10.22Copyright
©Joan Segur-Ferrer, Carolina Moltó-Puigmartí, Roland Pastells-Peiró, Rosa Maria Vivanco-Hidalgo. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 11.10.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.