This paper is in the following e-collection/theme issue:

Non-randomized Protocols and Methods (ehealth) (567) Design and Formative Evaluation of Mobile Apps (692) Mobile Apps for Pregnancy and Parenting Education (43) Mobile Health and Apps for Maternal and Child Health (149) Perinatal Education for Parents (219)

Published on in Vol 10, No 1 (2021): January

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/19073, first published .
Mobile Health Intervention in the Maternal Care Pathway: Protocol for the Impact Evaluation of hAPPyMamma

Mobile Health Intervention in the Maternal Care Pathway: Protocol for the Impact Evaluation of hAPPyMamma

Mobile Health Intervention in the Maternal Care Pathway: Protocol for the Impact Evaluation of hAPPyMamma

Authors of this article:

Manila Bonciani1 Author Orcid Image ;   Sabina De Rosis1 Author Orcid Image ;   Milena Vainieri1 Author Orcid Image
Article Authors Cited by (12) Tweetations (1) Metrics

    Protocol

    Management and Healthcare Laboratory, Institute of Management and Department EMbeDS, Sant’Anna School of Advanced Studies, Pisa, Italy

    *all authors contributed equally

    Corresponding Author:

    Manila Bonciani, MSc, PhD

    Management and Healthcare Laboratory

    Institute of Management and Department EMbeDS

    Sant’Anna School of Advanced Studies

    Piazza Martiri della Libertà, 33

    Pisa, 56127

    Italy

    Phone: 39 3462405234

    Email: m.bonciani@santannapisa.it


    Background: Mobile health (mHealth) has great potential to both improve the quality and efficiency of care and increase health literacy and empowerment of patient users. There are several studies related to the introduction of mHealth tools for supporting pregnancy and the postnatal period, with promising but not yet rigorously evaluated impacts. This article presents the protocol for evaluating an mHealth intervention (hAPPyMamma) applied in the maternal and child care pathway of a high-income country (in a pilot area of Tuscany Region, Italy).

    Objective: The protocol describes hAPPyMamma and the methods for evaluating its impact, including the points of view of women and practitioners. The research hypothesis is that the use of hAPPyMamma will facilitate a more appropriate use of available services, a better care experience for women, and an improvement in the maternal competencies of the women using the app compared to the control group. The protocol also includes analysis of the organizational impact of the introduction of hAPPyMamma in the maternal pathway.

    Methods: A pre-post quasiexperimental design with a control group is used to undertake difference-in-differences analysis for assessing the impact of the mHealth intervention from the mothers’ points of view. The outcome measures are improvement of maternal health literacy and empowerment as well as experience in the maternal care pathway of the control and intervention groups of sampled mothers. The organizational impact is evaluated through a quantitative and qualitative survey addressing professionals and managers of the maternal care pathway involved in the intervention.

    Results: Following study recruitment, 177 women were enrolled in the control group and 150 in the intervention group, with a participation rate of 97%-98%. The response rate was higher in the control group than in the intervention group (96% vs 67%), though the intervention group had less respondent loss at the postintervention survey (10% compared to 33% of the control group). Data collection from the women was completed in April 2018, while that from professionals and managers is underway.

    Conclusions: The study helps consolidate evidence of the utility of mHealth interventions for maternal and child care in developed countries. This paper presents a protocol for analyzing the potential role of hAPPyMamma as an effective mHealth tool for improving the maternal care pathway at individual and organizational levels and consequently helps to understand whether and how to scale up this intervention, with local, national, and international scopes of application.

    International Registered Report Identifier (IRRID): DERR1-10.2196/19073

    JMIR Res Protoc 2021;10(1):e19073

    doi:10.2196/19073

    Keywords

    mHealth; maternal care pathway; impact evaluation; quasiexperimental study 


    Background

    Mobile health (mHealth) can play a disruptive role in transforming health promotion and health care service provision. mHealth refers to health-related practice supported by mobile and wireless devices, such as mobile phones, smartphones, and tablets, including mobile apps [1]. The growing spread of mobile devices has pushed the use of apps providing digital services, including for health care.

    mHealth has great potential to both improve quality and efficiency of care [2,3] and to increase health literacy and empowerment of patient users [4]. Using mHealth apps, people can manage their health and well-being more actively and consciously [3]. Due to its characteristic of ubiquity and the possibility of personalization, it is expected to be a powerful tool for patient-centered care.

    Nevertheless, there is contradictory evidence about the impacts of mHealth interventions on health promotion practice and health outcomes [5-8]. There are several studies focusing, in particular, on the introduction of mHealth tools for supporting pregnancy and the postnatal period, with promising but not yet rigorously evaluated impacts [9]. The evidence for effectiveness in behavioral change is also inconsistent, with both ineffective [10] and effective interventions [11,12] targeted at pregnant women and mothers. Furthermore, there is evidence of demonstrated positive outcomes from mHealth tools for pregnant women and future mothers, but also of the difficulties related to the routine integration of mHealth tools into established prenatal and newborn health services [13]. Nonetheless, it is worth pointing out that most of the research studies on mHealth interventions in the field of maternal, neonatal, and newborn care have been undertaken in low- and middle-income countries [14-29].

    This protocol presents an mHealth intervention for maternal and child care in a high-income country, including the methods adopted for evaluating its impact at individual and organizational levels.

    Study and Policy Context Concerning mHealth and the Maternal Care Pathway

    The context of this study is the region of Tuscany (Italy), which shows characteristics of eHealth diffusion in line with both the national and wider European contexts [30,31].

    Within the framework of the Italian public health care system ensuring universal health care coverage, maternal care is guaranteed for all women free of charge as an essential level of care [32,33]. Although this includes services provided by hospitals and family care centers along the entire maternal journey until the postpartum period, the majority of women prefer to be supported by a private gynecologist during pregnancy [34]; this may limit communication of the publicly available community services offered to pregnant women and new mothers. To standardize the prenatal visits and treatments within its territory, the Tuscany Region provides women with a pregnancy booklet, delivered by a midwife at the family care center. Despite such efforts to strengthen the maternal care pathway, some critical issues remain unresolved in Tuscany [35-38]. A first problem concerns the lack of coordination between services, especially where the maternal care pathway requires integration between local health authorities (LHAs) and teaching hospitals. A second issue is related to the communication channels: Tuscan women clearly expressed the preference to be informed using information and communication technology (ICT) such as text messages or emails [35]. A third issue concerns poorer health care service use by specific categories of women, such as foreign women and those with a low level of education [36]. Due to their weaker health literacy, these groups are often the least likely to exercise choice and to have a direct and appropriate relationship with health care services and consequently, equal access to antenatal and postnatal care [39]. A final issue regards infant vaccination coverage, which in 2016 was <90% for measles [40].

    To address these weaknesses and gaps, Tuscany financed and promoted the development and pilot study of the mobile app hAPPyMamma in an LHA. hAPPyMamma was designed as a supportive tool for women and professionals who can provide women with information about the maternal care pathway and its services, also targeted at specific categories (such as low-income women). This may increase the opportunity for contact and interaction as well as women’s self-management and can facilitate disadvantaged groups in accessing and using services [41].

    Design and Implementation of the Mobile App hAPPymamma

    The mobile app offers different functionalities. From the home page, gestational age (during pregnancy) or newborn age (in the postpartum period) are addressed with personalized messages (Figure 1). It includes a digital translation of the pregnancy booklet and infant vaccination calendar into the planner within the app (Figure 2), including alert mechanisms to notify women about visits and diagnostic tests to reserve or, if reserved, to attend.

    ‎
    Figure 1. hAPPyMamma home page: identification of gestational age (during pregnancy) or newborn age (in the postpartum period), specific week-by-week messages, and memos for upcoming appointments.
    View this figure
    ‎
    Figure 2. hAPPyMamma agenda: digitization of the pregnancy booklet and infant vaccination calendar in the app planner, with different visualization options and alert mechanisms.
    View this figure

    Women can digitally book, directly from the app, the 3 obstetric ultrasounds (the first visit in pregnancy and the postpartum visit; Figure 3), which are the mandatory touchpoints of the maternal care pathway in Tuscany.

    hAPPyMamma contains information on health promotion and prevention as well as on the health care services concerning pregnancy, childbirth, and the postpartum period, divided into thematic sections (Figure 4). It also proactively shows information through pop-up messages based on data reported by women (ie, smokers will receive pop-up messages on the specific topic of smoking during pregnancy).

    hAPPyMamma includes a section with logistic details of primary care and hospital services along the maternity care pathway, with a georeferencing system (Figure 5). It first presents services and providers related to the woman’s residence area. Additionally, hAPPyMamma is integrated with the regional mobile app that allows access to personal health records developed by the Tuscan Regional Health System. Finally, women using the app can provide direct feedback and answer questionnaires proposed by the app to evaluate their experience in the maternity care pathway.

    The design of hAPPyMamma was user-driven, as described, and its development was shared with the professionals involved at both the primary care and hospital levels and in both the maternal and child care areas. It involved the researchers of the Sant’Anna School, who facilitated the app design process and evaluated the results of this innovation.

    ‎
    Figure 3. hAPPyMamma e-booking of visits and tests, integration into the app planner, and synchronization with the local health authority booking system.
    View this figure
    ‎
    Figure 4. hAPPyMamma information repository: provision of professionally validated information (in a frequently asked questions format) on health promotion, prevention, and health care services concerning pregnancy, childbirth, and the postpartum period, with information proactively suggested via pop-up messages.
    View this figure
    ‎
    Figure 5. hAPPYMamma facilities repository: information on family care centers and delivery hospitals, with logistic details, services provided, and georeferencing system embedded.
    View this figure

    Objectives

    The aim of the study reported in this protocol is to evaluate whether and to what extent the mobile app hAPPyMamma is able to increase maternal health literacy (MHL) and empowerment of women as well as access to and utilization of health care services during the maternal care pathway. Our hypothesis is that implementation of the mHealth solution will bring a positive impact on the maternal care pathway, in terms of more appropriate use of the available services, a better experience for women, and an improvement in the maternal competencies of women using hAPPyMamma. To meet these objectives, the study proposes to compare prestudy and poststudy differences in outcomes for the control group versus the intervention group using hAPPyMamma.

    Moreover, the protocol includes an analysis of the use of hAPPyMamma for the intervention group and an analysis of the organizational impact of the introduction of the app in the maternal pathway from the professionals’ perspectives.


    Impact evaluation of the hAPPyMamma mHealth intervention is provided in detail in this section.

    Study Design

    In order to assess the impact of the use of hAPPyMamma in the maternal care pathway, this study uses a pre-post quasiexperimental design that compares 2 groups: women who use the app (intervention group) and women who do not use the app (control group). The manipulation of the independent variable (app use) is done by recruiting the 2 groups consequentially: control group before the app was available on the app stores and the experimental group when the app was introduced into the maternal care pathway. The ethical committee of the LHA where the mHealth intervention is implemented approved the research protocol (Authorization n. 42379 signed by the ethical committee on July 13, 2016 and registered with the approval number n. 0133972 on August 3, 2016).

    The choice of not randomizing the samples using and not using the mobile app is in line with the ethical principles. Indeed, the mobile app can represent a potential tool to improve the quality of the maternity care pathway, and the randomization of women could have brought unfair advantages to women who use the app compared to the others. Therefore, the choice of a quasiexperimental study design also addresses this ethical principle.

    The data concerning the measures of maternal health literacy and empowerment are collected prospectively through web-based prestudy and poststudy surveys, enabling difference-in-differences analysis to assess the impact of the mHealth intervention [42]. In particular, there are 2 data collection moments through the web platform. The first data collection (prestudy survey) is carried out at the beginning of the maternal care pathway, when women receive the pregnancy booklet, corresponding to the beginning of the pregnancy. The second data collection (postsurvey) is implemented around 6 months after childbirth in the postpartum period, when data on women’s experiences in the maternal care pathway are also collected. This second survey questionnaire also contains a section reserved for the intervention group on their experience with hAPPyMamma use.

    The organizational impact will be evaluated through a quantitative and qualitative survey and in-depth interviews addressing professionals and managers of the maternal care pathway in the participating LHA and teaching hospital. Both data collection methods focus on the perceived changes the practitioners identify in their job activities due to the mHealth intervention. The survey uses a web questionnaire with close-ended and open-ended questions, in order to measure and assess practitioners’ perceptions and opinions on the hAPPyMamma impact through rating questions and narratives. The survey results are discussed with some practitioners during the in-depth interviews. Considering that the organizational impact can be better evaluated through a midterm evaluation, this component of the study protocol is planned to be implemented after implementation of the data collection with mothers.

    Study Population and Sample Size

    The sample size for the pre-post survey addressing mothers was estimated with respect to differences concerning some key measures considered within the study, namely MHL level and other experience measures such as access to maternal care services and satisfaction with the maternal care received. We determined the sample size required to detect a 10% difference between the 2 groups concerning the key measures considered within the study, when this difference genuinely exists in the populations of the mothers, with a power of 80% and an alpha error of 5%. With these parameters, a sample size of 300 pregnant women, divided equally between the control and intervention groups, was considered appropriate for a study population of around 2000 births in the pilot LHA yearly.

    The quantitative and qualitative survey focusing on the organizational impact of the mHealth intervention uses a convenience sample of professionals and managers working in the LHA and teaching hospital of the study area. All midwives, obstetricians, pediatric doctors, and other professionals of the maternal care pathway are invited to fill in the web questionnaire. The results of this data collection are discussed with 15 practitioners during in-depth interviews.

    Eligibility Criteria

    All women receiving the pregnancy booklet in the family care centers of the 3 districts of the participating pilot LHAs during the recruitment period were included. The only exclusion criterion is not speaking Italian, because the app does not have the multilanguage function activated during the experimentation phase. Finally, women without smartphones are not able to use hAPPyMamma and thus are unable participate in the intervention group of the study.

    Outcomes

    The study objectives are addressed in the following ways.

    Difference in Improvements in MHL and Empowerment Between the Control and Intervention Groups

    Key outcomes of the study concern MHL and empowerment of women involved in the maternity pathway, which may be particularly improved in the intervention group thanks to the use of the mobile app hAPPyMamma. In order to measure these outcomes, some items of internationally validated tools are used [43-45]. Table 1 shows the dimensions included in the MHL construct: critical, functional, self-efficacy, and social capital. The first 3 dimensions of MHL focus mainly on underlining the competence of mothers for promoting and protecting their health and that of their children, as well as orienting among health information and services. The fourth dimension concerns social capital that can be considered both a demonstration and a consequence of MHL. In order to define the MHL items for the pre-post questionnaire, 2 different researchers translated the items from the international scales, compared and discussed the 2 different translations, and finally identified a shared version of the Italian items.

    Empowerment is evaluated in our study in terms of self-efficacy on breastfeeding and duration of breastfeeding (total and exclusive). These measures that are internationally considered a proxy indicator of mother’s empowerment can be positively affected by the more direct and easy access to information on breastfeeding through the app.

    Table 1. Dimensions included in the pre-post survey questionnaires addressing sampled women in the maternal care pathway.
    DimensionsDescriptionControl groupIntervention group
    Prestudy surveyPoststudy surveyPrestudy surveyPoststudy survey
    Maternity pathway expectationsExpectations about pregnancy, delivery, and the postpartum period (1 item)XN/AaXN/A
    Maternal health literacy
    CriticalSearch of different sources of information, check of validity and reliability of information, use of information to make decisions on own health (7 items)XXXX
    FunctionalAbility to understand health information, difficulty in reading and interpreting health information materials, self-confidence to fill in modules with health information (4 items)XXXX
    Self-efficacySelf-confidence to follow health indications, self-confidence to be autonomous in taking care of own child, capability to identify different positive solutions facing obstacles (4 items)XXXX
    Social capitalSupport from someone in case of concerns or doubts on own condition, continuity of social life after pregnancy (2 items)XXXX
    Intention to breastfeedExpectations of breastfeeding and its duration (2 items)XN/AXN/A
    Breastfeeding - empowermentTotal breastfeeding, exclusive breastfeeding, and its duration (3 items)N/AXN/AX
    Experience in the maternal care pathwayBooking of exams during pregnancy, awareness of prenatal diagnostic tests, sources of information on prenatal diagnosis tests, attendance at antenatal classes, difficulties in accessing health services during pregnancy, awareness of labor and delivery, use of health services after delivery, orientation difficulties in maternal care pathway, satisfaction with the maternal care received, suggestions on maternal care pathway improvement (14 items)N/AXN/AX
    Type of health care during pregnancy and deliveryProfessionals and services involved in the pregnancy follow-up, number of visits and ultrasounds, characteristics of pregnancy and delivery (8 items)N/AXN/AX
    Interest in ICTb servicesInformation and reminders for visits and exams, antenatal classes, and postpartum services; information on healthy lifestyle during pregnancy; health information records for pregnancy, delivery, and the postpartum period; communication with family pediatrician—by different ICT tools (8 items)N/AXN/AX
    Experience with hAPPyMamma useDuration and frequency of use, habits in smartphone and internet use, comparison with other apps concerning maternal care, utility of different app functionalities, suggestions to improve the app use (11 items)N/AN/AN/AX
    Usability of hAPPyMammaLearnability, memorability, understandability, attractiveness, errors, efficiency, evaluation of quality, satisfaction in terms of willingness to recommend
    (15 items)    		N/AN/AN/AX
    Sociodemographic characteristicsAge, citizenship, education level, employment status, gestational age, number of children (6 items)XN/AXN/A

    aN/A: not applicable.

    bICT: information and communication technology.

    Control and Intervention Group Experiences in the Maternal Care Pathway

    The experience of women in the maternal care pathway is measured through a selection of the questions included in the validated questionnaires that have been used periodically in the Tuscany Region to evaluate the users’ perspective in the maternal care pathway [35,36]. As shown in the Table 1, this dimension of experience concerns the use of health services during the maternal care pathway, such as prenatal care, antenatal classes, and postpartum care, and they explore particularly the difficulties in access to and orientation among the health services of the maternal care pathway. The questions focus also on the perceived quality of the maternal care pathway and the willingness to recommend its services to friends and family members. The use of the same survey tool ensures continuity in the research approach and the possibility to assess the trend in the outcomes measured.

    Intervention Group’s Use of hAPPyMamma

    A specific section concerning the experience of the hAPPyMamma use and its usability is introduced in the post-survey questionnaire for the intervention group only (Table 1). The first dimension explores mainly the duration and frequency of the use of hAPPyMamma, perceived utility of the hAPPyMamma functionalities, and opinions of mothers concerning the comparison of hAPPyMamma with other apps related to maternity. The usability includes the principal criteria included in the scales used at the international level [46-49], such as learnability, memorability, understandability, attractiveness, errors, efficiency, quality, and satisfaction. Moreover, some questions focusing on the mothers’ interest for ICT services are introduced in the post-survey questionnaire; these investigate mothers’ possible interest in receiving—by email, message, or app—information on and reminders for visits and exams, antenatal classes, and postpartum services; information on a healthy lifestyle during pregnancy; health information records for pregnancy, delivery, and the postpartum period; and communication with the family pediatrician. They are submitted both to intervention and control groups in order to also assess the interest of mothers who are not using the mHealth solution. The intervention group has an additional option for answering these questions, namely that they already receive information and reminders via hAPPyMamma.

    Organizational Impact of the Introduction of the hAPPyMamma App in the Maternity Pathway From the Professionals’ Perspectives

    The organizational impact is assessed through a web survey and in-depth interviews with the professionals and managers of the maternal care pathway (midwives, obstetricians, neonatal doctors), focusing on the aspects of the experimental use of the mobile app affecting the health care process. In particular, the web survey and interviews explore their expectations before mHealth solution implementation, their perceived difficulties and worries, feelings of uncertainty, fear of replacement, desire to innovate, and the resistance to change that may characterize the opinions of the involved professionals. The descriptive analysis of the questionnaire allows elucidation of the experience of professionals and managers regarding the mHealth intervention. These results represent the basis of the discussion with some professionals and managers during the in-depth interviews. The implementation of the web survey and in-depth interviews after the end of the experimentation allow description of the impact of hAPPyMamma on health care services and practices from the points of view of professionals, with a certain period of distancing from the mHealth intervention to better appreciate its organizational impact.

    Statistical and Qualitative Data Analysis

    Descriptive statistics will be used to characterize the study population to give an overview of the 2 groups concerning demographic characteristics and use of services during pregnancy, birth, and the postpartum period.

    The impact of the app hAPPyMamma on the MHL will be evaluated through a difference-in-differences analysis, which allows measuring the difference in changes prestudy and poststudy between the 2 groups. In particular, the panel data of the study are used to measure the differences between the treatment and control group in the changes in the outcome variable (MHL) that occur over time. In particular, we will calculate the effect of the use of hAPPyMamma (ie, an explanatory variable or an independent variable) on MHL (ie, a response variable or dependent variable) by comparing the average change over time in the outcome variable for the treatment group and the average change over time for the control group. We will also verify the impact of the use of hAPPyMamma on the subdimensions of MHL.

    Other multivariate analyses of variance will be carried out in order to evaluate empowerment related to breastfeeding results and the experience of women in the maternity pathway, specifically comparing the control and intervention groups.

    Descriptive statistics and multivariate models will also be used to analyze the experience with hAPPyMamma use and its usability.

    All statistical analyses are performed using SAS and Stata software.

    Descriptive statistics will be performed for the web questionnaires completed by professionals and managers. These results will be presented to key practitioners involved in the in-depth interviews. Qualitative analysis of the in-depth interviews will be performed using QSR NVivo software. After importing the narrative answers to the web questionnaire and the transcriptions of the in-depth interviews, data coding will be implemented by 2 researchers. Content analysis will be carried out in order to identify emerging themes and patterns of the perceptions of professionals on the organizational impact of hAPPyMamma in the maternal care pathway [50].


    Recruitment of the 2 samples of mothers (control and intervention groups) was carried out in sequence. Data collection with the control group started at the beginning of 2017. In May 2017, hAPPyMamma was made available on app stores, and recruitment of the intervention group started. Both groups were recruited using the same procedures. At the time of receiving the pregnancy booklet, women were informed about the study by midwives. Women who decided to participate signed a consent form, leaving their email address. For the control group, we sent an invitation by email to fill in the first web questionnaire at the beginning of the maternity pathway, while the intervention group received the invitation concerning the first web questionnaire directly from the app. The invitation for the second questionnaire was sent by email to both groups, with the intervention group also having the possibility to access the web questionnaire from the app.

    The participation rate was high for both the control and intervention groups (around 97%-98%), since the consensus for attending the study was given by 177 women in the former group and by 150 women in the latter group. The response rate for the first questionnaire was different between the 2 groups: 96% (170/177) in the control group and 67% (100/150) in the intervention group. The difference in respondent loss in the follow-up questionnaires was reversed: 33% (56/170) in the control group and only 10% (10/100) in the intervention group. Data collection was completed in April 2018.

    Data analysis as well as data collection with professionals and managers are currently underway.


    This paper provides the protocol to evaluate the implementation of an mHealth intervention and its impact at individual and organizational levels in terms of improvement in maternal health literacy, mothers’ empowerment, and access to and utilization of health care services in the maternal care pathway. We are collecting data to describe the experimental use of the app hAPPyMamma and report on benefits for the mothers using the app. This study is innovative in the Italian context and compared with other interventions worldwide. The mHealth intervention has been realized thanks to collaboration between university researchers and health professionals in maternal care and is promoted by the Regional Health Authority with the aim of improving the quality of the maternal care pathway. Therefore, the study results assume an institutional perspective and will provide insights on the impact of hAPPyMamma use from the organizational as well as the user perspectives and on the perceptions on the provision of several services in the maternal care pathway through this mHealth channel.

    Interviews with key professionals in the maternal care pathway will help to deeply understand their point of view alongside insights emerging from the quantitative and qualitative web survey. This will contribute to identifying and explaining factors positively and negatively affecting the implementation and deployment of hAPPyMamma from their perspectives and those that may facilitate or inhibit the normalization of the innovative tool within the maternal care pathway.

    The findings of this study will be relevant for the academic community as well as for policy makers and practitioners. First, there is scarce empirical evidence of the real potential of mHealth in improving women’s access to care, their literacy and self-management skills, and quality of services along the maternal pathway [9]. This is particularly true for western countries because the literature focuses on the impact of technologies in developing countries [14-29]. In high-income countries, technology-supported interventions targeted at pregnant women and new mothers are often aimed at improving their lifestyle-related behaviors [9,51]. Conversely, the current critical circumstances that impose social distancing and limit physical access to care have highlighted the need for evidence-based technologies to be introduced to support digital and at-distance health care services in a time of crisis, which could be maintained in normal times. This is the second key point that supports the need for evidence on the effectiveness of mHealth services for women in the maternal pathway in developed countries. Service innovation is urgently needed in health care [52], and hAPPyMamma is an innovative way of providing women-centered services along a pathway, also allowing the evaluation of several different outcomes.

    There are several strengths in the design of this study. Its methodological approach as a quasiexperimental study allows overcoming the limitations of observational studies in measuring the effectiveness of interventions and their impact at the individual and organizational levels [53,54]. It is an appropriate method for evaluating policies or interventions, such as hAPPyMamma, collecting data before the recipients are exposed to policy or intervention activities [55]. The results of the study contribute to verifying the possibilities and potential of the scaling up of the mHealth intervention.

    The study faces some potential methodological and practical challenges. The nonrandomization of the sample, which is an important aspect from an ethical point of view, represents a weakness of the study. Indeed, the noncontemporaneous recruitment of the intervention and control groups does not allow excluding the possibility of influencing factors due to environmental or organizational context [54,56].

    Moreover, the differences in response rate and loss to follow-up among maternal samples have to be taken into account, and data analysis will verify if these affect the results [57].

    In conclusion, this study contributes to defining the potential role of the mHealth intervention hAPPyMamma in the maternal care pathway. The findings of this study could provide valuable insights on the benefits of hAPPyMamma use for women’s experiences in maternal care pathways. Therefore, this study could significantly support analysis to understand if scaling up hAPPyMamma implementation from the pilot area to the Tuscany Region, as well as to the entire country, would be beneficial. As anticipated, the findings that will result from the evaluation of this mHealth intervention will also provide useful insights for supporting the introduction of mobile-based innovations in maternal and newborn care pathways in other (developed as well as developing) countries.

    Acknowledgments

    We thank the CEO, Board of Directors, and staff of the local health unit and teaching hospital involved in the study, who supported the implementation of the mHealth intervention. We thank the members of the Maternal Care Pathway Committee of the local health unit involved in the study, in particular Patrizia Scida, Marta Lupetti, Massimo Srebot, Grazia Fazzino, Giuseppina Trimarchi, and Monica Funaioli, who collaborated on the implementation of the mobile app hAPPyMamma and its impact evaluation study. We thank all the women who participated in the study. Special thanks to Kendall Jamieson Gilmore for the support in the final revision for the English language. The study is part of the research activity plan of Laboratorio Management e Sanità, funded by the Tuscany Region Health Authority under the collaboration agreement signed in 2014 with the Sant’Anna School of Advanced Studies, renewed in 2016. The Tuscany Region Health Authority promoted the development of the mobile app hAPPyMamma and the relative study on its impact, but the Tuscany Region Health Authority was not involved in the study design, data collection, and analysis.

    Authors' Contributions

    MB coordinated the design and implementation of the mobile app hAPPyMamma. MB, SDR, and MV designed the study, formulated the research question, and elaborated the study protocol. MB and SDR coordinated the implementation of the study. MB, SDR, and MV wrote, revised, and approved the final manuscript.

    Conflicts of Interest

    None declared.

    1. World Health Organization. mHealth. New horizons for health through mobile technologies. Based on the findings of the second global survey on eHealth. In: Global Observatory for eHealth series - Volume 3. Geneva, Switzerland: WHO Press; 2011:1-111.
    2. Kreps G, Neuhauser L. E-health and health promotion. J Comput Mediat Commun 2010;15:527-529. [CrossRef]
    3. European Commission. Green Paper on mobile Health (mHealth). https://ec.europa.eu/. 2014.   URL: https://ec.europa.eu/digital-single-market/en/news/green-paper-mobile-health-mhealth [accessed 2020-12-30]
    4. Broderick J, Devine T, Langhans E, Lemerise A, Lier S, Harris L. Designing Health Literate Mobile Apps. Institute of Medicine, National Academy of Science.: National Academy of Sciences; 2014.   URL: https://nam.edu/wp-content/uploads/2015/06/HealthLiterateApps.pdf [accessed 2020-12-30]
    5. Kampmeijer R, Pavlova M, Tambor M, Golinowska S, Groot W. The use of e-health and m-health tools in health promotion and primary prevention among older adults: a systematic literature review. BMC Health Serv Res 2016 Sep 05;16 Suppl 5:290 [FREE Full text] [CrossRef] [Medline]
    6. Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, et al. The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 2013 Jan;10(1):e1001362 [FREE Full text] [CrossRef] [Medline]
    7. Spring B, Duncan JM, Janke EA, Kozak AT, McFadden HG, DeMott A, et al. Integrating Technology Into Standard Weight Loss Treatment. JAMA Intern Med 2013 Jan 28;173(2):105. [CrossRef]
    8. Volpp KG, Troxel AB, Mehta SJ, Norton L, Zhu J, Lim R, et al. Effect of Electronic Reminders, Financial Incentives, and Social Support on Outcomes After Myocardial Infarction. JAMA Intern Med 2017 Aug 01;177(8):1093. [CrossRef]
    9. Chen H, Chai Y, Dong L, Niu W, Zhang P. Effectiveness and Appropriateness of mHealth Interventions for Maternal and Child Health: Systematic Review. JMIR Mhealth Uhealth 2018 Jan 09;6(1):e7. [CrossRef]
    10. Katz KS, Rodan M, Milligan R, Tan S, Courtney L, Gantz M, et al. Efficacy of a randomized cell phone-based counseling intervention in postponing subsequent pregnancy among teen mothers. Matern Child Health J 2011 Dec;15 Suppl 1:S42-S53. [CrossRef] [Medline]
    11. Naughton F, Prevost A, Gilbert H, Sutton S. Randomized controlled trial evaluation of a tailored leaflet and SMS text message self-help intervention for pregnant smokers (MiQuit). Nicotine Tob Res 2012 May;14(5):569-577. [CrossRef] [Medline]
    12. Jareethum R, Titapant V, Chantra T, Sommai V, Chuenwattana P, Jirawan C. Satisfaction of healthy pregnant women receiving short message service via mobile phone for prenatal support: A randomized controlled trial. J Med Assoc Thai 2008 Apr;91(4):458-463. [Medline]
    13. Tamrat T, Kachnowski S. Special delivery: an analysis of mHealth in maternal and newborn health programs and their outcomes around the world. Matern Child Health J 2012 Jul;16(5):1092-1101. [CrossRef] [Medline]
    14. Modi D, Gopalan R, Shah S, Venkatraman S, Desai G, Desai S, et al. Development and formative evaluation of an innovative mHealth intervention for improving coverage of community-based maternal, newborn and child health services in rural areas of India. Glob Health Action 2015 Feb 16;8(1):26769 [FREE Full text] [CrossRef] [Medline]
    15. Modi D, Patel J, Desai S, Shah P. Accessing completeness of pregnancy, delivery, and death registration by Accredited Social Health Activists [ASHA] in an innovative mHealth project in the tribal areas of Gujarat: A cross-sectional study. J Postgrad Med 2016;62(3):170-172 [FREE Full text] [CrossRef] [Medline]
    16. Modi D, Desai S, Dave K, Shah S, Desai G, Dholakia N, et al. Cluster randomized trial of a mHealth intervention "ImTeCHO" to improve delivery of proven maternal, neonatal, and child care interventions through community-based Accredited Social Health Activists (ASHAs) by enhancing their motivation and strengthening supervision in tribal areas of Gujarat, India: study protocol for a randomized controlled trial. Trials 2017 Jun 09;18(1):270 [FREE Full text] [CrossRef] [Medline]
    17. Ag Ahmed MA, Gagnon M, Hamelin-Brabant L, Mbemba GIC, Alami H. A mixed methods systematic review of success factors of mhealth and telehealth for maternal health in Sub-Saharan Africa. Mhealth 2017 Jun 06;3:22-22 [FREE Full text] [CrossRef] [Medline]
    18. Nurmatov UB, Lee SH, Nwaru BI, Mukherjee M, Grant L, Pagliari C. The effectiveness of mHealth interventions for maternal, newborn and child health in low- and middle-income countries: Protocol for a systematic review and meta-analysis. J Glob Health 2014 Jun;4(1):010407 [FREE Full text] [CrossRef] [Medline]
    19. Nurmatov UB, Lee SH, Nwaru BI, Mukherjee M, Grant L, Pagliari C. The effectiveness of mHealth interventions for maternal, newborn and child health in low- and middle-income countries: Protocol for a systematic review and meta-analysis. J Glob Health 2014 Jun;4(1):010407 [FREE Full text] [CrossRef] [Medline]
    20. Ilozumba O, Abejirinde IO, Dieleman M, Bardají A, Broerse JEW, Van Belle S. Targeting strategies of mHealth interventions for maternal health in low and middle-income countries: a systematic review protocol. BMJ Open 2018 Feb 24;8(2):e019345 [FREE Full text] [CrossRef] [Medline]
    21. Sondaal S, Browne J, Amoakoh-Coleman M, Borgstein A, Miltenburg A, Verwijs M, et al. Assessing the Effect of mHealth Interventions in Improving Maternal and Neonatal Care in Low- and Middle-Income Countries: A Systematic Review. PLoS One 2016;11(5):e0154664 [FREE Full text] [CrossRef] [Medline]
    22. Al Dahdah M, Desgrées Du Loû A, Méadel C. Mobile health and maternal care: A winning combination for healthcare in the developing world? Health Policy and Technology 2015 Sep;4(3):225-231. [CrossRef]
    23. Colaci D, Chaudhri S, Vasan A. mHealth Interventions in Low-Income Countries to Address Maternal Health: A Systematic Review. Ann Glob Health 2016;82(5):922-935 [FREE Full text] [CrossRef] [Medline]
    24. Watterson J, Walsh J, Madeka I. Using mHealth to Improve Usage of Antenatal Care, Postnatal Care, and Immunization: A Systematic Review of the Literature. Biomed Res Int 2015;2015:153402 [FREE Full text] [CrossRef] [Medline]
    25. Atnafu A, Otto K, Herbst CH. The role of mHealth intervention on maternal and child health service delivery: findings from a randomized controlled field trial in rural Ethiopia. Mhealth 2017;3:39 [FREE Full text] [CrossRef] [Medline]
    26. White A, Crowther S, Lee S. Supporting rural midwifery practice using a mobile health (mHealth) intervention: a qualitative descriptive study. Rural Remote Health 2019 Aug;19(3):5294 [FREE Full text] [CrossRef] [Medline]
    27. Cho Y, Lee S, Islam SMS, Kim S. Theories Applied to m-Health Interventions for Behavior Change in Low- and Middle-Income Countries: A Systematic Review. Telemed J E Health 2018 Oct;24(10):727-741 [FREE Full text] [CrossRef] [Medline]
    28. Gholami M, Nasiripoor A, Maleki M. The Relation Between Social Determinant of Health with Access to Health Services in Gonbad kavoos. Community Health (SALĀMAT-I IJTIMĀĪ) 2016;3(1):54 [FREE Full text]
    29. Lee S, Begley CE, Morgan R, Chan W, Kim S. m-Health Policy Readiness and Enabling Factors: Comparisons of Sub-Saharan Africa and Organization for Economic Cooperation and Development Countries. Telemed J E Health 2018 Nov;24(11):908-921 [FREE Full text] [CrossRef] [Medline]
    30. De Rosis S, Barsanti S. E-patient (r)evolution: quando è il paziente a coinvolgere il medico. In: Proceedings of X Annual Meeting of SIM. Milan, Italy: Società Italiana Marketing; 2013.
    31. The National eHealth Information Strategy. National context, state of implementation and best practices. Italian Health Ministry. 2011.   URL: http://www.salute.gov.it/imgs/C_17_pubblicazioni_1653_allegato.pdf [accessed 2020-12-30]
    32. Decreto 1SETTEMBRE1. Aggiornamento del decreto ministeriale 6 marzo 1995 concernente l'aggiornamento del decreto ministeriale 14 aprile 1984 recante protocolli di accesso agli esami di laboratorio e di diagnostica strumentale per le donne in stato di gravidanza ed a tutela della maternità. Italian Ministry of Health.: Gazzetta Ufficiale 20 ottobre 1998, n. 245; 1998.   URL: http://www.salute.gov.it/imgs/C_17_normativa_1653_allegato.pdf [accessed 2020-12-30]
    33. Accordo 1D2. Accordo, ai sensi dell'articolo 9 del decreto legislativo 28 agosto 1997, n. 281, tra il Governo, le regioni e le province autonome di Trento e Bolzano, le province, i comuni e le comunità montane sul documento concernente «Linee di indirizzo per la promozione ed il miglioramento della qualità, della sicurezza e dell'appropriatezza degli interventi assistenziali nel percorso nascita e per la riduzione del taglio cesareo». (Rep. atti n. 137/CU) (11A00319). Italian Ministry of Health. G.U. Serie Generale, n. 13 del 18 gennaio 2011; 2011.   URL: https://www.gazzettaufficiale.it/eli/id/2011/01/18/11A00319/sgo [accessed 2020-12-30]
    34. Lauria L, Lamberti A, Buoncristiano M, Bonciani M, Andreozzi S. Percorso nascita: promozione e valutazione della qualità di modelli operativi. Le indagini del 2008-2009 e del 2010-2011. Rapporti ISTISAN 12/39. Roma: Istituto Superiore di Sanità; 2012.   URL: https://www.dors.it/documentazione/testo/201301/12_39_web.pdf [accessed 2020-12-30]
    35. Nuti S, Murante A. Il Percorso Nascita in Toscana: l'esperienza delle donne. Report 2012-2013. Laboratorio Management e Sanità 2013.
    36. Murante A, Nuti S, Matarrese D. Report of the maternity pathway. Firenze, Italy: Edizioni Polistampa; 2015.
    37. Bonciani M, Lupi B, Nuti S. Performance evaluation in healthcare: the experience of maternity pathway from Tuscany to the Italian network of regions. Ital J Pediatr 2014 Aug 11;40(S1) [FREE Full text] [CrossRef]
    38. Nuti S, Bonini A, Murante AM, Vainieri M. Performance assessment in the maternity pathway in Tuscany region. Health Serv Manage Res 2009 Aug;22(3):115-121. [CrossRef] [Medline]
    39. Bonciani M, Corazza I, Lupi B, De RS. How to improve maternal pathway for migrant women: insights for retention strategies from Tuscany Region. Micro&Macro Marketing 2019 Dec;3:585-607 [FREE Full text] [CrossRef]
    40. Nuti S, Vainieri M, Cerasuolo M. Il sistema di valutazione della performance della sanità toscana. Report 2016. Pisa, Italy: Pacini Editori; 2017.
    41. Bonciani M, De Rosis S, Vainieri M. Promoting the women’s health literacy and their access to maternal-care-pathway in Italy through an integrated mHealth intervention. Int J Integr Care 2018 Oct 23;18(s2):380 [FREE Full text] [CrossRef]
    42. Imbens GW, Wooldridge JM. Recent Developments in the Econometrics of Program Evaluation. Journal of Economic Literature 2009 Mar 01;47(1):5-86. [CrossRef]
    43. Guttersrud O, Naigaga MDAS, Pettersen KS. Measuring Maternal Health Literacy in Adolescents Attending Antenatal Care in Uganda: Exploring the Dimensionality of the Health Literacy Concept Studying a Composite Scale. J Nurs Meas 2015 Aug 01;23(2):50-66. [CrossRef]
    44. Naigaga MD, Guttersrud O, Pettersen KS. Measuring maternal health literacy in adolescents attending antenatal care in a developing country - the impact of selected demographic characteristics. J Clin Nurs 2015 Oct 12;24(17-18):2402-2409. [CrossRef] [Medline]
    45. Dennis C, Heaman M, Mossman M. Psychometric testing of the breastfeeding self-efficacy scale-short form among adolescents. J Adolesc Health 2011 Oct;49(3):265-271. [CrossRef] [Medline]
    46. Monkman H, Kushniruk A. A health literacy and usability heuristic evaluation of a mobile consumer health application. Stud Health Technol Inform 2013;192:724-728. [Medline]
    47. Ryu YS, Smith-Jackson TL. Reliability and Validity of the Mobile Phone Usability Questionnaire (MPUQ). Journal of Usability Studies 2006;2:1-53.
    48. Nayebi F, Desharnais J, Abran A. The state of the art of mobile application usability evaluation. 2012 Presented at: 25th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE); April 29-May 2, 2012; Montreal, Canada. [CrossRef]
    49. ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs). 1998.   URL: https://www.iso.org/standard/16883.html [accessed 2020-12-30]
    50. Bazeley P, Jackson K. Qualitative data analysis with NVIVO. London, England: SAGE Publishing; 2013.
    51. Mertens L, Braeken MA, Bogaerts A. Effect of Lifestyle Coaching Including Telemonitoring and Telecoaching on Gestational Weight Gain and Postnatal Weight Loss: A Systematic Review. Telemed J E Health 2019 Oct 01;25(10):889-901. [CrossRef] [Medline]
    52. Berry LL. Service innovation is urgent in healthcare. AMS Rev 2019 May 24;9(1-2):78-92. [CrossRef]
    53. Cook T. Quasi-Experimental Design. In: Cooper CL, Flood PC, Freeney Y, editors. Wiley Encyclopedia of Management. Hoboken, NJ: John Wiley & Sons, Ltd; 2015.
    54. Axelrod D, Hayward R. Nonrandomized Interventional Study Designs (Quasi-Experimental Designs). In: Penson DF, Wei JT, editors. Clinical Research Methods for Surgeons. Totowa, NJ: Humana Press; 2006.
    55. White H, Sabarwal S. Quasi-experimental design and methods. Methodological briefs: impact evaluation 2014;8:1. [CrossRef]
    56. Sterne J, Hernán M, McAleenan A, Reeves B, Higgins J. Chapter 25: Assessing risk of bias in a non-randomized study. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.0. London, England: Cochrane Collaboration; 2019.
    57. Dettori J. Loss to follow-up. Evid Based Spine Care J 2011 Mar;2(1):7-10 [FREE Full text] [CrossRef] [Medline]

    ‎
    ICT: information and communication technology
    LHA: local health authority
    MHL: maternal health literacy

    Edited by G Eysenbach; submitted 02.04.20; peer-reviewed by M allman-Farinelli, D Koutsouris; comments to author 04.05.20; revised version received 24.06.20; accepted 28.07.20; published 19.01.21

    Copyright

    ©Manila Bonciani, Sabina De Rosis, Milena Vainieri. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 19.01.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.