Participatory Methods for Systems Modeling of Youth Mental Health: Implementation Protocol

Introduction

Globally, it is estimated that more than 700,000 people lose their life to suicide each year, and suicide is the fourth leading cause of death among young people aged 15 to 29 years [1]. In Australia, suicide is the leading cause of death among people aged 15 to 49 years [2,3]. An estimated 340,000 (15%) Australians aged 18 to 24 years experience high or very levels of psychological distress [4]. Most mental health issues experienced during adulthood begin during childhood or adolescence and can result in disengagement from education and reduced social connections. This can, in turn, lead to reduced employment opportunities and poorer socioeconomic outcomes [5]. A recent national inquiry into mental health in Australia made recommendations that called upon multiple sectors to take action to be responsive to the needs of, and improve outcomes and experiences for, people with mental health conditions. The sectors identified by the inquiry went far beyond those usually associated with mental health services and included primary, secondary, tertiary, and vocational education; mainstream health services; early childhood education and care; disability support; workplace health and safety; finance; housing; insurance; justice and law; and digital health sectors [5]. Putting this broad context together with the individual variation in the etiology and course of mental ill health, deciding where and how to invest resources is difficult [6]. This challenge has traditionally resulted in investment in a comprehensive and all-encompassing package of interventions in which individual programs are often not sufficiently scaled and resourced to deliver impacts [7]. More effective decision support methods are needed to ensure that investment is well-targeted, coordinated, and implemented at a sufficient scale to deliver real impacts [7,8].

Governments have long been under pressure to provide more programs and services within constrained resources [9]. Policy and program decision processes are often complex, contextually dependent, and influenced by a range of competing priorities [10]. Decision support methods and tools are needed that can not only provide insights into which interventions or programs work but also facilitate identification of the context in which they work or whether other interventions, or combinations of interventions, will work better [9,11]. Systems modeling has a long history of providing decision support capability in a range of disciplines [12-14], including engineering, manufacturing, defense, business, and environmental sciences. It is increasingly being used for health applications. Systems models can be developed and used to leverage a range of data and evidence sources by combining them with local contextual knowledge and expertise to inform mental health system investment planning [8]. Involving decision makers and other stakeholders in participatory model development processes can increase the validity, credibility, and utility of models, ensuring they remain focused on priority policy questions and accelerating the mobilization of model insights into practice [11,15,16].

Participatory Systems Modeling

The process of participatory systems modeling involves engaging multidisciplinary stakeholders in group model-building processes [17]. It can be used in conjunction with multiple modeling methods, including system dynamics, discrete event simulation, and agent-based modeling [18-21]. Various terms have been used to describe these activities, including participatory modeling, group model building, companion modeling (ComMod), and participatory simulation [21]. In the participatory modeling process, participants coconceptualize a problem and use modeling to describe and quantify the problem; identify, develop, and test potential solutions; and inform the decision-making and actions of the group [22]. For the purpose of this study, the term participatory systems modeling has been adopted and is defined as “a purposeful learning process for action that engages the implicit and explicit knowledge of stakeholders to create formalized and shared representations of reality [22] using computer simulation.”

The terms stakeholders and participants are both used in this study. By stakeholders, we refer to all who have a stake or involvement in the system [22], and the term participants is used to refer to those people who engage in the participatory modeling process. Therefore, the term stakeholders refers to a broader group of people.

Participatory systems modeling focuses on collaborative learning, and the tools and methods used in these programs promote system understanding and awareness among all stakeholders. The tools and methods used in participatory approaches may differ; however, the underlying principles are very similar and subscribe to the same basic aim—to actively engage end users and other stakeholders in model development to increase the robustness, validity, utility, and credibility of the models and facilitate their use to support decision-making processes [18,19,21,23-25]. Participatory modeling has been an important method in system dynamics modeling since its inception [18] and has been widely adopted in environmental modeling projects [19,23,26-29]. The advantages of engaging stakeholders in model development processes include the following [11,16,24,30]:

1. The contribution of extensive domain expertise of participants to model development
2. Social learning between participants and throughout the model development process
3. Joint problem framing to ensure that the model is focused on priority policy questions
4. Production of regionally customized and socially robust solutions, that is, solutions that are more likely to be trusted and accepted by decision makers and stakeholders
5. Identification and prioritization of evidence gaps
6. Identification of opportunities to insert the model into policy and program decision-making dialogues
7. Development of strategies to address communication challenges

Advances in modeling technologies have allowed greater model transparency and meaningful engagement in the model-building process by interdisciplinary groups [20,25]. Although modeling expertise is still required, modeling is no longer restricted to the computational and mathematical sciences and models are being designed to be broadly accessible across disciplines [31]. Participants engaged in the modeling process are able to inspect and critique the logic, parameters (values used), and assumptions of a model, and simulate scenarios independent of the modelers, using interactive model interfaces [25]. Broader access to, and engagement with, models can support faster model evolution and learning, particularly in identifying discrepancies between model results and empirical observations or knowledge concerning the world and helping to refine mental models across the group [14,19,32,33].

It may be difficult to understand and anticipate the impact of policy decisions on system behavior as a whole [12,14]. However, a quantified systems model can facilitate an increased understanding of system behavior by playing out the logical implications of introducing new policies and initiatives into complex systems, thereby making implicit assumptions explicit [25]. For example, in the mental health sector, participatory systems models co-developed with local health services have pointed to the importance of aftercare for people who have experienced suicidal behavior, a finding that is consistent across applications in diverse regions [34,35]. In addition, modeled intervention effects that initially seem counterintuitive may point to unanticipated consequences. For example, evidence-based community mental health education and awareness raising programs can lead to demand exceeding service capacity, increasing waiting times and disengagement with services, and increasing mental health–related emergency department presentations [36].

Transparent models can help to connect knowledge across the breadth of a team, enhance their ability to identify areas where their knowledge falls short, uncover logical inconsistencies, and contribute to more robust and strategic decision-making [25]. From this perspective, the discovery of an inconsistency between what the model suggests in simulation scenarios and empirical data is an opportunity to facilitate learning and refine the model to improve its forecasting capability, thereby increasing its value as a long-term decision support asset [20,25,28].

Principles of Participatory Systems Modeling

Frameworks, guidelines, and principles for participatory systems modeling have primarily been developed within the environmental sciences field where it has been widely acknowledged that sustainability issues involve social processes and stakeholder engagement is necessary to support effective action [21,24,37-39]. These have ranged from highly prescriptive scripts used for group model building often associated with system dynamics modeling [18,19,23,40-42] to more general guidelines and considerations [20,21,24,32,43].

Participatory systems modeling projects are diverse, and flexible principles guiding the conduct of participatory processes that are also easily modifiable and applicable across sectors provide a practical approach to inform existing and future practices [21,24]. The following principles are described in Multimedia Appendix 1 and have been formulated based on recommendations from the literature [44] and the experience of the authors conducting participatory systems modeling for physical and mental health applications [16,17,30,34,45-49]:

1. Selecting and planning stakeholder engagement to ensure that appropriate expertise is available to guide model development
2. Being aware of social and group dynamics to facilitate inclusivity and give all participants the opportunity to contribute and having flexibility in the process to accommodate the priorities and preferences of participants
3. Maximizing transparency and openness in the model development process by ensuring that assumptions and data sources are made explicit
4. Iterating and refining by actively engaging participants throughout the model development process and incorporating their feedback
5. Encouraging learning and managing uncertainty through scenario analyses and hypothesis testing

Most strategic mental health policies and planning decisions are complex and not easily addressed using traditional analytic tools. Policy and planning decisions are challenged by multiple interacting factors with uncertain outcomes, competing options for action and investment, differing expert and local views of effective actions, and the potential for unintended consequences [17,50]. Scenario analysis can assist in identifying optimal combinations of interventions that remain effective even when the conditions in the system are varied [51]. An iterative, participatory approach to modeling allows the identification of data gaps and priorities for new data collection and development of ways to address these [11,20,21]. Models have significant potential to assist in good decision-making through the participatory process by bringing together best evidence, data, and knowledge and consolidating and testing shared hypotheses [25,32].

Aims and Objectives of the Method Blueprint

Although the importance of involving stakeholders in model development processes to increase the relevance, validity, usability, and credibility of models has been recognized, the methods for doing so are not always well understood and terminology can be used loosely, leading to confusion. This paper aims to promote a shared understanding of our participatory systems modeling approach and provide a practical, detailed blueprint to support the implementation of the approach.

This paper was developed as a semistandardized guide for implementation across subnational regions as part of a broader program of participatory action research that aims to explore the feasibility, value, impact, and sustainability of building regional capacity in the use of more advanced decision support tools and technologies to inform systems strengthening and empower communities to address the mental health needs of young people [52,53]. The Right Care, First Time, Where You Live research program will use participatory methods to deliver contextually relevant systems models focusing on youth mental health services in multiple sites across Australia. More information about the research program is available [52]. The systems modeling program aims to provide regional health authorities, social service providers, and community stakeholders the tools, processes, and insights needed to more effectively allocate limited available resources and make compelling cases for further investments [53]. This paper outlines the participatory systems modeling procedures, methods, and activities that will be implemented to support this mental health multisite study related to youth.

The process described in this study builds on the experience of the authors conducting participatory systems modeling for a range of public health issues [6,8,16,17,30,34,45-48,54,55]. However, this study focuses on modeling for mental health applications in this multisite program. Operationalizing participatory systems modeling for mental health applications involves ensuring that the process is inclusive for people with lived experience of mental ill health and their support people.

Methods

Role Descriptions for the Interdisciplinary Core Project Team Members

The project roles mentioned later will be key to successful implementation. The descriptions are adapted from Atkinson et al [46] and Freebairn et al [16]. It should be noted that one person can play multiple roles in a project, that is, the project lead may also be the domain expert or the modeler may be experienced in modeling for policy so may also act as a translator. The term primary partner agency refers to the main stakeholder organization that will facilitate the modeling project in the local community. In the Australian health service context, this agency may be the Primary Health Network or jurisdictional health services.

A project lead facilitates the brokering and management of a project. This person will have the primary responsibility of engaging and maintaining relationships with participants and health service partners (ie, end users of the model). The project lead shares the duties of facilitating the modeling workshops with the lead domain expert and overseeing model development, associated documentation, and external communications.

A lead domain expert is a well-respected authority on the focus issue who can play a lead role in project planning and workshop facilitation.

A translator is a person who can contextualize the policy environment and data for the modeling team and translate the model requirements and development process to the participants.

Expert participants are people with a range of perspectives from across the system being modeled and policy, planning, and content area expertise, including representatives from local Aboriginal and Torres Strait Islander governance bodies, people with lived experience of mental health and suicidal behavior, representatives from federal and state governments, health and social policy agencies, local councils, nongovernment organizations, emergency services, research institutions, community groups, and primary care providers.

A dynamic simulation modeler is a person with expertise in systems modeling and ideally with a background in biostatistics, data science, or mathematics.

An economist is a person with expertise in economic evaluation of policy interventions, including multiple costing methodologies and different valuation techniques, with knowledge of decision analysis and priority-setting processes.

Superusers are nominated persons from within decision-making or primary partner agencies who will be socialized to the model and build competency in using it to explore policy scenarios and interpret and report model findings for reports, policy briefs, business cases, and advocacy.

Research or project support officers are responsible for coordinating the participatory systems modeling process, including logistical arrangements for workshops and liaising directly with the modeler and workshop participants to source and manage evidence and data requirements for the model-building process.

An expert technical adviser provides an independent review of the model, including model conceptualization, equations, and dimensional consistency, to identify errors and ensure that the model is robust and computationally efficient.

Considerations When Including People With Lived Experience of Mental Health Issues and Their Support People

The advantages of involving people with lived experience of mental ill health, including carers, are well established and include ensuring that their essential knowledge about current care pathways, barriers, shortfalls, and what is needed from the mental health system, is embedded in research and service design to improve outcomes at individual, service, organization, and system levels [56-62]. The participatory modeling procedures described in this implementation protocol are informed by the literature regarding best practice principles for supporting consumer and carer participation in mental health research and as described by the National Mental Health Commission [56]. These procedures are also informed by experiences of the authors in implementing previous participatory modeling projects in the mental health sector in collaboration with people with lived experience of mental health issues and suicidal behavior.

The choice of location, venue, and timing for workshops should consider the needs of participants. Barriers to participation can include distance and travel times and environments that cannot accommodate individual requirements such as mobility access, space to take time-out breaks or dietary needs, and insufficient lead in time for workshops to allow participants to prepare for engagement, including taking a time-out from other commitments. Efforts will be made to provide an inclusive culture and safe environment at the workshops that supports the engagement of all participants [56]. In practical terms, this may include being explicit about the ground rules for safe and acceptable disclosure, ensuring that participants have access to and are aware of supports available, for example, debriefing and referral to professional support, allowing people to take a break when needed, and observing levels of psychological distress within the group; checking in with participants; and offering support where appropriate. People with lived experience may prefer to bring their peer support worker or carer to workshops as they can detect signs of distress well and can provide support as needed throughout the event, particularly when discussing suicidal behavior.

Workshop facilitators and support staff will ensure that large and small group discussions are respectful and inclusive for all participants, for example, valuing contributions from all participants, minimizing instances where people are interrupted or cut off or not listened to, and minimizing the use of jargon. Informal conversations during breaks and outside of workshops can be used to encourage participation in the process by providing additional information and clarification about the process and method and opportunities to contribute outside of the formal workshop process, for example, by sharing stories and experiences of the mental health system in direct, one-on-one conversation with the project team members.

The language used throughout the modeling process, but particularly during participatory workshops, will need to be inclusive and increase the likelihood that participants with lived experience of mental health issues feel respected and valued and able to contribute actively to the model development process. Language should be age-appropriate, respectful, nonjudgmental, jargon-free, and accessible to lay people. In addition, the language should be person-centered, that is, person with mental health condition rather than they are mentally ill, and recovery-oriented—conveying the potential for hope and opportunity. The Mental Health Coordinating Council provides an extensive and practical guide for using recovery-oriented language [63].

Ongoing evaluation of the process using participatory action research will be undertaken as part of the broader program to facilitate the identification of opportunities to improve consumer and carer engagement in participatory system modeling processes.

Procedures for Building and Using Systems Models With Stakeholders

Overview

Participatory model development involves an iterative process of understanding local systems, pathways, and drivers by engaging with local stakeholders and the academic literature, populating the model with data, validating its structure and performance with historic data, and ensuring face validity with stakeholders. It centers around 3 workshops where participants interact and actively engage in group model-building activities (Figure 1). Experts and key participants with an important stake in the topic are identified and invited to participate in the model development group (participants). Their expert knowledge and local practice experience are triangulated with research evidence and primary and secondary or administrative data sources to articulate the causal mechanisms driving mental health and suicide outcomes in a region.

‎View this figure

Script-based approaches [18,19,41,42,64,65] have traditionally been used in system dynamics modeling. Our methods favor more organic, minimally structured activities that allow people to tell their stories and share their experiences, which elicits valuable systems information without the constraint of conforming to closely scripted activities or disciplinary conventions of systems science. This process aims to elicit information about what really happens on the ground, hidden incentives or disincentives embedded in the system, and information that is not available in academic papers or data sets but has significant implications for key outcomes and system functioning locally. Providing opportunities for interaction between participants as a group is valuable, as it often reveals more about the system than if the project team has met with participants individually. System learning also occurs when people hear the perspectives and challenges of those in other parts of the system who they may not frequently interact with.

The following sections describe the process for conducting participatory systems modeling workshops for mental health applications. It should be noted that this paper provides guidance specific to facilitating participatory systems modeling; however, it does not cover more general activities involved in workshop organization, for example, booking venues, arranging travel, organizing catering, managing RSVPs, and registration of participants. This is assumed knowledge for this paper.

Early Stakeholder Engagement and Workshop Preparation

Establishing effective partnerships is key to the successful implementation of participatory systems modeling projects. Early engagement with the primary partner agency for each site is important to ensure that the modeling will address their decision analysis needs. In some regions, improving mental health outcomes and reducing suicidal behavior will need to prioritize decision-making around investments to address youth justice, substance misuse, and unemployment challenges, where others may need to prioritize improving access to services. These discussions help to scope the model without changing the primary research question and the overarching purpose of the work.

There are four key activities in the engagement process: (1) determining the capacity of the stakeholder community to participate and actively engage in the activities; (2) communicating the purpose and goals of the systems modeling research and gaining commitment from stakeholders; (3) initiating engagement with the community and understanding current issues, challenges, and alternative perspectives; and (4) establishing relationships and building trust between researchers and community stakeholders [66,67]. The site and stakeholders engagement process to be implemented across sites will be undertaken using participatory action research principles.

Important considerations for this phase of the project include the following:

1. Identification of decision-making priorities for modeling: participating in a systems modeling process requires significant time investment for stakeholders. Motivation for stakeholders to participate is higher when addressing the mental health challenge is a high priority for the region, where the policy and planning environment is complex, where there are contested potential solutions, or where previous attempts at addressing the issue have not delivered impacts [16]. It is important to invest time in discussing local needs to ensure that the modeling is focused on answering the priority questions of the stakeholder group.
2. Identification and engagement of key contacts within the primary partner agency: once a key contact has been identified and engaged, their assistance will be sought to facilitate the identification and engagement of key stakeholders and organizations within their local community for youth mental health services.
3. Venue selection: there are two main considerations for selecting a venue for participatory workshops—first, the venue needs to have a facility to project presentations (including presentations of model architecture) onto a large screen where small text remains legible to participants. Second, the room needs to provide sufficient space for the participants to easily move around a large conference table. An important activity for workshop 1 is for participants to interact and engage in the conceptual mapping of the system. This involves the participants working with paper, tape, and sticky notes to contribute to the system diagram that is laid out on a large table.

At least 2 members of the project team will facilitate the workshops and at least another 2 will provide workshop support. Wherever possible, the workshop facilitators should include a local domain expert who works together with the modeler and project lead to facilitate the workshop. For example, in workshop 1, the local domain expert presents an overview of the epidemiology of mental health and suicidal behavior of the region being modeled and the modeler presents an introduction to systems modeling. The workshop facilitators jointly explain and support participatory activities. The workshop support staff provide logistic assistance to ensure smooth operation of the workshop. This includes welcoming and registering participants; ensuring that any necessary paperwork, for example, regarding participation, photography consent, and confidentiality have been completed; liaising with the venue staff; taking photographic records; ensuring that audio recording devices are in place and turned on during the appropriate sessions; setting out materials; and rearranging the room as needed to facilitate workshop activities. A list of materials for the core workshop activities is provided in Multimedia Appendix 1. It is beneficial, when there is sufficient capacity, for the workshop support staff to be workshop observers, recording field notes that can be used to support the project team debriefing and reflective analysis of each workshop. Field notes and observations would focus on levels of participant engagement in workshop sessions and questions and issues raised by participants during discussions and interactive activities. Runsheets may be developed for each workshop to clearly outline the roles of presenters and workshop facilitators and describe the activities and timing of each session in the workshop.

Participant Selection and Recruitment

Participant selection will be conducted in collaboration with the primary partner agency for each site to embed contextual knowledge into the participant selection process.

Purposive sampling will be used to recruit participants with diverse perspectives and expertise, including young people with lived experience of mental health issues, carers, members of the local Aboriginal and Torres Strait Islander community, mental health professionals, educators, policy makers, service planners, primary health care providers, health service managers, and other service providers. A comprehensive list of stakeholder categories is provided in Textbox 1.

The aim is to include participants who are recognized as local leaders in consumer and carer experience, providing services, planning, and commissioning services and developing policies and the local Aboriginal and Torres Strait Islander community.

Selected participants will be provided with information relating to the aims of the project, the time commitment required, the likely timing of the workshops, and background information regarding systems modeling. Where possible, the initial invitation of participants is extended by the local domain expert or primary partner agency.

Before the first workshop, participants will be asked to provide written consent to participate, be recorded and photographed at workshops, and agree to not disclose confidential information shared by other participants during the model development process, which is important for providing a safe environment for sharing information that is important for model development but which the participants may not wish to be shared publicly.

Workshop 1: Introduction to Participatory Systems Modeling and Conceptually Mapping the System

Overview

The logistics involved in planning and implementing workshop 1 and the activities undertaken to achieve the workshop objectives are discussed in detail. Although the overall aim of the participatory workshops is to maximize the active engagement and interaction of participants, it is necessary to initially present some background and context setting information. Therefore, once the welcome and introductory activities are complete, the initial sessions in the workshop involve presentations of relevant information to support the participatory systems modeling process. An example agenda is included in Multimedia Appendix 1. The five main objectives for workshop 1 are as follows:

1. Present an overview of the epidemiology of mental health issues and suicidal behavior relevant to the region or population catchment being modeled
2. Introduce systems modeling to the participant group to facilitate meaningful engagement in the model development process
3. Jointly conceptualize and map system structure and drivers
4. Prioritize the interventions and outcomes to be explored by the model
5. Introduce the economic analysis and explain that it will be developed following workshop 1, as the model’s purpose and scope develop

Welcome and Overview of the Project Aims and Objectives

It is important that participants are welcomed by the host organization in accordance with local customs, for example, in Australia, this would include an acknowledgment of the traditional owners of the country upon which the workshop is taking place and would also include an acknowledgment of the lived experience of mental health issues and recovery.

The aims and objectives of the project, drawn from project scoping discussions with the key stakeholder organization, will be presented to the participants. The overall aim may be quite broad, for example, “this project aims to use a co-designed systems model to provide a robust, evidence-based, interactive decision support tool to improve population mental health” or it may be quite specific “...a decision support tool to inform suicide prevention in young people,” depending on priorities identified in scoping discussions.

Session 1: Introduction to Dynamic Simulation Modeling (Time Allowed 30 Minutes; Purpose of the Session and Method)

The purpose of this session is to provide participants with an overview of what systems modeling is and what it can offer in a decision-making context.

A slide-supported presentation covering the points mentioned in Textbox 2.

‎View this figure

‎View this figure

Session 2: Introduction to System Dynamics Modeling (Time Allowed 30 Minutes; Purpose of the Session and Method)

The purpose of this session is to provide participants with sufficient knowledge of the basic concepts and graphical language underpinning system dynamics modeling to enable them to participate meaningfully in the model development process, including in the conceptual mapping activity in session 4, and the ability to critique the model in workshop 2.

The examples used here apply to system dynamics modeling; however, the introduction should focus on whichever modeling method is being used in the project, for example, state charts for agent-based modeling or process diagrams for discrete event simulation.

A slide-supported presentation covering the key terms and concepts of system dynamics modeling using visual diagrams (Figure 4) and plain language. The core concepts explained include the following:

1. Stocks and accumulations
2. Flows
3. Reinforcing and balancing feedback loops
4. How data will be used to calibrate the model

‎View this figure

Session 3: Defining the System and Outcomes of Interest (Time Allowed 30 Minutes; Purpose of the Session and Method)

The purpose of this session is to provide participants with an overview of the mental health system and suicidal behavior in the local context and initiate discussions and priority outcomes to be tracked in the model.

It is preferable for the overview to be presented by a local, trusted domain expert and would cover the topics listed in Textbox 3.

Session 4: Participatory Mapping Exercise (Time Allowed 1.5-2 hours With a Break Partway Through; Purpose of the Session and Method)

The activity undertaken in this session provides the most critical outcome for workshop 1, a co-designed conceptual map of the system to be modeled. In this activity, participants interact with each other and the project team to jointly conceptualize and qualitatively map the youth mental health system in the form of a draft model structure.

The mapping exercise is undertaken using the procedures listed in Textbox 4.

‎View this figure

Session 5: Prioritizing the Interventions of Interest (Time Allowed 45 Minutes; Purpose of the Session and Method)

The purpose of this session is for participants to identify the priority interventions of interest in modeling projects to ensure that the appropriate level of detail is included in the model structure to capture their effects.

This activity will either be run as a facilitated discussion and voting process conducted at the workshop or a larger community engagement process, including the facilitated discussion described later, and a postworkshop survey sent out via workshop participant networks where community members vote on which interventions are the highest priority for inclusion in the model. The decision regarding which selection method to use would be made in consultation with the primary partner agency to best meet the needs of the site. Regardless of the method used, diverse views are likely to be expressed. Both prioritization methods are described in subsequent sections. The voting or survey processes generally identify the priority interventions (or outcomes); however, this is followed up with a discussion in workshop 2 that aims to achieve a broader consensus. The workshop discussion option is described in Textbox 5 and the post-workshop survey is described later.

‎View this figure

Postworkshop Survey Activity to Facilitate Wider Community Consultation

When broader community consultation to prioritize interventions is preferred by the primary partner organization, it is facilitated using the following procedures:

1. Follow the aforementioned steps outlined in the first two bullet points in Textbox 5.
2. Present a slide showing a list of potential interventions that have been derived from the preworkshop engagement discussions (Figure 6).
3. Explain to the participants that their support will be required to distribute a survey to the wider community, asking people to prioritize which interventions are most important, from their perspective, to be included in the modeling process. Explain that it will be important for the participants to encourage people in their network to complete the survey to ensure that the voice of the community is heard in the model-building process.
4. Describe the survey process, for example, how the survey will be distributed and the timeframe for responses, and that the results from the survey will be combined with the workshop discussions and presented back to the participants in workshop 2 as a prioritized list of interventions.

Session 6: Economics, Next Steps, and Data Contribution (Time Allowed 30 Minutes; Purpose of the Session and Method)

There are two purposes for this session as follows:

1. To introduce the economic approach that will be used and explain the data requirements for the analysis.
2. To highlight the progress and valuable contributions made in workshop 1, discuss potential timing for workshop 2, identify sources of data and evidence to inform the model, and invite participants to contribute their expertise outside the workshop process.

Facilitated discussion supported by slide presentation:

1. Economics lead introduces the role of economic analysis in the project and how it can be used to contribute to the decision support purpose of the modeling process. A broad overview of the economic approach is provided, including an explanation of the types of economic analyses that may be used and how this can be guided by model purpose and available data.
2. Facilitators present a summary of the achievements over the course of the day, for example, applying systems approaches to understanding the mental health system and suicide prevention, including collaboratively mapping the contributing factors and service system. Link these achievements back to the overall project aims and objectives by explaining that this is the start of the process to develop a decision support tool.
3. Revisit the participatory process, explaining the activities in each of the 3 workshops and engagement that will happen between the workshops.
4. Propose approximate timing for workshops 2 and 3 and encourage participants to identify how they can play an active role in the development of the model by indicating to the project team any data sources that may be of use for model development and their willingness to be contacted out of session.

Concluding Session (Time Allowed 10 Minutes)

This session is an opportunity for a representative from the hosting stakeholder organization to thank the participants for contributing their time and expertise to model development.

Allow the opportunity for any participants to contribute concluding remarks either in a group format or individually to the project team.

Workshop 2: Defining, Refining, and Mapping Interventions

The main objectives of workshop 2 are to provide an update on progress since workshop 1, present the current version of the systems model to the participant group, jointly conceptualize and map the interventions to be explored in the model, refine the outcomes to be measured, and outline the health economic components of the project in detail. The workshop sessions and activities undertaken to achieve these objectives are discussed in detail later, and an example agenda is provided in Multimedia Appendix 1.

Welcome Back and Recap From Workshop 1: Purpose of the Session and Method

The purpose of this session is to reintroduce participants to each other, the project, and the methodology.

A facilitated discussion and slide-supported presentation covering:

1. Welcome back and housekeeping
2. Recap of workshop 1—a short presentation recapping the overall project aim, the activities, and outcomes from workshop 1; the consultation and interactions that have taken place since workshop 1; and the project timeline
3. Aims of this workshop—a brief overview of the purpose of and activities planned for this workshop

Session 1: Presentation of the System Dynamics Model (Time Allowed 60 Minutes; Purpose of the Session and Method)

There are 3 purposes for this session as follows:

1. To present the current draft version of the model structure and logic.
2. To ensure that the model is transparent and familiar to participants.
3. To elicit feedback from participants on the model structure, logic, and data used to parameterize or calibrate the model.

A slide-supported presentation by the lead modeler with facilitated discussion:

• Recap the building blocks of system dynamics, explaining stock and flow diagrams and model initialization values and parameters, and then demonstrate how they are combined to build a representation of a complex mental health system (see the example slide in Figure 5 from workshop 1).
• An overview diagram of the model is presented, showing the main components of the model and how they fit together (Figure 7). This can be emphasized using examples of within-component and between-component dynamics [48]:

...for example, within the health system component, the proportion of the population waiting for services, receiving services, or disengaging from services changes over time based on service system capacity and the rates of flow into, within, and out of the service system. Dynamics also occur between the model components, for example, as unemployment rises, not only does it directly act to increase the incidence of high to very high psychological distress in the modeled population (which has flow-on effects on rates of substance misuse, and adverse early life exposures), but it also increases rates of domestic violence and homelessness, both of which further increase the rate of psychological distress.

• The modeler explains the logic and structure of each model component. Example diagrams of model components, for example, the psychological distress component, are available elsewhere [48]. Uncertain parameter estimates and assumptions needing clarification for each component will be identified and discussed with the participant group in this session. Where available, it can be useful to include outcomes of the calibration process, demonstrating how well the model fits historical data trends which build confidence in the model’s causal hypothesis. Participants are encouraged to probe, ask questions, and provide further information and feedback that will assist in refining the model.

‎View this figure

Session 2: Intervention Mapping (Time Allowed 30 Minutes; Purpose of the Session and Method)

The purpose of this session is to provide feedback to participants about the interventions prioritized for inclusion in the model.

A slide-supported presentation by the project lead and local cofacilitator with facilitated discussion:

1. Present results from the previously conducted intervention prioritization process. Two options have been proposed in this study to elicit the intervention priorities. The first option involves an activity in workshop 1 where participants nominate and vote on interventions to assign priority. The second option involves distributing a survey through participants’ networks to gauge community perceptions of which interventions are the highest priority to model. The results of the chosen process are represented in this session.
2. Include a slide with other intervention options that have been identified or discussed after workshop 1 but were not included in the original list, if appropriate.
3. Discuss with the participants whether they still agree with the prioritized list of interventions or modifications need to be made. This is a facilitated discussion to achieve a broad agreement on the suite of interventions to be modeled. However, where very strongly held views remain about the inclusion of an intervention or outcome that is not prioritized, the modeling team attempts to accommodate this if possible or note it for later development of the model.

Session 3: Intervention Mapping Exercise (Time Allowed 1.5-2 Hours; Purpose of the Session and Method)

The activity undertaken in this session provides the most important outcome for workshop 2, mapping intervention effects to the model structure. In this activity, participants interact with each other and the project team to jointly define the prioritized interventions and identify where they are likely to have their effect in the model structure.

A facilitated activity conducted in small groups according to the procedures listed in Textbox 6.

The preprinted copies of the model structure are provided to each of the small groups to map the mechanism of the effect of the intervention directly to the model structure. Different color markers can be used for each intervention on a single printout, or multiple printouts can be used, one for each intervention. The important factor is ensuring that each intervention is clearly differentiated.

A member of the project team will work with each small group to assist them in working through the questions and the mapping activity and respond to any questions raised.

It can be useful to split the intervention mapping work over 2 sessions with a break in the middle to allow the project team to discuss whether there are gaps in the discussion, allow the participants to interact, or approach the project team for a one-on-one conversation to provide information that they would prefer not to share in a group discussion.

Session 4: Economic Component (Time Allowed 30 Minutes; Purpose of the Session and Method)

The purpose of this session is to provide an overview of the aims and intended approach and encourage participant feedback.

A slide-supported presentation of approximately 25 minutes, with an additional 5 minutes of questions at the end (Textbox 7).

Session 5: Concluding Session (Time Allowed 15 Minutes; Purpose of the Session)

The purpose of this session is to acknowledge the valuable contribution made by the participants at the workshop and note the likely timing for workshop 3. This session provides an opportunity for a representative from the primary partner organization to thank the participants for their ongoing contribution to model development. This session is also an opportunity for the participants to contribute concluding remarks either in a group format or individually to the project team.

Workshop 3: Introducing the User Interface and Delivering Model Insights

The main objectives of workshop 3 are to present the penultimate version of the systems model to the participants by walking them through a high-level summary of the model, highlighting any major changes since the previous workshop and the user interface and demonstrating how it can be used to simulate intervention scenarios and preliminary insights from the model. The activities undertaken to achieve these objectives are discussed in detail later, and an example agenda is provided in Multimedia Appendix 1.

Welcome Back and Progress Update: Purpose of the Session and Method

The purpose of this session is to reorientate participants to the project and methodology.

A facilitated discussion and slide-supported presentation covering:

1. Welcome back and housekeeping.
2. Recap of progress—a short presentation recapping the activities and outcomes from workshops 1 and 2, including presenting back the list of interventions that were prioritized at workshop 2 and the consultation and interactions that have taken place outside the workshop settings.
3. Aims of this workshop—a brief overview of the purpose of and activities planned for this workshop.

Session 1: Demonstration of the System Dynamics Model (Time Allowed 45-60 Minutes; Purpose of the Session and Method)

This session has 3 objectives as follows:

1. To represent the high-level model structure and logic.
2. To advise on any major changes based on feedback from workshop 2.
3. To demonstrate model use to participants.

A slide-supported presentation by the lead modeler with facilitated discussion:

1. An overview of the model will be presented, showing the main components of the model and how they fit together. An example of this is shown in Figure 7.
2. Updates or revisions to the model since the previous workshop will be described. The description will emphasize where participant feedback has been incorporated into the model.
3. Additional slides focusing on revised and newly added model components, for example, the structure for one or more example interventions, can be presented.
4. Demonstrate the use of the model interface, including running scenarios in the live model.
5. Run a set of intervention scenarios to draw out key model insights. The modeler is often required to explain the reason behind some insights, particularly if they are counterintuitive.
6. Discuss the policy and planning implications of the initial model insights.

Session 2: User Interaction With Model (Time Allowed 60 Minutes; Purpose of the Session and Method)

The purpose of this session is to provide an opportunity for participants to gain experience using the model to explore scenarios and provide feedback on the functionality of the user interface.

This is an interactive session in which the participants interact with the model interface. A member of the project team is stationed with each computer to provide guidance, interpretation of findings, or technical assistance, where required.

The project team should ensure that access to one computer per 5 to 7 participants is available so that a diverse group has the opportunity to interact directly with the model.

Feedback questions can be printed or presented on a slide to guide feedback on the model interface as the participants interact with it. The feedback questions can be tailored to ensure relevance to different modeling projects but, in general, would include the following:

• What should the available ranges be on the slides?
• Are there any labeling or language issues that we need to address?
• Any other comments on the interface or model?

Feedback can be given directly on printed screenshots of the model interface provided for each small group. This session will likely generate further questions about the model and discussion about the results of the simulated scenarios. The lead modeler will be available to move between groups, as necessary, to respond to technical questions and assist with interpretation. This small group discussion is an opportunity for the project team and local domain lead to engage with the participants to ascertain where further clarification is needed, for example, how to vary input values or how to interpret results of simulated scenarios.

As highlighted in the aforementioned feedback questions, it is very important to ensure that the language used in the model interface is accurate, understandable, relevant, and acceptable for end users. The highest priority is to ensure that the language used does not inadvertently alienate or offend participants. Ideally, the model interface will be user-tested with participants, for example, from the primary partner agency and other key user groups, such as reference groups for people with lived experience or Aboriginal and Torres Strait Islander communities, before being presented at the workshop.

Session 3: Health Economics (Time Allowed 30 Minutes; Purpose of the Session and Method)

The purpose of this session is to provide a recap on the approach taken and demonstration of key analyses and findings. If the model is not yet fully developed, then an update should be given with timelines for completion.

A slide-supported presentation of 25 minutes with 5 minutes for questions and clarifications:

1. Recap on the approach of the economics and how that is aligned with model purpose and participant needs following discussions at workshops 1 and 2. Then provide results (or updates) on the three stages of analysis.
2. Stage 1: establishing business as usual—estimating the financial and human cost.Provide a selection of activity-based service costs for exposition (such as hospitalizations). Describe the estimation process and how modeled populations acquire costs as they reside in service stocks to illustrate how the economics is layered into the model. Repeat for health utilities and explain how quality-adjusted life years are estimated. If relevant, continue the exposition for wider impacts, such productivity and carer impacts.
3. Stage 2: costing interventions. Provide the costing estimates for the priority set of interventions and explain how these were derived. Highlight if, and why, certain interventions could not be costed properly because of insufficiently defined service delivery models. Reiterate that a what-if analysis can be conducted.
4. Stage 3: making the value proposition—creating the business case and supporting the moral case. Provide an illustration by selecting 2 examples to demonstrate the value of investing in interventions to reduce distress. One example can select a single intervention (which could be a what-if analysis) and the other should be a combination of interventions to demonstrate the capability of the model to develop an optimal intervention portfolio relative to a budget. These examples may include one or a combination of valuation methods, namely, return on investment, cost-effectiveness (utility), and cost benefit.
5. Use screenshots from the economic component of the model dashboard for ease of exposition and demonstrate to participants how they can also use the controls in the dashboard to select interventions and generate economic outputs and how this can feed into a relevant business case for investment.

Session 4: Concluding Session: Next Steps, Closing Remarks, and Feedback

The workshop facilitators will describe the achievements from the project and the valuable contributions made by the participants. The facilitators will explain how the model will be made available to the participant group, how the model will be used to inform regional decision-making, how the model will be maintained, and how ongoing technical support will be provided. Ideally, additional modeling informed strategy dialogues are hosted by the primary partner agency to build a collaborative consensus for action through discussions with broader community stakeholders and interaction with the model. The project team will provide support as needed to that process, including superuser training to build capacity in the independent use of the model.

This session is also an opportunity for a representative from the hosting stakeholder organization to thank the participants for their contribution to the model development. It will also give opportunity for any participants to contribute concluding remarks either in a group format or individually to the project team.

In our approach, we invite all participants to let the project team know if they are interested in coauthoring peer-review publications from this study. This is an important recognition of the substantial time and intellectual contribution that the participants have made to the development of their local model. Participants who do not engage in coauthoring papers are, with their consent, acknowledged as members of the modeling consortium.

Ethics

Systems modeling processes do not routinely require ethics approval as they involve secondary analysis of data and are considered a process of evidence synthesis. However, ethics approval was requested and granted by the Sydney Local Health District Human Research Ethics Committee (protocol number X21-0151 and 2021/ETH00553) for the participatory action evaluation research being conducted alongside the participatory systems modeling processes across the 8 sites.

Results

This study has been developed for implementing participatory systems modeling in the Right Care, First Time, Where You Live program, which is funded from 2021 to 2025. The methods described in this paper will be implemented at 2 sites per year from 2022 to 2025. The 8 selected sites include urban, regional, and rural or remote mental health service settings and are chosen to capture variations in socioeconomic conditions and demographics, population density, mental health risk profile, and access to mental health care and other services. Initial site visits were conducted between August and December 2021.

Discussion

As decision makers navigate complex policy environments, including mental health, there is a need to leverage interdisciplinary problem-solving and advanced decision support tools, such as systems modeling, to mobilize a wide range of evidence, data, and other forms of information to inform effective decision-making [14,32,33,68-70]. The involvement of stakeholders and coproduction of knowledge are critical to ensuring that model findings are policy-relevant and can be used to inform decision-making [11,16,25]. In the mental health context, decision makers must consider multisectoral determinants of mental health issues, regional variation and changing local population needs, competing views about what works, and restricted resources and workforce [6].

Systems modeling provides a quantitative method to combine the consideration of individual behavioral, social, cultural, economic, and service risk factors and captures the complex, nonlinear interrelationships, feedback loops, and threshold effects that characterize mental health systems [71]. Actively engaging stakeholders in the development of the systems model ensures transparency and increases their understanding of the model as a decision support tool, which, in turn, increases the likelihood of the model being embedded in policy and planning cycles to target investments more strategically in mental health programs and services [6,7,16]. This protocol describes a structured process that combines diverse perspectives and facilitates interdisciplinary dialogue in the development of systems models for mental health services, which can be adapted for other applications, topics, and settings. Importantly, the protocol includes considerations to ensure that people with lived experience of mental health issues and their supporters can meaningfully contribute to the process.

The participatory process detailed in this protocol emphasizes participant interaction and opportunities to draw out participant knowledge and expertise and actively involves participants in decision-making, giving them a voice and a stake in the outcome. The protocol builds on previous research that identified that the benefits of the participatory process include the enhancement of professional networks; increased transparency and, therefore, familiarity and trust in the model; integration of significant knowledge and evidence into the models; identification and facilitation of policy insights and opportunities to apply model findings in practice; and identification of key messages to deliver to a broader policy and practice audience in a way that is compelling and engaging [16,17,30].

Despite providing acknowledgment of the importance of including end-user stakeholders in model development, many participatory modeling projects do not explicitly describe or reflect on the participatory process component of the project [11,22,24]. This protocol responds to international interest in these methods and provides a blueprint for operationalizing participatory systems modeling based on years of applied systems modeling research for mental health and broader public health applications. Researchers are encouraged to use and challenge this blueprint to advance participatory systems modeling methods.