Use of Simulation Modeling to Inform Decision Making for Health Care Systems and Policy in Colorectal Cancer Screening: Protocol for a Systematic Review

Introduction

The benefits of colorectal cancer (CRC) screening have been well cited including a reduced incidence of CRC, earlier stage of presentation, improved outcomes for patients with detected malignancy, and reduced CRC-associated mortality [1-4]. Screening uses fecal tests, diagnostic imaging, and endoscopic examination to assess the possiblity of CRC occurring among asymptomatic individuals at increased risk of developing CRC. Participation of eligible individuals is voluntary and remains low in many regions across our country [5]. Determining the best screening modality, delivery method, resource allocation, and follow-up for screening is complex because health policy and system decision makers must weigh the cost and benefits of screening, taking into account the sensitivity, specificity, and accessibility of various screening modalities. Furthermore, such interventions often cannot be tested in clinical trials because of multiple environmental, sociocultural, and health system factors that negate the feasibility and safety of a trial [6]. Researchers and decision makers have increasingly relied on simulation models to evaluate interventions in CRC screening [7,8].

A simulation model is a computer-generated representation of a real-world system or process that can be used to analyze the evolving behavior of a system over time, or modified to predict results of a variety of “what-if” scenarios [9]. Simulation models have been applied to a broad range of areas in health care to predict outcomes, unintended consequences, and costs of proposed interventions, thereby offering an invaluable decision aid for policy makers and health care leaders [7,10-12]. The purpose of simulation models is well defined, that is, to provide decision makers with evidence to facilitate decision making; however, the extent to which simulation modeling has fulfilled this purpose in CRC screening is unknown [13]. Simulation modeling has the potential to provide strong evidence for multiple aspects of informed decision making at the policy and health system level, including a proposed intervention’s resource utilization, cost-effectiveness, feasibility, sustainability, potential impact, and acceptance among stakeholders [14,15]. For instance, by simulating what-if scenarios informed by clinical trials and observational data, the model can help to identify the appropriate age to initiate screening, superiority of one screening modality over another, or the most cost-effective frequency of screening a particular population in the long term. However, the extent to which simulation modeling has realized this potential impact in CRC screening is unknown.

For simulation models to be useful for decision makers, models must be sufficiently accurate and valid for application [13]. There have been concerns with model credibility in health care and reporting of model conceptualization, parameterization, and validation is not consistent in the literature [16-18]. For this reason, in this systematic review, each study will be assessed for adherence to best practice recommendations in model validation as detailed in the Methods section of this proposal [16].

From our review of the literature, we identified two gaps that we plan to address with this systematic review: (1) no systematic review has examined the application of simulation in CRC screening within the last 10 years and (2) no systematic review has specifically addressed the impact of simulation modeling on decision making in health care. The most recent systematic review that had examined CRC screening only included articles until 2007 inclusively [19]. Since that time, systematic reviews have been conducted examining the quality and cost-effectiveness of simulation modeling in breast cancer screening, but not for CRC screening [20,21]. For instance, a systematic review by Sobolev and colleagues [22] looked at the reported “utility” of simulation models in surgical patient flow and reinforced the need for evaluating the impact of models on decision making, but did not formally evaluate this in their review. Therefore, this study aims to address these knowledge gaps by assessing the validity and impact of simulation modeling on health system and policy decision making in CRC screening delivery. Publication of this protocol will allow for critical peer review of the aims and methods outlined for the intended study. This will help strengthen the rigor by which it will be conducted and validate its utility.

Methods

Protocol and Registration

This systematic review will be conducted in accordance with the Cochrane Library systematic reviews guide and the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) [23,24]. This protocol is reported in accordance with the PRISMA-Protocol checklist (Multimedia Appendix 1). The protocol has been submitted for registration in PROSPERO, and any amendments will be filed with PROSPERO (no. 130823).

Search Strategy and Eligibility Criteria

Studies will be selected by searching the academic and gray literature published between January 1, 2008, and March 1, 2019, conducted to identify articles that include (1) simulation modeling methods and (2) a focus on CRC screening. Only full-text articles available in English will be included. Studies will be identified from academic databases (Medline, Embase, Cochrane Central, Scopus, Web of Science, IEEEXplore, ACM Digital Library, Econolit, National Health Service Economic Evaluation Database, Health Assessment Database, and Cost-Effective Analysis Registry) using controlled vocabulary (Medical Subject Heading) and text word search terms selected by the author HS and an experienced librarian Alexandra Davis (Textbox 1).

Terms were selected to capture the most commonly used types of simulation models in health care (discrete event simulation, Markov chain model, Monte-Carlo simulation, agent-based model, and system dynamics model) [11]. This will be supplemented by hand searching of the gray literature and conference proceedings as well as citation searches of selected articles.

Simulation modeling can be used in a broad scope of applications in cancer screening. To help refine the appropriate inclusion criteria and feasibility of this review, a preliminary search of the literature was conducted. A search of the literature published between January 1, 1946, and March 1, 2019, was conducted, and a pilot screening of 100 abstracts was performed by three independent reviewers (HS, PV, and CK) using the following inclusion criteria: (1) simulation model use and (2) CRC screening. This yielded 56 of 100 included abstracts after 24 conflicts were resolved through extensive discussion among all authors (Figure 1). To reduce conflicts, the inclusion and exclusion criteria were further refined to include only original articles describing a simulation model derived from clinical data focused on the delivery of CRC screening individuals with average CRC risk using one or all of the following modalities of screening recommended by the Canadian guidelines within the last 10 years: fecal occult blood test, fecal immunohistochemical testing, flexible sigmoidoscopy, and colonoscopy [25-27]. Excluded articles are those describing other screening modalities not recommended in the Canadian screening guidelines as identified above, commentary or review articles, simulation models that include screening of other cancers, or articles that have no mention of screening delivery. The time frame was also further restricted to only include articles published after 2008 because a systematic review on the use of simulation modeling in health care, including CRC screening, was identified and had included articles published until the end of 2007 [19]. A second pilot screening was conducted using the revised criteria, which yielded fewer conflicts, and the revised inclusion and exclusion criteria were adopted for this systematic review.

‎View this figure

Selection of Studies for the Review

All article titles and abstracts will be screened by three independent reviewers (HS, PV, and CK) using the abstract screening program Abstrakr (Brown University) followed by screening of selected full-text studies for compliance with the eligibility criteria as mentioned above using DistillerSR, version 2019 (Evidence Partners) [28,29]. Articles or reports of the same study will be linked together. Authors of the articles will be contacted to clarify study eligibility, where appropriate. Conflicts will be resolved through discussion between reviewers or consultation with author (RB), as needed.

Data Extraction

All included studies will be reviewed by three independent reviewers (HS, CK, and PV). Using DistillerSR version 2019, data will be extracted regarding the study and model description, and model validation as outlined in Table 1 [13]. Discrepancies will be identified and resolved through discussion, and missing data will be requested from the study authors as needed.

The validation of a simulation model is an important determinant of the risk of bias and applicability of a simulation model. All models will be assessed in accordance with the guidelines of the International Society for Pharmacoeconomics and Outcomes Research-Society for Medical Decision Making (ISPOR-SMDM) report [13]. From the literature review, several tools were identified to assist model developers and users with model validation [13,16,17,30-33]. The ISPOR-SMDM Taskforce has developed good practice guidelines for modeling in health care including recommendations on conceptualization, parameterization, and validation. Of the validation tools identified in the literature, the taskforce report had the broadest scope and most rigorous development process; therefore, it was selected to guide validation assessment in this review [16]. Authors (HS, CK, and PV) will individually assess whether authors report or conduct face validity (wherein experts evaluate model structure, data sources, assumptions, and results), verification or internal validity (check accuracy of coding), cross validity (comparison of results with other models analyzing the same problem), external validity (comparing model results with real-world results), and predictive validity (comparing model results with prospectively observed events), as outlined in Table 1 [13].

Studies will then be assessed for the extent to which the study has or could potentially have contributed evidence toward informed decision making. For reported contribution, each article will be searched in its entirety for statements referring to the simulation model results informing decision making. If not clearly stated in the publication, the information will be requested from study authors.

Recognizing that the impact of a simulation model on a specific decision may not be communicated at the time of publication, we plan to also assess the potential contribution a simulation model could have made to evidence-informed decision making based on whether the results align with important factors for making an informed decision [34]. We will assess articles to determine whether they include evidence considered to be important for decisions, as outlined in the internationally recognized Grading of Recommendations Assessment, Development and Evaluation Evidence to Decision (GRADE EtD) framework [14]. The GRADE EtD framework has been developed as part of the Developing and Evaluating Communication strategies to support Informed Decisions and practice based on Evidence (DECIDE) project in collaboration with researchers in the health system and public health internationally. It outlines a set of important factors that decision makers should consider and address with research evidence to guide their decisions in health policy or systems [35]. These criteria include information on an intervention’s resource utilization, cost-effectiveness, impact on health equity, and feasibility (Table 2). We will assess whether the study results apply to the GRADE EtD criteria. Subgroup analysis of articles based on their GRADE EtD criteria will be conducted to identify methods associated with decision support capacity (ie, quantitative or mixed methods).

Results

A preliminary search of the literature published between January 1, 1946, and March 1, 2019, was conducted, yielding 617 references and a pilot screening of 100 randomly selected abstracts was performed by three independent reviewers (HS, PV, and CK) using the following inclusion criteria: (1) simulation model use and (2) CRC screening (Figure 1). This resulted in inclusion of 56 of 100 (56%) abstracts after 24 conflicts were resolved through extensive discussion among all authors, leading to revision and clarification of the inclusion criteria as described in the section “Methods”. The revised search yielded 484 references to review. A repeated pilot screening resulted in the inclusion of 8 of 100 (8%) abstracts after 16 conflicts were resolved with minimal discussion. The publication of this article was funded by University of Ottawa Telfer School of Management Research Grant and a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada.

The project was funded in May 2019. Data collection took place from January 2008 to March 2019. Data analysis was completed in November 2019, and are expected to be published in spring 2020.

Discussion

The purpose of simulation modeling in health care is generally to inform a decision [13,33,36]. The extent to which simulation modeling has fulfilled this purpose has not been assessed. We anticipate that this systematic review will help address this knowledge gap by assessing the contribution simulation modeling has made to informed decision making in an area of health care where it has been frequently used: CRC screening delivery. We will use the GRADE EtD framework to structure our analysis of potential decision impact of the models. This includes the model’s contribution to determining the feasibility of screening, acceptability of proposed screening strategies by stakeholders, and sustainability of screening over the long term. This analysis will help guide researchers by identifying methods in simulation modeling that have been associated with a greater success in decision support, such as mixed methods, participatory simulation model development, and group model building, which has been reported as beneficial in other applications of simulation modeling in health care [37,38]. It will assist decision makers and model users to identify areas where simulation modeling has proven to be useful, such as for identifying resource requirements or conducting cost-benefit analysis.

The dataset search yielded 484 references, which suggests that the body of literature on this topic is fairly robust. We anticipate there will be an adequate number of relevant models to conduct an informative systematic review on this topic.

We foresee several potential limitations to this study. The heterogeneity of articles may make it challenging to evaluate studies using a uniform framework from validation and decision-making criteria. Furthermore, the impact and decision support that a study provide are difficult to quantify and therefore will be subject to both authors’ and reviewers’ bias. We aim to mitigate this by using the GRADE EtD framework and by having reviewers with clinical (HS), health informatics (CK), and simulation (PV) expertise review each included study. Finally, our assessment of model validity will be limited by a lack of validation standards in the literature and reporting by authors on their validation process and outcomes [18].

In conclusion, the proposed systematic review will provide an insight into the contribution and validity of simulation modeling in CRC screening. The results have the potential to inform researchers, health care leaders, and policy makers to develop valid, informative simulation models that will support decision making. This analysis could be expanded to assess the use of simulation modeling in other areas of health care.