Background: Patient-reported outcomes—symptoms, treatment side effects, and health-related quality of life—are important to consider in chronic illness care. The increasing availability of health IT to collect patient-reported outcomes and integrate results within the electronic health record provides an unprecedented opportunity to support patients’ symptom monitoring, shared decision-making, and effective use of the health care system.
Objective: The objectives of this study are to co-design a dashboard that displays patient-reported outcomes along with other clinical data (eg, laboratory tests, medications, and appointments) within an electronic health record and conduct a longitudinal demonstration trial to evaluate whether the dashboard is associated with improved shared decision-making and disease management outcomes.
Methods: Co-design teams comprising study investigators, patients with advanced cancer or chronic kidney disease, their care partners, and their clinicians will collaborate to develop the dashboard. Investigators will work with clinic staff to implement the co-designed dashboard for clinical testing during a demonstration trial. The primary outcome of the demonstration trial is whether the quality of shared decision-making increases from baseline to the 3-month follow-up. Secondary outcomes include longitudinal changes in satisfaction with care, self-efficacy in managing treatments and symptoms, health-related quality of life, and use of costly and potentially avoidable health care services. Implementation outcomes (ie, fidelity, appropriateness, acceptability, feasibility, reach, adoption, and sustainability) during the co-design process and demonstration trial will also be collected and summarized.
Results: The dashboard co-design process was completed in May 2020, and data collection for the demonstration trial is anticipated to be completed by the end of July 2022. The results will be disseminated in at least one manuscript per study objective.
Conclusions: This protocol combines stakeholder engagement, health care coproduction frameworks, and health IT to develop a clinically feasible model of person-centered care delivery. The results will inform our current understanding of how best to integrate patient-reported outcome measures into clinical workflows to improve outcomes and reduce the burden of chronic disease on patients and health care systems.
International Registered Report Identifier (IRRID): DERR1-10.2196/38461
Over half of Americans are currently living with a chronic illness such as cancer or chronic kidney disease [, ], and supporting their symptom and care management needs is an urgent priority. Most patients living with a chronic illness experience distressing symptoms and side effects, most commonly fatigue, pain, and emotional distress such as fear or sadness [ - ]. Furthermore, these symptoms often go unrecognized by clinicians during regular visits [ , ], leading patients to experience chronically unmanaged symptoms that can escalate and increase the risk of potentially avoidable health care use [ , ]. Following distress screening guidelines from multiple organizations such as the Centers for Medicare and Medicaid Services [ , ], the National Comprehensive Cancer Network [ ], and the Commission on Cancer [ ], health care systems have begun to implement patient-reported outcome measures to facilitate more timely identification of symptom management needs. The most common strategies for implementing patient-reported outcome measures in clinical care involve feedback of results to clinicians to trigger clinical action such as referrals or feedback to patients to promote self-management of symptoms [ , ]. In this study, we will co-design and pilot-test an alternative system that will feed patient-reported outcome data back to patients and clinicians simultaneously through a shared interface to be used primarily during health care encounters ( ). If implemented successfully, our approach has the potential to improve patient well-being and other clinical outcomes by supporting a collaborative health care communication and shared decision-making process [ - ].
Efforts to improve care delivery for advanced cancer and chronic kidney disease offer strong opportunities to develop and evaluate strategies for implementing patient-reported outcome measures. Medical advances in treatments have made it possible to live months or years with these diagnoses, meaning that patients need to manage their symptoms and treatments on a regular basis and across ongoing health care encounters [, ]. Both diagnoses are accompanied by significant symptom burden [ ], putting patients at risk of decreased health-related quality of life and increased health care expenditures. Currently, national costs associated with cancer and chronic kidney disease exceed US $100 billion per year, with most of those expenditures occurring during the advanced stages of both illnesses [ - ]. In addition, patients need to navigate complex and preference-sensitive decisions about their health care that involve considering their health-related quality of life priorities [ , ]. For instance, the decision to select treatments focused on extending life (eg, chemotherapy and dialysis) versus treatments focused on optimizing comfort (eg, best supportive care) requires thoughtful consideration of how one will feel in the future, both physically and emotionally, in response to treatment options [ , ]. Regularly monitoring patient-reported outcomes can also decrease the use of low-value health care services by creating opportunities for patients to discuss bothersome side effects with their clinicians and revisit alternative treatment options that might provide more value [ , ]. Therefore, incorporating patient-reported outcome measures into care for advanced cancer and chronic kidney disease can improve patients’ health-related quality of life, align care provision with patient preferences and needs, and decrease the burden on the health care system by reducing unnecessary costly services.
Given these benefits, patient-reported outcome data can enhance the health care coproduction process . Coproduction is an approach to health care delivery that posits that health care services and outcomes result from an interaction between clinician- and consumer-driven factors; thus, health care should be coproduced by patients, families, and health care professionals to achieve optimal health [ , ]. At the individual level, health care coproduction includes empowering patients and clinicians to coassess health status, coplan health care decisions, co-design acceptable and effective health services, and ultimately co-deliver those services [ , ]. As health IT has advanced, proponents have suggested that health care systems can encourage coproduction by implementing data visualization dashboards that are linked to the electronic health record and are accessible to both patients and clinicians [ , , , ]. These dashboards can display patient-generated data on symptoms, health-related quality of life, and goals of care along with key clinical data, allowing patients, care partners, and clinicians to view and discuss information together during and after scheduled visits ( ). During in-clinic evaluations, patient-reported outcome clinical dashboards have demonstrated feasibility and acceptability, and preliminary evidence shows that dashboard users report higher-quality shared decision-making, a key process of coproduction, compared with nonusers [ , - ]. However, no previous study has evaluated whether these dashboards are associated with changes from baseline to follow-up time points in shared decision-making and related disease management outcomes.
This paper details the protocol for our study. We will design, implement, and longitudinally evaluate a clinician- and patient-facing dashboard for displaying patient-reported outcomes and other data from the electronic health record. The dashboard will be co-designed by patients and care partners, clinicians, and researchers to optimize the likelihood that it will be acceptable, feasible, and effective . By fully integrating relevant information in a way that is easily viewed, discussed, and revisited by patients and clinicians, the dashboard may also promote other coproduction components such as symptom coassessment, coplanning through shared decision-making, and codelivery of health services by activating and empowering patients to self-manage their symptoms and care [ ]. Although the study’s primary focus is on enhancing the coproduction process for individual patients, the study falls within a quality improvement initiative that ultimately aims to achieve coproduction at the health care system level as well; situated within a learning health system, the dashboard implementation will continuously collect aggregated patient- and clinician-generated data to ensure that stakeholder perspectives are represented in system-wide policy decisions [ , , ]. Therefore, the dashboard has the potential to improve health care system–level outcomes and care efficiency.
This study has two objectives: (1) to co-design a clinical dashboard that integrates patient-reported outcomes with other clinical data (eg, vital signs, laboratory tests, and medications) through the electronic health record and (2) to conduct a longitudinal demonstration trial evaluating whether the dashboard is associated with improved shared decision-making and disease management outcomes. Within objective 2, we hypothesize that patients who use the dashboard will experience increases from baseline to the 3- and 6-month follow-ups in perceived quality of shared decision-making, satisfaction with care, self-efficacy in managing treatments and symptoms, and health-related quality of life. We also hypothesize that, compared with historically and demographically matched controls, patients who use the dashboard will have a greater reduction in use of potentially avoidable, high-cost, and low-value health care services (see the Methods section for specific indicators that will be tested). By carefully delineating the dashboard design and evaluation process, this study will enable the dashboard’s scalability and inform future investigations aiming to adapt the patient-reported outcome dashboard to other health care systems.
The Northwestern University Institutional Review Board approved all procedures described in this manuscript and a detailed data security plan (STU00210091, STU00211654, and STU00212634). Depending on the assessment point, survey data will be collected and stored securely through the health care system’s electronic patient portal or through a REDCap (Research Electronic Data Capture; Vanderbilt University)  server hosted at the Northwestern University Feinberg School of Medicine, both of which are protected by firewalls. Health service use data will be extracted from the Northwestern Medicine Enterprise Data Warehouse by trained Northwestern Medicine data analysts and entered into the study’s REDCap database by the study coordinator (AC) approved by the ethics committee. Audio-recorded focus group data will be collected and stored in compliance with the Health Insurance Portability and Accountability Act. Audio files will be transcribed and deidentified, and transcriptions will be stored on a secure and password-protected Northwestern University Feinberg School of Medicine server accessible only to research staff listed on the study protocol approved by the ethics committee. After the transcription process is complete, all audio files will be deleted. All study procedures were considered low-risk by the Northwestern University Institutional Review Board, and the ethics review concluded that the benefits outweighed any minimal risks.
Objective 1: Dashboard Co-design Process
The dashboard will build on an existing infrastructure called the Northwestern Medicine Patient-Reported Outcomes system. This is the health care system’s current technological framework for administering patient-reported outcome measures electronically and integrating data into the electronic health record to inform clinical care delivery . Specifically, patients receive an email alert through their electronic patient portal 72 hours before an upcoming appointment, prompting a response to a patient-reported outcome questionnaire about symptoms and supportive care needs. To optimize measurement precision using as few items as possible, measures from the Patient-Reported Outcomes Measurement Information System (PROMIS) are used to assess anxiety, depression, pain, fatigue, and physical function. The electronic questionnaire automatically scores the patients’ responses, stores them in the electronic health record, and generates alerts to their care team for any endorsed needs or clinically elevated symptoms.
In this study, this system will be adapted, with patient and other stakeholder input, into a clinical dashboard to facilitate symptom management and shared decision-making during scheduled health care visits for patients with advanced cancer and chronic kidney disease. Patient and care partner input will maximize the dashboard’s patient-centeredness. By including clinicians in the design process, attention will be paid to the possible impact of dashboard participation on digital fatigue. To direct attention to and encourage discussion of the more potentially bothersome “hard-stop” alerts, alert thresholds on the patient-reported outcome measures will be set relatively high. The dashboard will be viewable by both patients (via the electronic patient portal) and physicians (via the health system’s electronic health record software) and will automatically populate electronic health record–linked data, including patient-reported outcomes, laboratory test results, medications, and vital signs.
Participants and Eligibility Criteria
The first objective of the study is to collaborate with stakeholders who represent “end users” to co-design our clinical dashboard. End users include patients with advanced cancer or advanced chronic kidney disease, their care partners, and their clinicians. Clinician participants will include 2 oncologists (specializing in the treatment of gastrointestinal and lung cancer), a nephrologist, a nephrology physician assistant, and 2 primary care physicians. Patients will be eligible to participate if they have a history of receiving care from one or more of the participating clinicians at Northwestern Memorial Health Care. Patients with gastrointestinal cancer must have a confirmed diagnosis of stage 4 gastrointestinal cancer and have been receiving intravenous chemotherapy for at least three months. Patients with lung cancer must have a confirmed diagnosis of stage 3C or 4 lung cancer and have been receiving first- or second-line chemotherapy for at least 3 months. Patients with chronic kidney disease must have a confirmed diagnosis of at least stage 3 (defined as a clinical diagnosis or an estimated glomerular filtration rate of <60). Care partners will be eligible to participate if they assist in the care of a patient who would meet the inclusion criteria.
For each of the two disease groups of interest (advanced cancer and chronic kidney disease), we will convene a co-design team comprising approximately 20 highly engaged stakeholders, including investigators, patients, care partners, clinicians, and health IT professionals. Co-design teams will iteratively develop the dashboard over the course of monthly meetings held during the first year of the study. Co-design meetings will involve team-based working sessions with predefined objectives and deliverables for each meeting (). The co-design process will occur in 4 broad phases informed by the Dartmouth Model for Co-design and Implementation ( [ , ]), which has been successfully used to co-design and implement coproduction dashboards for people living with other chronic illnesses [ , ]. The 4 phases of co-design will include (1) defining the problem from the perspective of the end users, (2) understanding the context of use and lived experience, (3) building a design consensus, and (4) establishing and pilot-testing design specifications [ ].
|Activity||Co-design phase ()||Objectives|
|Co-design launch meeting||1|
|Working session 1||1 and 2|
|Working session 2||2|
|Working session 3||2 and 3|
|Working session 4||1, 2, and 3|
|Working session 5||2, 3, and 4|
|Working session 6||2, 3, and 4|
|Working session 7||2 and 3|
|Working session 8||2 and 3|
|Working session 9||2 and 3|
|Working session 10||2 and 3|
|Working session 11||3 and 4|
|Working session 12||2 and 3|
|Working session 13 (physician champion working meeting)||2, 3, and 4|
|Co-design wrap-up meeting||4|
provides a detailed outline of the objectives of each co-design meeting. In phase 1 of the co-design process (defining the problem from the perspective of the end users), the investigators will introduce team members to the project and to the concepts of health care coproduction and shared decision-making. Team members will brainstorm facilitators of and barriers to shared decision-making and optimal care planning. In phase 2 (understanding the context of use and lived experience), co-design teams will discuss current clinical workflows for patient-reported outcome assessment and shared decision-making, brainstorm challenges and opportunities for solutions, and document end users’ priorities for the dashboard’s design and features. In phase 3 (building a design consensus), co-design meetings will focus on drafting, discussing, and revising functional mock-ups of the dashboard, including design specifications and which clinical and patient-reported outcomes should be incorporated. Although patients will inform the specific symptoms and patient-reported outcomes that are important to assess, the investigators will select the appropriate measures for each outcome based on their measurement expertise. These will include measures from the Functional Assessment of Chronic Illness Therapy [ ], PROMIS [ ], and the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) [ ]. In phase 4 (establishing and pilot-testing design specifications), co-design teams will provide iterative user experience feedback on the dashboard until its usability, feasibility, and compatibility with the electronic health record and clinical workflow systems are optimized. After each co-design meeting, the participants will be sent a brief survey via email to provide additional feedback.
In parallel to these intensive co-design team meetings, we will also hold supplemental focus groups. The primary purpose of these focus groups is to validate the evolving consensus and work product of the co-design teams with additional stakeholders. In tandem with phases 1 and 2 of co-design, we will conduct a focus group with 24 patients and care partners to gather an initial assessment of attitudes, beliefs, and perceptions to inform the dashboard’s development. In tandem with phase 3 of co-design, we will conduct focus groups with a total of 72 stakeholders separated by disease group (cancer vs chronic kidney disease) and role (patients and care partners vs clinicians) to provide additional feedback on the appropriateness and desirability of the proposed dashboard design and content. In tandem with phase 4 of co-design, we will conduct focus groups to confirm the acceptability and usability of the dashboard (n=16-24 patients and care partners per disease group). Trained members of the research team will facilitate the discussions according to guides containing semistructured questions to follow in each session. Research staff will take notes during these sessions, which will also be audio recorded in case project staff are unable to manually document all responses.
The primary outcome of the co-design process is to build a fully functional dashboard that can be implemented for clinical testing in objective 2 of this study. However, we also anticipate the development of a rich source of quantitative and qualitative data on the dynamics of development and implementation outcomes . Data sources will include recordings and notes from the co-design meetings, focus groups, and postmeeting surveys. Appropriateness will capture stakeholders’ perceptions of the compatibility between the proposed dashboard and the target clinical setting’s goals and needs, whereas feasibility will assess perceptions of whether and how it can be successfully implemented into the clinical infrastructure. These 2 outcomes will be captured qualitatively through phase 1 and 2 co-design meetings and focus groups as well as through quantitative and qualitative responses provided by clinicians in a prelaunch survey conducted just before dashboard implementation. The acceptability of the near-final dashboard will reflect stakeholders’ satisfaction with its design, content, and functionality. This outcome will be captured qualitatively through feedback provided during co-design meetings, co-design postmeeting surveys, and phase 4 focus groups’ usability evaluations of the co-designed dashboard. The fidelity of the co-design process, or how well it was carried out as intended, will be assessed quantitatively through responses to postmeeting evaluations sent to each member of the co-design teams.
Each of the outcomes will be summarized through qualitative analysis or descriptive statistics depending on the data type. For qualitative analyses, transcripts and notes from the sessions will be analyzed using a directed content analysis approach  guided by two established implementation science frameworks: the Consolidated Framework for Implementation Research [ ] and the Reach, Effectiveness, Adoption, Implementation, and Maintenance framework [ ]. We will use a combination of traditional qualitative analysis and rapid qualitative analysis [ , ], which will allow investigators to provide rapid feedback to operational partners to inform the dashboard building and implementation processes. Across the study phases, qualitative analyses will be conducted by approximately 5 to 7 coders trained in qualitative methods and relevant implementation frameworks. The team will establish coding frameworks or dictionaries (eg, by all coding an initial transcript, discussing and refining codes, and reaching a consensus about themes and categories). The remaining transcripts will then be double-coded and supervised by a lead team member. The full team will meet to discuss the final results to resolve any questions and elucidate the resultant themes. These strategies will be used to aggregate and summarize the main points and evaluate the relevant implementation outcomes (appropriateness, feasibility, and acceptability).
Objective 2: Demonstration Trial
The second objective of this study will include testing the co-designed dashboard in the flow of clinical care service delivery. First, as a quality improvement initiative, the dashboard will be implemented at the clinic level with all eligible patients who receive care from participating clinicians. This step will allow us to collect clinic-wide quality metrics associated with dashboard implementation. Second, the project will include a longitudinal follow-up study among a subset of patients to assess changes in patient outcomes associated with using the dashboard.shows the flow diagram of the dashboard’s clinical implementation and evaluation. provides a schedule of the survey measures completed by the patients at each assessment point of the demonstration trial. These procedures and measures are described in more detail in the following sections.
|Domain and subdomains||Baseline symptom and goals assessment||Subsequent symptom and goals assessments||Dashboard visit evaluation||3-month follow-up||6-month follow-up|
|Sociodemographic data||Age, gender, race, ethnicity, marital status, employment, and education||—b||—||—||—|
|Goals of care||De novo measure||De novo measure||—||—||—|
|Health-related quality of life|
|Health perception||PROMISc Global01 ||PROMIS Global01||—||—||—|
|Global||FACT-G7d ||FACT-Ge GF7f item ||—||FACT-G7||FACT-G7|
|Fatigue||PROMIS 2-item custom SFg ||PROMIS 2-item custom SF||—||PROMIS CATh ||PROMIS CAT|
|Anxiety||PROMIS 2-item custom SF ||PROMIS 2-item custom SF||—||PROMIS CAT ||PROMIS CAT|
|Pain||PROMIS 2-item custom SF ||PROMIS 2-item custom SF||—||PROMIS CAT ||PROMIS CAT|
|Depression||PROMIS 2-item custom SF ||PROMIS 2-item custom SF||—||PROMIS CAT ||PROMIS CAT|
|Physical function||PROMIS 2-item custom SF ||PROMIS 2-item custom SF||—||PROMIS CAT ||PROMIS CAT|
|Shortness of breath||PROMIS DYSSV014 item ||PROMIS DYSSV014 item||—||—||—|
|Urinary frequency (chronic kidney disease only)||PRO-CTCAEi 63a item ||PRO-CTCAE 63a item||—||—||—|
|Edema||PRO-CTCAE 22b item ||PRO-CTCAE 22b item||—||—||—|
|Nausea||PROMIS GISX49 item ||PROMIS GISX49 item||—||—||—|
|Appetite||PROMIS GISX55 item ||PROMIS GISX55 item||—||—||—|
|Itching (chronic kidney disease only)||PROMIS PIQSeverity04 item ||PROMIS PIQSeverity04 item||—||—||—|
|Neuropathy (cancer only)||FACTj and GOG-NTX-4k (version 4) ||FACT and GOG-NTX-4 (version 4)||—||—||—|
|Constipation (cancer only)||PRO-CTCAE 15a item ||PRO-CTCAE 15a item||—||—||—|
|Diarrhea (cancer only)||PROMIS GISX38 item ||PROMIS GISX38 item||—||—||—|
|Side effect bother||FACT-G GP5l item ||FACT-G GP5 item||—||—||—|
|Shared decision-making||CollaboRATE ||—||CollaboRATE||CollaboRATE||CollaboRATE|
|Self-efficacy-managing treatments||PROMIS 4-item custom SF ||—||—||PROMIS 4-item custom SF||PROMIS 4-item custom SF|
|Self-efficacy-managing symptoms||PROMIS 3-item custom SF ||—||—||PROMIS 3-item custom SF||PROMIS 3-item custom SF|
|Treatment satisfaction||FACIT-TSm TS40 item ||—||FACIT-TS TS40 item||FACIT-TS TS40 item||FACIT-TS TS40 item|
|Treatment satisfaction (cancer only)||CAHPSn Cancer Care Survey ||—||—||CAHPS Cancer Care Survey||CAHPS Cancer Care Survey|
|Health care communication||CASEo (information factor) ||—||—||CASE (information factor)||CASE (information factor)|
|Medication adherence||PMASp ||—||—||PMAS||PMAS|
|Social isolation||PROMIS UCLAq 14x2 item ||—||—||PROMIS UCLA 14x2 item||PROMIS UCLA 14x2 item|
|Health literacy||SILSr ||—||—||SILS||SILS|
|Financial toxicity||COSTs-FACITt FT12 item ||—||—||COST-FACIT FT12 item||COST-FACIT FT12 item|
|Usability, acceptability, and adoption||—||—||—||SUSu , SPHEREv [ ], and NoMADw [ ]||SUS, SPHERE, and NoMAD|
|Fidelity of dashboard use||—||—||De novo item||—||—|
|Usability, acceptability, and adoption||—||—||—||SUS, SPHERE, and NoMAD||SUS, SPHERE, and NoMAD|
|Fidelity of dashboard use||—||—||—||De novo item||De novo item|
|Dashboard sustainability||—||—||—||—||CSATx |
aSingle-item names were obtained from the referenced parent measures or item banks.
bEmpty cells indicate that a given domain was not included at that particular assessment point.
cPROMIS: Patient-Reported Outcomes Measurement Information System.
dFACT-G7: Functional Assessment of Cancer Therapy-General, 7-item version.
eFACT-G: Functional Assessment of Cancer Therapy-General.
fGF7: FACT-G global quality of life item.
gSF: short form.
hCAT: computerized adaptive test.
iPRO-CTCAE: Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events.
jFACT: Functional Assessment of Cancer Therapy.
kGOG-NTX-4: Gynecologic Oncology Group-Neurotoxicity.
lFACT-G GP5: FACT-G side effect bother item.
mFACIT-TS: Functional Assessment of Chronic Illness Therapy-Treatment Satisfaction.
nCAHPS: Consumer Assessment of Health Plans Study.
oCASE: Communication and Attitudinal Self-Efficacy scale.
pPMAS: PROMIS Medication Adherence Scale.
qUCLA: University of California, Los Angeles.
rSILS: Single Item Literacy Screener.
sCOST: Comprehensive Score for Financial Toxicity.
tFACIT: Functional Assessment of Chronic Illness Therapy.
uSUS: System Usability Scale.
vSPHERE: Stroke Prevention in Healthcare Delivery Environments study.
wNoMAD: Normalization Measure Development.
xCSAT: Clinical Sustainability Assessment Tool.
Clinic-Wide Dashboard Implementation
The co-designed dashboard will be enabled in the health system’s electronic health record software within the clinical departments of all participating clinicians. The first month of dashboard implementation will constitute a “soft launch” period where participating clinicians and their eligible patients will be trained on using the dashboard and will work with the project team to troubleshoot and resolve any technical and practical issues. After the soft launch period, clinicians will be encouraged to use the dashboard during clinical encounters with all their patients who meet the inclusion criteria (as described previously in objective 1). As outlined in, eligible patients will receive a baseline symptom and goals assessment (see for specific measures) through the electronic patient portal approximately 72 hours before their initial dashboard-eligible visit. If a patient does not respond to the symptom and goals assessment at home, the study staff will assist patients to complete it over the telephone or in-clinic before the appointment. Responses to the symptom and goals assessment, in addition to their most recent results on other clinical data, will populate the dashboard in real time. Clinicians will use the dashboard during the upcoming visit to improve communication and shared decision-making with patients. Following the visit, patients will receive a dashboard visit evaluation survey through the electronic patient portal to provide feedback on their care experience and the shared decision-making process that occurred during their visit. This process will be repeated for any subsequent visit that a patient has with their participating clinician during the study period.
The dashboard will display results from the patient’s most recent symptom and goals assessment and other clinical measures stored in the electronic health record. The included assessments will be determined through extensive stakeholder input during the dashboard co-design process described in objective 1. The symptom and goals assessment will include a mix of items and scales selected to optimize clinical relevance and feasibility. For instance, each assessment will include the PROMIS Global01 health perception item , the Functional Assessment of Cancer Therapy-General (FACT-G) GF7 global quality of life item [ ], and the FACT-G GP5 side effect bother item [ ]. Two-item PROMIS custom short forms will assess anxiety [ ], depression [ ], fatigue [ ], pain [ ], and physical function [ ]. Additional symptoms will mostly be assessed with single items, for example, the PROMIS DYSSV014 dyspnea item [ ], the PRO-CTCAE 63a urinary frequency item (chronic kidney disease only) [ ], or the PRO-CTCAE 15a constipation item (cancer only) [ ]. provides a complete list of health-related quality of life measures to be included in each symptom and goals assessment, and the results will be displayed within the dashboard in a user-friendly format.
In addition to rating health-related quality of life domains, patients will respond to 5 open-response questions in the symptom and goals assessment regarding their goals of care that will populate the dashboard. These questions will prompt patients to specify (1) the top 1 or 2 concerns they would like to discuss during the visit, (2) their most concerning side effects, (3) overall goals regarding their cancer or kidney disease treatment, (4) personal goals and values, and (5) how they can work together with their care team to achieve their goals. Finally, the dashboard will display the most recent results on other clinical measures typically collected during the care process and stored within the electronic health record. For patients with cancer, the additional clinical data will include medications, weight, white blood cell count, absolute neutrophils, hemoglobin, albumin, and an appointment schedule. For patients with chronic kidney disease, the additional clinical data will include weight, blood pressure, glomerular filtration rate, hemoglobin A1c, microalbuminuria, urine protein, and an appointment schedule. Patients and clinicians will view the patient’s individualized dashboard on a computer screen during the visits. Patients will also be able to view their dashboard in between visits using their own devices (eg, computer, tablet, and mobile phone) that can access their electronic patient portal.
Longitudinal Follow-up Study
To assess the longitudinal changes in our primary and secondary study outcomes, a subset of approximately 200 patients will participate in a longitudinal follow-up study. Patients will be eligible to participate if they meet all other study inclusion criteria, have completed at least two dashboard-eligible visits, and provide informed consent to complete a follow-up survey at 3 and 6 months. In parallel to collecting these follow-up patient outcomes, all participating clinicians will also complete surveys at 3 and 6 months to report on their experience with the dashboard’s implementation (). All survey data collected during the longitudinal follow-up study will be collected outside the electronic patient portal using a web-based survey platform (REDCap) [ ].
Dashboard-Naïve Comparison Group
For analyses focusing on health service use (see outcome description in the Health Service Use section), we will identify two cohorts of patients who were not exposed to the dashboard: (1) a cohort of eligible patients with cancer and chronic kidney disease who received care concurrently from participating Northwestern Medicine clinicians but who did not enroll in the dashboard study and (2) a matched cohort of patients with cancer and chronic kidney disease treated contemporaneously by nonparticipating Northwestern Medicine clinicians. Health service use variables will be the only study outcome data available for these 2 comparison groups.
Primary Study Outcome
The central outcome of the study is shared decision-making in the treatment of patients with advanced cancer or chronic kidney disease. We will use the CollaboRATE measure  to assess patients’ perceptions of the quality of shared decision-making occurring in patient-clinician interactions. Patients will complete the measure at baseline, 3-month follow-up, and 6-month follow-up. In addition, they will respond to the CollaboRATE measure in each dashboard visit evaluation survey to assess the perceived quality of shared decision-making that occurred during each dashboard visit. CollaboRATE is a validated 3-item measure that was developed with significant patient input and asks patients to rate the extent to which their clinicians helped them understand health issues, how much effort was made to listen to their priorities, and how much effort was made to include their priorities in treatment selection. We will use the 5-category response scale version (0=“No effort was made,” 1=“A little effort was made,” 2=“Some effort was made,” 3=“A lot of effort was made,” and 4=“Every effort was made”). Scores will be generated by summing responses to the 3 items and generating a score ranging from 0 to 12, with higher scores indicating greater shared decision-making. The CollaboRATE measure is generic and, therefore, it is appropriate for use with patients with various chronic conditions.
Secondary Study Outcomes
Satisfaction With Health Care Quality
Satisfaction with health care will be assessed at baseline, 3 months, and 6 months with a single item from the Functional Assessment of Chronic Illness Therapy-Treatment Satisfaction measure . Patients will also respond to the item in each dashboard visit evaluation survey to assess their satisfaction with the dashboard visit. Patients will complete item TS40—“How do you rate the care you received?”—on a scale of 0 (poor) to 4 (excellent). The scale has met psychometric standards for a variety of chronic health conditions [ ]. In addition, 5 items from the Consumer Assessment of Health Plans Study Cancer Care Survey, Drug Therapy Version [ ], will be assessed at baseline, 3 months, and 6 months among patients with cancer only. These items measure patients’ experiences with cancer care. For example, the item “In the last 6 months, did your drug therapy team advise you about or help you deal with these changes in your energy levels?” is rated as 1 (Yes, definitely), 2 (Yes, somewhat), or 3 (No).
Self-efficacy for Managing Chronic Conditions
A total of 2 subdomains of the PROMIS Self-Efficacy for Managing Chronic Conditions domain will be assessed at baseline and at the 3- and 6-month follow-ups. A 4-item custom short form created from the PROMIS Self-Efficacy for Managing Medications and Treatment Item Bank version 1.0  will assess patients’ confidence in their ability to follow treatment and medication plans. A 3-item custom short form created from the PROMIS Self-Efficacy for Managing Chronic Conditions-Managing Symptoms Item Bank version 1.0 [ ] will assess confidence in their ability to manage their symptoms outside of their health care encounters. Sample items include “I can fit my medication schedule into my daily routine” (self-efficacy for managing treatment and medication) and “I can manage symptoms when I am at home” (self-efficacy for managing symptoms). Similar to all PROMIS measures, these measures are scored on a T-score metric with a mean of 50 and SD of 10. Higher scores indicate more of the construct being measured. These PROMIS measures have all demonstrated reliability, precision, and construct validity based on their correlation with legacy instruments [ ].
Health-Related Quality of Life
To evaluate changes in health-related quality of life as an outcome associated with using the dashboard, a subset of health-related quality of life domains included in the symptom and goals assessment will be assessed again at the 3- and 6-month follow-ups (seefor a schedule of assessments). To assess changes in global health-related quality of life, patients will respond to the 7-item version of the FACT-G [ ] in the baseline symptom assessment and in the 3- and 6-month follow-up surveys. The 7-item version of the FACT-G was derived from the 28-item version of the FACT-G [ ] to offer a brief yet comprehensive assessment of multiple health-related quality of life domains relevant to patients with cancer. It has demonstrated adequate to good reliability, evidence of construct validity (convergent and known groups), and responsiveness to changes in health associated with an intervention [ , ]. To evaluate changes from baseline in core symptoms, the 3- and 6-month follow-up surveys will also contain computerized adaptive tests of the PROMIS anxiety [ ], depression [ ], fatigue [ ], pain [ ], and physical function [ ] item banks.
Health Service Use
After study completion, we will retrospectively extract data on the use of potentially avoidable, high-cost, or low-value health care services from the periods 6 months before and 6 months after the first dashboard use. Specific indicators will include (1) unplanned all-cause hospital admissions, (2) potentially avoidable all-cause emergency department use, (3) excess (all-cause) days in acute care within 30 days of hospital discharge, and (4) 7-day readmissions. In addition, among patients with cancer, we will assess the following disease-specific indicators: (5) admissions and emergency department visits for patients receiving outpatient chemotherapy, (6) chemotherapy within the last 14 days of life, (7) use of a triage clinic, (8) completion of an advance directive, and (9) hospice use of >3 days. Among patients with chronic kidney disease, the following additional indicators will be collected: (10) use of emergency start dialysis; (11) chronic kidney disease–related emergency department or inpatient use; and (12) progression from chronic kidney disease stage 3 to stage 4, stage 4 to stage 5, or stage 3 to stage 5.
Patient- and clinician-reported survey measures will capture relevant implementation outcomes  of the demonstration trial supplemented by data from the electronic health record and qualitative data gathered by the study staff throughout the project. To assess the project’s reach, we will extract data from the electronic health record on patient enrollment rates in the demonstration trial and response rates to the symptom and goals assessments. To assess the fidelity of the dashboard’s use during the study period, patients will report whether the dashboard was discussed during their eligible clinical encounter in each dashboard visit evaluation survey, which is completed immediately after the appointment. Moreover, the study staff will conduct regular clinician observations to assess whether the dashboard is being used consistently and as intended. To assess adoption, clinicians will self-report in their 3- and 6-month surveys the frequency with which they use the dashboard with eligible patients. To assess usability, acceptability, and adoption, both patients and clinicians will rate the dashboard at the 3- and 6-month follow-ups on the System Usability Scale [ ], the Stroke Prevention in Healthcare Delivery Environments study acceptability measure [ ], and the Normalization Measure Development measure of adoption of new health care elements [ ]. Finally, clinicians will rate the perceived sustainability of the clinical dashboard at the 6-month follow-up using the Clinical Sustainability Assessment Tool [ ]. In addition to these implementation outcomes, we will closely document the implementation process, including strategies that were used to successfully carry out the project and implement the dashboard. These may include, but are not limited to, strategies for stakeholder engagement and co-design, patient recruitment and outreach, and training resources for using the dashboard to facilitate shared decision-making. Quantitative data will be analyzed using descriptive statistics, and qualitative data will be analyzed using the approaches described in objective 1.
For all statistical tests, a nominal 2-sided P value of <.05 will be considered statistically significant. Unless otherwise noted, analyses will be conducted on the full analysis set and stratified by populations with chronic kidney disease versus cancer.
The primary analyses for objective 2 are within-group mean changes in shared decision-making (CollaboRATE), self-efficacy for managing chronic conditions (PROMIS), satisfaction with care (Functional Assessment of Chronic Illness Therapy-Treatment Satisfaction and Consumer Assessment of Health Plans Study Cancer Care Survey), and health-related quality of life and symptoms (PROMIS, Functional Assessment of Chronic Illness Therapy, and PRO-CTCAE). First, we will test whether within-group means have changed significantly from the baseline and symptom assessments (as appropriate) to the 3-month follow-up (primary) and 6-month follow-up (secondary) using paired samples t tests. We will supplement the paired t tests with multivariable regression models to adjust for key covariates (eg, demographic and clinical characteristics). Regarding health service use, we will first use paired-sample t tests to compare rates in the occurrence of each indicator described in the Health Service Use outcome section between the 6 months before and after the date of first dashboard exposure. On a secondary basis, we will model these outcomes as count variables using zero-inflated Poisson regression to adjust for covariates [, ]. Finally, we will compare rates of health service use before and after the first dashboard exposure between patients exposed to the dashboard and the naïve comparison cohorts using 2-sample t tests and zero-inflated Poisson regression. To do so, we will use propensity score matching to create comparable cohorts of patients with and without exposure to the dashboard. We will match on age, sex, cancer type (where applicable), and disease severity or staging [ ].
Power and Sample Size Calculations
Sample size considerations for objective 2 were informed by statistical power analyses for differences in within-group changes in mean responses to the CollaboRATE shared decision-making measure and PROMIS measures from baseline to the 3-month follow-up. Within-group mean changes in the CollaboRATE 5-point response scale version ranged from approximately 2.0 to 4.5, with SDs of approximately 3.5 . For the CollaboRATE measure, we conducted a statistical power analysis making the following conservative assumptions: (1) a correlation between baseline and 3-month follow-up scores of r=0.30; (2) a common SD of 3.5 points; (3) a 2-sided P value of .05; and (4) mean changes of 1.0, 1.5, and 2.0 points. These analyses suggested that sample sizes of 137, 62, and 36 patients would be needed to detect mean changes of 1.0, 1.5, and 2.0 points, respectively, using a paired t test.
As noted in the Methods section, all PROMIS measures are scored as a T-score metric with a mean of 50 and SD of 10. Changes as low as 3 points can be clinically meaningful . For power analysis regarding PROMIS measures, we made the following conservative assumptions: (1) a correlation between baseline and 3-month follow-up scores of r=0.50; (2) a common SD of 10.0 points; (3) a 2-sided P value of .05; and (4) mean changes of 3.0, 4.0, and 5.0 points. These analyses suggested that sample sizes of 90, 52, and 34 patients would be needed to detect mean changes of 3.0, 4.0, and 5.0 points, respectively, using a paired t test.
The co-design sessions (objective 1), which focused on collaboratively designing the dashboard’s content and format for initial clinical testing, concluded in May 2020. Focus groups supplementing the co-design sessions were completed in October 2019, February 2020, and October 2021. Data collection for the demonstration trial is anticipated to be completed by the end of July 2022. Study investigators currently meet monthly as an entire team to discuss progress on manuscript development for results dissemination. Each study objective will have at least one resulting publication: (1) co-design process and outcomes and (2) demonstration trial process and outcomes.
In this study, we hypothesize that co-designing and implementing a patient-reported outcome clinical dashboard will be associated with improved processes underlying health care coproduction among adults with advanced cancer or chronic kidney disease, including shared decision-making, satisfaction with care, engagement in health care, self-efficacy in managing symptoms and treatments, health-related quality of life, and use of health services. Collaborating with patients, care partners, and clinicians to create scalable clinical dashboards can help improve coproduction outcomes [, ]. Thus, this study draws upon a co-design framework to actively engage patients and care partners in the iterative design and evaluation of a patient-reported outcome dashboard to be used during clinical encounters with patients with advanced cancer and chronic kidney disease. However, few previous studies have attempted to improve the coproduction process by integrating patient-reported outcomes with other electronic health record–linked clinical results in a data visualization platform to facilitate shared decision-making [ , , ]. In particular, our dashboard aims to improve recognition and shared decision-making for the management of patients’ symptoms and treatment needs, which can promote more effective use of the health care system and translate into improved patient outcomes [ , , , ].
The dashboard’s design and evaluation processes have several strengths worth noting. First, the dashboard will be co-designed by investigators and key stakeholders (patients, care partners, and clinic staff). Collaborating with stakeholders promotes person-centered care delivery and increases the likelihood that a new health care element can be successfully implemented and optimally impactful . Unfortunately, stakeholder engagement is often skipped when developing interventions owing to time and money constraints [ ]. Second, we will carefully document each step of the dashboard design and implementation process, including any unanticipated modifications needed for improving its acceptability, feasibility, and adoption. This will ensure that the dashboard can be easily disseminated to other investigators and adopted by other health care systems and that the study can be replicated. Third, the dashboard is designed to be dynamic and individualized. It will be fully integrated within the electronic health record, providing a more complete picture of a patient’s health status and facilitating informed decision-making and treatment planning by displaying patient-reported outcomes along with other clinical data. New data populate the dashboard in real time so that health care can efficiently adapt to patients’ changing symptom and care needs. Fourth, patients can view the dashboard at any time through their electronic patient portal, which will display symptom scores over time in user-friendly graphs. This will allow patients who do not complete regular health care visits to still experience benefits through enhanced self-reflection and self-monitoring of personalized symptom feedback even if explicit health care decision-making does not occur. In summary, our patient-centered dashboard and design process provide an efficient, feasible, and scalable model for integrating patient-reported outcome data collection into person-centered, value-based care delivery.
Despite these strengths, this project has limitations that warrant discussion. First, the study will focus only on patients in the advanced stages of 2 chronic conditions and on those who regularly access their electronic patient portals. Although advanced cancer and chronic kidney disease are two of the most common and burdensome illnesses to individuals and the health care system [, , ], we are actively expanding the dashboard into other areas of medicine such as rheumatology. Second, the electronic health record software in which the dashboard will be developed has limitations in how data are visualized, which limits the opportunities for user-centered design. Nonetheless, integration into the electronic health record will allow us to improve other aspects of the user experience, including reducing the clinical burden that might be associated with using an external system, providing a shared dashboard environment that is accessible to both patients and clinicians, and allowing patient-reported outcome results to be integrated with other routinely collected health information. Third, the dashboard will be evaluated using a single-arm design, precluding the ability to draw causal conclusions about any observed changes in patient outcomes during the demonstration project. Fourth, the dashboard and study process will only be available in English and may underrepresent people with low health literacy and less digital acumen or computer access. Future efforts will need to adapt the dashboard to other languages, chronic illness groups, and health ITs represented in the local health care system. There may also be an opportunity in the future to conduct a formal randomized controlled trial of the dashboard’s efficacy.
This study demonstrates how to combine stakeholder engagement, a health care coproduction framework, and health IT to develop a clinically feasible, acceptable, and scalable model of patient-centered care delivery. This study’s premise integrates emerging research indicating that better patient outcomes and appropriate health care use can be enhanced through patient-clinician collaboration in the context of well-designed workflows and systems. Thus, this study focuses on dealing with the manifold challenges of implementing significant changes in workflow and clinical approach in the real world of ambulatory care of patients with a serious chronic illness. We aim to build on implementation research to understand the needed adoption, acceptability, and feasibility of new tools to optimize a successful and sustainable transformation of the clinical encounter over time. Our work serves as an example for transforming care delivery in a way that embraces value-based care and has the potential to improve the lives of patients with chronic conditions.
This publication was made possible by a grant from the Peterson Center on Healthcare. The statements made and views expressed are solely the responsibility of the authors. LMP was supported by the National Institutes of Health/National Cancer Institute training grant T32CA193193.
Although we have collected data for this study, this manuscript does not report any results. Our intention is to share data associated with the published results as they emerge.
Conflicts of Interest
AH is a health care consultant for the Yale New Haven Health System Center for Outcomes Research and Evaluation, where she is involved in re-evaluation activities for the Centers for Medicare and Medicaid value-based payment programs. Neither the Yale New Haven Health System Center for Outcomes Research and Evaluation nor the Centers for Medicare and Medicaid provided support or funding for this work.
- Boersma P, Black LI, Ward BW. Prevalence of multiple chronic conditions among US adults, 2018. Prev Chronic Dis 2020 Sep 17;17:E106 [FREE Full text] [CrossRef] [Medline]
- Buttorff C, Ruder T, Bauman M. Multiple chronic conditions in the United States. RAND Corporation. 2017. URL: https://www.rand.org/pubs/tools/TL221.html [accessed 2022-08-18]
- Jhamb M, Abdel-Kader K, Yabes J, Wang Y, Weisbord SD, Unruh M, et al. Comparison of fatigue, pain, and depression in patients with advanced kidney disease and cancer-symptom burden and clusters. J Pain Symp Manag 2019 Mar;57(3):566-75.e3 [FREE Full text] [CrossRef] [Medline]
- Dhingra L, Barrett M, Knotkova H, Chen J, Riggs A, Lee B, et al. Symptom distress among diverse patients referred for community-based palliative care: sociodemographic and medical correlates. J Pain Symp Manag 2018 Feb;55(2):290-296 [FREE Full text] [CrossRef] [Medline]
- Henchoz Y, Büla C, Guessous I, Rodondi N, Goy R, Demont M, et al. Chronic symptoms in a representative sample of community-dwelling older people: a cross-sectional study in Switzerland. BMJ Open 2017 Jan 17;7(1):e014485 [FREE Full text] [CrossRef] [Medline]
- Pearman T, Garcia S, Penedo F, Yanez B, Wagner L, Cella D. Implementation of distress screening in an oncology setting. J Community Support Oncol 2015 Dec;13(12):423-428. [CrossRef] [Medline]
- NCCN practice guidelines for the management of psychosocial distress. National Comprehensive Cancer Network. Oncology (Williston Park) 1999 May;13(5A):113-147. [Medline]
- Hsia R, Niedzwiecki M. Avoidable emergency department visits: a starting point. Int J Qual Health Care 2017 Oct 01;29(5):642-645. [CrossRef] [Medline]
- Giannouchos TV, Kum H, Foster MJ, Ohsfeldt RL. Characteristics and predictors of adult frequent emergency department users in the United States: a systematic literature review. J Eval Clin Pract 2019 Jun 02;25(3):420-433. [CrossRef] [Medline]
- Basch E, Wilfong L, Schrag D. Adding patient-reported outcomes to medicare's oncology value-based payment model. JAMA 2020 Jan 21;323(3):213-214. [CrossRef] [Medline]
- Diamond LH, Howard AD. The ESRD quality incentive program: the current limitations of evidence and data to develop measures, drive improvement, and incentivize outcomes. Adv Chronic Kidney Dis 2016 Nov;23(6):377-384. [CrossRef] [Medline]
- National Comprehensive Cancer Network. Distress management. Clinical practice guidelines. J Natl Compr Canc Netw 2003 Jul;1(3):344-374. [CrossRef] [Medline]
- Commission on Cancer. American College of Surgeons. URL: https://tinyurl.com/da3pzaxy [accessed 2022-08-18]
- Penedo FJ, Oswald LB, Kronenfeld JP, Garcia SF, Cella D, Yanez B. The increasing value of eHealth in the delivery of patient-centred cancer care. Lancet Oncol 2020 May;21(5):e240-e251. [CrossRef]
- Gandrup J, Ali SM, McBeth J, van der Veer SN, Dixon WG. Remote symptom monitoring integrated into electronic health records: a systematic review. J Am Med Inform Assoc 2020 Nov 01;27(11):1752-1763 [FREE Full text] [CrossRef] [Medline]
- Elwyn G, Nelson E, Hager A, Price A. Coproduction: when users define quality. BMJ Qual Saf 2020 Sep 05;29(9):711-716 [FREE Full text] [CrossRef] [Medline]
- Nelson EC, Dixon-Woods M, Batalden PB, Homa K, Van Citters AD, Morgan TS, et al. Patient focused registries can improve health, care, and science. BMJ 2016 Jul 01;354:i3319 [FREE Full text] [CrossRef] [Medline]
- Oliver B, Nelson EC, Kerrigan CL. Turning feed-forward and feedback processes on patient-reported data into intelligent action and informed decision-making: case studies and principles. Med Care 2019 May;57 Suppl 5 Suppl 1:S31-S37. [CrossRef] [Medline]
- Howell D, Mayer DK, Fielding R, Eicher M, Verdonck-de Leeuw IM, Johansen C, Global Partners for Self-Management in Cancer. Management of cancer and health after the clinic visit: a call to action for self-management in cancer care. J Natl Cancer Inst 2021 May 04;113(5):523-531 [FREE Full text] [CrossRef] [Medline]
- Kalantar-Zadeh K, Jafar TH, Nitsch D, Neuen BL, Perkovic V. Chronic kidney disease. Lancet 2021 Aug;398(10302):786-802. [CrossRef]
- Kaye DR, Min HS, Norton EC, Ye Z, Li J, Dupree JM, et al. System-level health-care integration and the costs of cancer care across the disease continuum. J Oncol Pract 2018 Mar;14(3):e149-e157. [CrossRef]
- Mariotto AB, Yabroff KR, Shao Y, Feuer EJ, Brown ML. Projections of the cost of cancer care in the United States: 2010-2020. J Natl Cancer Inst 2011 Jan 19;103(2):117-128 [FREE Full text] [CrossRef] [Medline]
- Saran R, Li Y, Robinson B, Abbott KC, Agodoa LY, Ayanian J, et al. US renal data system 2015 annual data report: epidemiology of kidney disease in the United States. Am J Kidney Dis 2016 Mar;67(3 Suppl 1):Svii, S1-Svii,305 [FREE Full text] [CrossRef] [Medline]
- Reyna VF, Nelson WL, Han PK, Pignone MP. Decision making and cancer. Am Psychol 2015;70(2):105-118 [FREE Full text] [CrossRef] [Medline]
- Loiselle M, Michaud C, O'Connor A. Decisional needs assessment to help patients with advanced chronic kidney disease make better dialysis choices. Nephrol Nurs J 2016;43(6):463-493. [Medline]
- Martin S, Gerhart JI, Rochefort C, Perry L, Hoerger M. Affective forecasting in health psychology. In: The Wiley Encyclopedia of Health Psychology. Hoboken, New Jersey, United States: Wiley; 2020.
- Sypes EE, de Grood C, Whalen-Browne L, Clement FM, Parsons Leigh J, Niven DJ, et al. Engaging patients in de-implementation interventions to reduce low-value clinical care: a systematic review and meta-analysis. BMC Med 2020 May 08;18(1):116 [FREE Full text] [CrossRef] [Medline]
- Batalden M, Batalden P, Margolis P, Seid M, Armstrong G, Opipari-Arrigan L, et al. Coproduction of healthcare service. BMJ Qual Saf 2016 Jul 16;25(7):509-517 [FREE Full text] [CrossRef] [Medline]
- Lindblad S, Ernestam S, Van Citters A, Lind C, Morgan T, Nelson E. Creating a culture of health: evolving healthcare systems and patient engagement. QJM 2017 Mar 01;110(3):125-129. [CrossRef] [Medline]
- Gremyr A. Improving health with and for individuals with schizophrenia using a learning health system approach: from idea to daily practice. Jönköping University, School of Health and Welfare. 2022. URL: https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A1652999&dswid=5300 [accessed 2022-08-18]
- Van Citters AD, Holthoff MM, Kennedy AM, Melmed GY, Oberai R, Siegel CA, et al. Point-of-care dashboards promote coproduction of healthcare services for patients with inflammatory bowel disease. Int J Qual Health Care 2021 Nov 29;33(Supplement_2):ii40-ii47. [CrossRef] [Medline]
- Van Citters AD, Gifford AH, Brady C, Dunitz JM, Elmhirst M, Flath J, et al. Formative evaluation of a dashboard to support coproduction of healthcare services in cystic fibrosis. J Cyst Fibros 2020 Sep;19(5):768-776. [CrossRef] [Medline]
- Rabin B, Glasgow RE. An implementation science perspective on psychological science and cancer: what is known and opportunities for research, policy, and practice. Am Psychol 2015 Feb;70(2):211-220. [CrossRef] [Medline]
- Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L, et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc 2015 Jan;22(1):43-50 [FREE Full text] [CrossRef] [Medline]
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 Apr;42(2):377-381 [FREE Full text] [CrossRef] [Medline]
- Garcia SF, Wortman K, Cella D, Wagner LI, Bass M, Kircher S, et al. Implementing electronic health record-integrated screening of patient-reported symptoms and supportive care needs in a comprehensive cancer center. Cancer 2019 Nov 15;125(22):4059-4068 [FREE Full text] [CrossRef] [Medline]
- Tips and Tools. The Dartmouth Institute for Health Policy & Clinical Practice. URL: https://sites.dartmouth.edu/coproduction/knowledge-center/tools/ [accessed 2022-08-18]
- Attribution 4.0 International (CC BY 4.0). Attribution 4.0 International (CC BY 4.0). Creative Commons. URL: https://creativecommons.org/licenses/by/4.0/ [accessed 2022-08-23]
- Cella DF, Tulsky DS, Gray G, Sarafian B, Linn E, Bonomi A, et al. The functional assessment of cancer therapy scale: development and validation of the general measure. J Clin Oncol 1993 Mar;11(3):570-579. [CrossRef]
- Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, PROMIS Cooperative Group. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol 2010 Nov;63(11):1179-1194 [FREE Full text] [CrossRef] [Medline]
- Dueck AC, Mendoza TR, Mitchell SA, Reeve BB, Castro KM, Rogak LJ, National Cancer Institute PRO-CTCAE Study Group. Validity and reliability of the US national cancer institute's patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). JAMA Oncol 2015 Nov;1(8):1051-1059 [FREE Full text] [CrossRef] [Medline]
- Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar 19;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
- Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005 Nov 01;15(9):1277-1288. [CrossRef] [Medline]
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 07;4(1):50 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [CrossRef] [Medline]
- Neal JW, Neal ZP, VanDyke E, Kornbluh M. Expediting the analysis of qualitative data in evaluation. Am J Eval 2014 May 27;36(1):118-132. [CrossRef]
- Nevedal AL, Reardon CM, Opra Widerquist MA, Jackson GL, Cutrona SL, White BS, et al. Rapid versus traditional qualitative analysis using the consolidated framework for implementation research (CFIR). Implement Sci 2021 Jul 02;16(1):67 [FREE Full text] [CrossRef] [Medline]
- Yanez B, Pearman T, Lis C, Beaumont J, Cella D. The FACT-G7: a rapid version of the functional assessment of cancer therapy-general (FACT-G) for monitoring symptoms and concerns in oncology practice and research. Ann Oncol 2013 Apr;24(4):1073-1078 [FREE Full text] [CrossRef] [Medline]
- Lai J, Cella D, Choi S, Junghaenel DU, Christodoulou C, Gershon R, et al. How item banks and their application can influence measurement practice in rehabilitation medicine: a PROMIS fatigue item bank example. Arch Phys Med Rehabil 2011 Oct;92(10 Suppl):S20-S27 [FREE Full text] [CrossRef] [Medline]
- Pilkonis PA, Choi SW, Reise SP, Stover AM, Riley WT, Cella D, PROMIS Cooperative Group. Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS®): depression, anxiety, and anger. Assessment 2011 Sep 21;18(3):263-283 [FREE Full text] [CrossRef] [Medline]
- Amtmann D, Cook KF, Jensen MP, Chen WH, Choi S, Revicki D, et al. Development of a PROMIS item bank to measure pain interference. Pain 2010 Jul;150(1):173-182 [FREE Full text] [CrossRef] [Medline]
- Schalet B. Extending the ceiling of an item bank: development of above-average physical function items for PROMIS. In: Proceedings of the International Society for Quality of Life Research. 2016 Presented at: International Society for Quality of Life Research; October, 2016; Copenhagen, Denmark.
- Choi SW, Victorson DE, Yount S, Anton S, Cella D. Development of a conceptual framework and calibrated item banks to measure patient-reported dyspnea severity and related functional limitations. Value Health 2011 Mar;14(2):291-306 [FREE Full text] [CrossRef] [Medline]
- Spiegel B, Hays RD, Bolus R, Melmed GY, Chang L, Whitman C, et al. Development of the NIH Patient-Reported Outcomes Measurement Information System (PROMIS) gastrointestinal symptom scales. Am J Gastroenterol 2014 Nov;109(11):1804-1814 [FREE Full text] [CrossRef] [Medline]
- Silverberg JI, Lai J, Kantor RW, Dalal P, Hickey C, Shaunfield S, et al. Development, validation, and interpretation of the PROMIS Itch questionnaire: a patient-reported outcome measure for the quality of life impact of Itch. J Invest Dermatol 2020 May;140(5):986-94.e6 [FREE Full text] [CrossRef] [Medline]
- Cheng HL, Lopez V, Lam SC, Leung AK, Li YC, Wong KH, et al. Psychometric testing of the functional assessment of cancer therapy/gynecologic oncology group-neurotoxicity (FACT/GOG-Ntx) subscale in a longitudinal study of cancer patients treated with chemotherapy. Health Qual Life Outcomes 2020 Jul 23;18(1):246 [FREE Full text] [CrossRef] [Medline]
- Elwyn G, Barr PJ, Grande SW, Thompson R, Walsh T, Ozanne EM. Developing CollaboRATE: a fast and frugal patient-reported measure of shared decision making in clinical encounters. Patient Educ Couns 2013 Oct;93(1):102-107 [FREE Full text] [CrossRef] [Medline]
- Gruber-Baldini AL, Velozo C, Romero S, Shulman LM. Validation of the PROMIS measures of self-efficacy for managing chronic conditions. Qual Life Res 2017 Jul 26;26(7):1915-1924 [FREE Full text] [CrossRef] [Medline]
- Peipert JD, Beaumont JL, Bode R, Cella D, Garcia SF, Hahn EA. Development and validation of the functional assessment of chronic illness therapy treatment satisfaction (FACIT TS) measures. Qual Life Res 2014 Apr 24;23(3):815-824. [CrossRef] [Medline]
- Evensen CT, Yost KJ, Keller S, Arora NK, Frentzel E, Cowans T, et al. Development and testing of the CAHPS cancer care survey. J Oncol Pract 2019 Nov;15(11):e969-e978. [CrossRef]
- Wolf MS, Chang C, Davis T, Makoul G. Development and validation of the communication and attitudinal self-efficacy scale for cancer (CASE-cancer). Patient Educ Couns 2005 Jun;57(3):333-341. [CrossRef] [Medline]
- Peipert JD, Badawy SM, Baik SH, Oswald LB, Efficace F, Garcia SF, et al. Development of the NIH Patient-Reported Outcomes Measurement Information System (PROMIS) Medication Adherence Scale (PMAS). Patient Preference Adherence 2020 Jun;14:971-983. [CrossRef]
- Hahn EA, DeWalt DA, Bode RK, Garcia SF, DeVellis RF, Correia H, PROMIS Cooperative Group. New English and Spanish social health measures will facilitate evaluating health determinants. Health Psychol 2014 May 20;33(5):490-499 [FREE Full text] [CrossRef] [Medline]
- Morris NS, MacLean CD, Chew LD, Littenberg B. The Single Item Literacy Screener: evaluation of a brief instrument to identify limited reading ability. BMC Fam Pract 2006 Mar 24;7(1):21 [FREE Full text] [CrossRef] [Medline]
- de Souza JA, Yap BJ, Hlubocky FJ, Wroblewski K, Ratain MJ, Cella D, et al. The development of a financial toxicity patient-reported outcome in cancer: the COST measure. Cancer 2014 Oct 15;120(20):3245-3253 [FREE Full text] [CrossRef] [Medline]
- Lewis JR. The system usability scale: past, present, and future. Int J Human Comput Interact 2018 Mar 30;34(7):577-590. [CrossRef]
- Foraker R, Kite B, Kelley MM, Lai AM, Roth C, Lopetegui MA, et al. EHR-based visualization tool: adoption rates, satisfaction, and patient outcomes. EGEMS (Wash DC) 2015;3(2):1159 [FREE Full text] [CrossRef] [Medline]
- Rapley T, Girling M, Mair FS, Murray E, Treweek S, McColl E, et al. Improving the normalization of complex interventions: part 1 - development of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol 2018 Nov 15;18(1):133 [FREE Full text] [CrossRef] [Medline]
- Malone S, Prewitt K, Hackett R, Lin JC, McKay V, Walsh-Bailey C, et al. The Clinical Sustainability Assessment Tool: measuring organizational capacity to promote sustainability in healthcare. Implement Sci Commun 2021 Jul 17;2(1):77 [FREE Full text] [CrossRef] [Medline]
- Mah K, Swami N, Le LW, Chow R, Hannon BL, Rodin G, et al. Validation of the 7-item Functional Assessment of Cancer Therapy-General (FACT-G7) as a short measure of quality of life in patients with advanced cancer. Cancer 2020 Aug 15;126(16):3750-3757 [FREE Full text] [CrossRef] [Medline]
- Tooze JA, Grunwald GK, Jones RH. Analysis of repeated measures data with clumping at zero. Stat Methods Med Res 2002 Aug 02;11(4):341-355. [CrossRef] [Medline]
- Xie H, McHugo G, Sengupta A, Clark R, Drake R. A method for analyzing longitudinal outcomes with many zeros. Ment Health Serv Res 2004 Dec;6(4):239-246. [CrossRef]
- Rosenbaum PR, Rubin DB. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. Am Statistician 2012 Mar 12;39(1):33-38. [CrossRef]
- Barr PJ, Thompson R, Walsh T, Grande SW, Ozanne EM, Elwyn G. The psychometric properties of CollaboRATE: a fast and frugal patient-reported measure of the shared decision-making process. J Med Internet Res 2014 Jan 03;16(1):e2 [FREE Full text] [CrossRef] [Medline]
- Terwee CB, Peipert JD, Chapman R, Lai J, Terluin B, Cella D, et al. Minimal important change (MIC): a conceptual clarification and systematic review of MIC estimates of PROMIS measures. Qual Life Res 2021 Oct 10;30(10):2729-2754 [FREE Full text] [CrossRef] [Medline]
- Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 2016 Feb 20;34(6):557-565. [CrossRef]
- LeClair AM, Kotzias V, Garlick J, Cole AM, Kwon SC, Lightfoot A, et al. Facilitating stakeholder engagement in early stage translational research. PLoS One 2020 Jul 2;15(7):e0235400 [FREE Full text] [CrossRef] [Medline]
|FACT-G: Functional Assessment of Cancer Therapy-General|
|PRO-CTCAE: Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events|
|PROMIS: Patient-Reported Outcomes Measurement Information System|
|REDCap: Research Electronic Data Capture|
Edited by T Leung; submitted 04.04.22; peer-reviewed by M Singh, A Nordin, H Oh; comments to author 28.06.22; revised version received 18.07.22; accepted 31.07.22; published 21.09.22Copyright
©Laura M Perry, Victoria Morken, John D Peipert, Betina Yanez, Sofia F Garcia, Cynthia Barnard, Lisa R Hirschhorn, Jeffrey A Linder, Neil Jordan, Ronald T Ackermann, Alexandra Harris, Sheetal Kircher, Nisha Mohindra, Vikram Aggarwal, Rebecca Frazier, Ava Coughlin, Katy Bedjeti, Melissa Weitzel, Eugene C Nelson, Glyn Elwyn, Aricca D Van Citters, Mary O'Connor, David Cella. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 21.09.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.