Published on in Vol 5, No 4 (2016): Oct-Dec

A Checklist for the Conduct, Reporting, and Appraisal of Microcosting Studies in Health Care: Protocol Development

A Checklist for the Conduct, Reporting, and Appraisal of Microcosting Studies in Health Care: Protocol Development

A Checklist for the Conduct, Reporting, and Appraisal of Microcosting Studies in Health Care: Protocol Development

Authors of this article:

Jennifer Prah Ruger1, 2 Author Orcid Image ;   Marian Reiff3 Author Orcid Image

Protocol

1School of Social Policy & Practice, University of Pennsylvania, Philadelphia, PA, United States

2Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States

3Department of Medical Ethics & Health Policy, University of Pennsylvania, Philadelphia, PA, United States

Corresponding Author:

Jennifer Prah Ruger, MA, MSL, MSc, PhD

School of Social Policy & Practice

University of Pennsylvania

3701 Locust Walk

Philadelphia, PA, 19104

United States

Phone: 1 215 746 1330

Fax:1 215 573 2099

Email: jenpr@upenn.edu


Background: Microcosting is a cost estimation method that requires the collection of detailed data on resources utilized, and the unit costs of those resources in order to identify actual resource use and economic costs. Microcosting findings reflect the true costs to health care systems and to society, and are able to provide transparent and consistent estimates. Many economic evaluations in health and medicine use charges, prices, or payments as a proxy for cost. However, using charges, prices, or payments rather than the true costs of resources can result in inaccurate estimates. There is currently no existing checklist or guideline for the conduct, reporting, or appraisal of microcosting studies in health care interventions.

Objective: The aim of this study is to create a checklist and guideline for the conduct, reporting, and appraisal of microcosting studies in health care interventions.

Methods: Appropriate potential domains and items will be identified through (1) a systematic review of all published microcosting studies of health and medical interventions, strategies, and programs; (2) review of published checklists and guidelines for economic evaluations of health interventions, and selection of items relevant for microcosting studies; and (3) theoretical analysis of economic concepts relevant for microcosting. Item selection, formulation, and reduction will be conducted by the research team in order to develop an initial pool of items for evaluation by an expert panel comprising individuals with expertise in microcosting and economic evaluation of health interventions. A modified Delphi process will be conducted to achieve consensus on the checklist. A pilot test will be conducted on a selection of the articles selected for the previous systematic review of published microcosting studies.

Results: The project is currently in progress.

Conclusions: Standardization of the methods used to conduct, report or appraise microcosting studies will enhance the consistency, transparency, and comparability of future microcosting studies. This will be the first checklist for microcosting studies to accomplish these goals and will be a timely and important contribution to the health economic and health policy literature. In addition to its usefulness to health economists and researchers, it will also benefit journal editors and decision-makers who require accurate cost estimates to deliver health care.

JMIR Res Protoc 2016;5(4):e195

doi:10.2196/resprot.6263

Keywords



As health care costs increase worldwide, there is growing pressure to more efficiently use our limited health care resources. Economic evaluations are used to identify, measure, and compare the costs of health interventions and programs in order to help make decisions about resource allocation and program implementation [1]. A first step for controlling costs is the accurate measurement of the true costs of health interventions and programs. Many economic evaluations in health and medicine use charges, prices or payments as a proxy for cost. However, using charges, prices or payments rather than the true costs of resources can result in inaccurate estimates [2,3]. Microcosting is a cost estimation method that provides detailed and accurate cost data by direct enumeration and costing of all the resources used in the provision of an intervention [4,5]. In contrast to gross-costing studies, which use reimbursement amounts or charges or aggregate cost estimates, microcosting requires the collection of detailed data on resources utilized and the unit costs of those resources in order to identify actual resource use and economic costs. Microcosting findings more accurately reflect the true costs of an intervention to health care systems and to society, and are able to provide transparent and consistent estimates. Microcosting has been shown to improve the validity and reliability of total cost estimates for hospital services and for diagnostic or treatment interventions where costs are not available or evolving [6-9]. Microcosting involves the direct measurement of cost by observation and survey and is especially useful for identifying the actual costs of new health interventions or programs, when existing administrative data are not sufficiently sensitive or when there are no established estimates for their aggregate costs [10-12]. There is an increasing need for microcosting in decision making in health policy, and it is important that studies are conducted according to consistent principles and that they are reported in a way that allows for transparency and comparability across studies.

The importance of rigor and transparency in reporting of health economic evaluations has been addressed by systematic reviews of economic evaluation studies and the development of standards and guidelines for the conduct and reporting of economic evaluations of health interventions [11,13-16]. However, the existing guidelines and checklists do not provide sufficient detail for the methods and techniques involved in microcosting studies [6,10,11,13,17-19]. These instruments do not provide a methodological framework and analytic components specific to the inclusion of items to evaluate microcosting studies. The value of checklists for improving the quality of studies and reports in health care has been demonstrated [20,21]. However, there is currently no existing checklist or guideline for the conduct, reporting or appraisal of microcosting studies in health care interventions. We propose to develop a formal checklist, informed by a theoretically- and empirically-based framework, for the conduct, reporting, and appraisal of microcosting studies in health care. The checklist will (1) provide a framework and guidance for the conduct of microcosting studies; (2) assist in the development of manuscripts reporting microcosting studies and reviewing the manuscripts for publication; and (3) lead to more consistency and transparency in conducting and reporting of microcosting studies, allowing for comparison of the studies’ findings. Ultimately, this protocol will lead to the development of a checklist for the conduct, reporting, and appraisal of microcosting studies in health care, improving the quality of these studies.


The design of this protocol for the development of a microcosting checklist utilizes recommendations in the Guidance for Developers of Health Research Reporting Guidelines [20,22], and draws on approaches described in published reports of checklist development for reporting and appraisal of economic evaluations of health interventions [13-16,23-29].

The checklist will be developed in the following four stages: (1) identification of appropriate potential domains and items, (2) tem selection, formulation, and reduction, (3) external review (further item reduction and revision), and (4) testing and assessment.

Identification of Potential Checklist Items

To identify the important domains and items to be considered for inclusion in a standardized conducting and reporting guideline for microcosting studies in health care, the following three methods will be used: (1) systematic review of microcosting studies (Method 1), (2) review of checklists and guidelines for economic evaluations of health interventions, and selection of items relevant for microcosting studies (Method 2), and (3) theoretical analysis of economic concepts relevant for microcosting (Method 3). Triangulation of the three methods will produce a preliminary list of items that will be more comprehensive and inclusive than items identified from any one method alone.

Method 1

A systematic review is being conducted of all published microcosting studies of health and medical interventions, strategies, and programs [19]. A comprehensive database has been created, consisting of all microcosting studies published in English. A research objective is to evaluate the quality of published microcosting studies in health care. Details of the search criteria and methodology for data extraction are published elsewhere [19].

The research team will critically assess the quality of each microcosting study included in the systematic review using checklists recommended by the Campbell and Cochrane Economic Methods Group [30] for appraising reporting and methodological quality of economic evaluations. Specifically, the Drummond checklist [31] and the Evers checklist [26] will be used to evaluate the quality and risk of bias of single effectiveness studies; the Philips checklist [32] will be used to evaluate the quality and risk of bias of studies that use decision analytic modeling. The Fukuda and Immanaka criteria [16] will be used to assess the transparency of cost estimates. The Consolidated Health Economic Evaluating Reporting Standards (CHEERS) checklist will be used to assess reporting quality [14]. These criteria categorize studies into levels of transparency based on whether the study clarifies the cost components included, reports the quantity and unit price of resources separately, and reports an estimate of each component. These checklists were employed because they provided the most relevant criteria for assessing economic evaluations including costing, even though they were not developed specifically to assess microcosting studies.

The checklist items will be filled out independently by the two reviewers conducting the systematic review. Disagreements will be discussed and resolved by the two reviewers, and a third researcher will be consulted if needed. The strengths, inadequacies, and redundancies of the existing checklists used to assess quality and bias of the microcosting studies in the systematic review will be documented. Experience with using the existing checklists (ie, Drummond, Evers, Philips, Fukuda, and CHEERS) for study quality and risk of bias in economic evaluations in the systematic review will demonstrate which items are relevant to assessing the quality and reporting of microcosting studies and which are not. Items are scored as “yes”, “no”, “not clear” or “not applicable”. Those that are scored as “not applicable” by all three reviewers for all studies will be excluded. We will document which items in the existing checklists successfully identified relevant criteria for microcosting. Only the relevant items will be selected for consideration for a preliminary list of items to be included in the checklist. Some items may be modified to fit the needs of a microcosting evaluation. We will also note whether the checklists lacked items to assess specific criteria that are relevant, and should be included, for microcosting studies. Criteria that are inadequately covered will be identified and new items will be formulated for these criteria in the new checklist.

The research team conducting the systematic review will extract data from the microcosting studies using a standardized data collection form based on the CHEERS guidelines [14,15], guidance from the Campbell and Cochrane Economics Methods Group [30], and the research team's previous experience with systematic reviews of health economic studies and microcosting studies [33-38]. Data will be extracted from microcosting studies in a number of areas including (1) cost components included (eg, personnel costs, consumables/materials/supplies cost, medication costs, facility costs, transportation costs, productivity loss); (2) whether the study reports input utilization quantity and unit cost data separately; (3) method of quantity data collection used (eg, time-motion study, patient self-report, cost-accounting database, provider/staff interview); and (4) method of unit cost data collection (eg, invoice amount, hospital/clinic/provider price catalogue, standard fee schedule) [19]. New items will be formulated by the research team conducting the systematic review based on the data extracted from and critical review of the published microcosting studies.

Method 2

A comprehensive search for published checklists and guidelines used to evaluate the quality, conduct, and reporting of costing in economic analyses of health interventions and programs will be performed. References to published articles describing reporting guidelines or checklists to evaluate the quality of economic evaluations of health interventions will be identified. The references in the selected articles will also be manually reviewed in an iterative process to identify all relevant checklists and guidelines. We will also manually review the references from systematic reviews conducted to evaluate the conduct and reporting of economic evaluations of health interventions including the CHEERS statement [14,15], and the “Best practices for conducting economic evaluations in health care: a systematic review of quality assessment tools” AHRQ report [13].

The search will include the articles in the systematic review (Method 1). In addition, the terms used to index the relevant articles will be identified and used to perform a broad electronic literature search to identify additional checklists and guidelines. Searches based on terms identified to date include (1) (“microcost” OR “microcost”) AND (“questionnaire” OR “checklist” OR “guideline”); and (2) (“cost” OR “cost analysis”) AND (“questionnaire” OR “checklist” OR “guideline”) AND “health care quality, access and evaluation”). We will search PubMed, EconLit, BIOSIS Previews, Embase, Scopus and the National Health Service Economic Evaluation database (NHS EED) to identify relevant English language articles.

From the checklists and guidelines identified in the selected publications, the items relevant for assessing or reporting costing of health interventions or programs will be extracted. These items will be compiled into a comprehensive list and categorized into domains. Within each domain, we will review and narrow down selection of items and will remove any duplicates. Only items considered relevant to microcosting will be retained based on consensus of the research team. We will provide the rationale for inclusion or exclusion of each item and domain.

Method 3

A theoretical analysis of economic concepts relevant for microcosting will be conducted. A search has been done for literature in welfare economics and microeconomics and for literature in costing that defines microcosting and differentiates microcosting from gross costing and other costing methods. The latter search included articles in the systematic review, references from these articles, and references from checklists for economic evaluations. The difference between the use of charges, prices, or payments to assess costs and estimates of the real costs of resources will be examined. An analytical framework for conducting microcosting studies will be developed and conceptual domains relevant for microcosting will be discussed. Any domains that are missing or not adequately represented in any current economic evaluation checklist will be identified. Newly formulated checklist items (not included in existing checklists) will be developed for each conceptual domain for inclusion in the new checklist.

Creation of an Initial Item Pool

Items derived using each method (ie, the systematic review, checklist review, and theoretical analysis) will be compiled into a comprehensive list. The overlap and variation in domains and items will be documented and any duplicate or redundant items will be removed. The pool of remaining items will be discussed and evaluated by research team members.

New items may be formulated based on the findings of criteria deemed relevant and necessary for microcosting studies but inadequately covered by existing checklists or guidelines. The addition of new items will also be guided by the analysis of the systematic review and by the theoretical analysis. Through deliberation and discussion, items will be refined and a consensus will be reached in the selection of items to be included in a preliminary list.

External Review by Expert Panel

The checklist will be developed using a modified Delphi method, designed for reaching consensus among an expert panel. Delphi is a “method for structuring a group communication process” [39]. It consists of an iterative multistage process with the goal of consensus among a group of experts [40,41]. In conventional Delphi exercises a list of issues to be considered is usually developed by open-ended questions in the first round, and the participants usually remain anonymous. The following two major modifications that have been reported in the literature will be employed: (1) the use of a literature review to determine in advance the list of issues to be considered and ranked by the panel [40-42], and (2) the possible addition of an online/electronic panel discussion or workshop following the survey rounds to resolve any lack of consensus [40,43,44]. Usually not all panel participants are able to attend the online/electronic panel or workshop, and the discussion will breach the anonymity of participants who attend. If a panel or workshop is held, permission will be obtained from participants to use their names in any acknowledgements in subsequent publications. These modifications to the traditional Delphi process can save time and financial resources, and are appropriate for our topic, which is limited in scope and requires a narrow range of expertise for which there are only a limited number of qualified panel members [40,43]. Our Delphi process will be similar to that described by Husereau et al in the development of the CHEERS report [15].

An international expert panel will be recruited and a modified Delphi exercise will be conducted to rank the items in the preliminary list. Panel participants will be selected based on their expertise in the conduct and reporting of economic evaluations, and specific expertise in microcosting studies and methodology. Potential panel participants will include (1) content specialists who have conducted and published full microcosting studies and who are identified in the course of the systematic literature review; (2) international researchers who have expertise in economic evaluations for health interventions with specific interest in costing studies; (3) journal editors interested in publishing microcosting studies; and (4) methodologists with expertise in checklist development. Invitation emails will be sent to potential members of the expert panel, including a description of the project and the expected timeline. We will invite participants to complete an initial survey to rank the items and provide comments, and if possible, to complete a follow-up survey and discussion and/or workshop electronically, with results reported according to the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) [45].

The preliminary list of items will be sent to panel participants in the form of a survey. The procedures used in the development of the CHEERS checklist will be followed to obtain feedback from the panel participants [15]. Each panel participant will be asked to rate the importance of each item by using a 10-point Likert scale from 1 (“not at all important”) to 10 (“extremely important”). In addition, they will rate their confidence in judging the importance of each item on the basis of their current knowledge from 1 (“not confident”) to 3 (“very confident”). The participants will also be asked to comment on the wording and options for scoring, and recommend deletion or addition of items. The survey will be accessible either online or in print depending on the preference of the respondent.

Survey responses from round 1 will be recorded in an electronic spreadsheet. The items will be ranked by importance scores weighted by confidence ratings. Categories of importance will be created based on previous published reports [15,22]. Items will be labeled according to their weighted average score. There are various approaches to items rank ordering. We will pilot test the method where items with a weighted average score of more than 8 will be labeled as “very important/included”, 7-8 as “high importance/likely included”, 5-6 as “moderate importance/possibly included”, and 0-4 as “low importance/not included”. Comments for each item will be collated and summarized. The research team will review and revise the item list through discussion based on survey responses.

A revised list of items will be compiled including the information about item ranks and averages, and sent to members of the expert panel who agreed to participate in a second round of review. Respondents will be informed that items with a score of 6 or less (labeled “possibly included”) will be included in the final checklist only if they receive a higher score. Item-specific comments from round 1 will be included below each ranked item. After round 2, items with a score of 6 or less will be labeled “rejected” and not considered for the final checklist [15]. Responses will be categorized as for round 1, and comments will be collated and reviewed by the research team. An online meeting may be convened for expert panel participants to discuss the remaining items. The research team will revise the checklist based on the comments and discussion in the meeting.

Pilot Test

A selection of the full and predominant microcosting articles from the systematic review data will be used to pilot test the new checklist. The articles will be rated by two independent reviewers. Reliability estimates will be calculated and discrepancies will be discussed with the research team. Items may be modified or further explained in order to improve clarity and comprehension. The checklist scores will also be compared with scores for the coded checklists used in the systematic review (eg, the Drummond, Evers, Philips, Fukuda, and CHEERS checklists) in order to assess external validity.


The project is currently in progress.


Principal Findings

Currently, health systems in the United States and internationally are faced with increasing costs and limited resources. Accurate cost assessments are essential in order to plan programs and enact health care policies in a cost-efficient manner. Microcosting involves the direct measurement of cost by observation and survey and is used to find the actual cost of new health interventions and programs, or when existing administrative data are not sufficiently sensitive [10,12]. The concepts and methodologies for microcosting studies have been evolving over the past three decades [2,6,10,11,17-19,34-36,46-57], and an increasing number of studies have utilized microcosting techniques in recent years. However, existing instruments and guidelines for economic evaluations lack the framework and specific components required to guide the conduct and reporting of microcosting studies. The aim of this project is to develop a checklist, based on theoretical and empirical research and expert review, to assist with the conduct and reporting of microcosting studies. Standardization of the methods will enhance the consistency, transparency, and comparability of future microcosting studies.

Our review of guidelines and checklists included those that were intended for the conduct, reporting, and appraisal of economic evaluations for health interventions. In some cases, the specific purpose was stated, for example the stated purpose of the CHEERS guidelines is for reporting (and not for conduct) of studies. Several checklists have been designed specifically for quality assessment (eg, Drummond, Evers, Philips). These checklists consist of items that may be relevant for a combination of conduct, reporting, and appraisal without indicating a specific intention. In some cases, the content of the item may be similar, but the wording may indicate the relevance for conduct or reporting (eg, “resource use included” vs “resource use stated”). Checklists may not separate the quality of reporting from the validity of the design and conduct of a study, and elements of checklists intended for evaluating the quality of reporting may therefore be used as guidance in designing studies [13]. Our checklist is intended to provide a framework to consider when conducting, reporting, and/or appraising a microcosting study. In designing our checklist, we will be attentive to the purpose of each domain and each item, and will indicate the relevance for conduct, reporting and appraisal.

This protocol draws on recommendations for developing reporting guidelines [20,21], and on methodologies for developing published checklists for economic evaluations of health interventions [13-16]. Initial steps will include identifying a need for new guidance through a systematic literature review of microcosting studies, and a literature search to identify items used for costing in checklists for health economic evaluations and theoretical analyses. These three activities will be used to generate a preliminary list of items to include when conducting and reporting microcosting studies. The preliminary list will be disseminated to members of an expert panel, identified by the research team as having particular expertise in microcosting analysis and economic evaluation. Panel members will be asked to rate the importance of each item on a Likert scale and the average scores, weighted by confidence level, will be used to rank items. Items will then be categorized and ranked by the research team, based on scores and comments of panel members. Panel members will participate in a second round of review and possibly an online meeting to discuss items to be included in the final checklist. The checklist will be evaluated through a pilot test conducted by the research team in order to assess reliability and validity. The pilot test will also identify any issues that require clarification and determine how useable the checklist is in the real world [20].

Conclusion

This will be the first checklist for the conduct and reporting of microcosting studies, and will be a timely and important contribution to the health economic and health policy literature. In addition to its usefulness to health economists and researchers, it will also benefit journal editors and decision-makers who require accurate cost estimates in order to meet the goals of the health system to efficiently deliver health care, including electronic health (eHealth) interventions [58]. This framework will help to standardize the methods of microcosting, thereby allowing for greater transparency and comparability of costs among different health care interventions and programs.

Conflicts of Interest

None declared.

  1. Goeree R, He J, O'Reilly D, Tarride J, Xie F, Lim M, et al. Transferability of health technology assessments and economic evaluations: a systematic review of approaches for assessment and application. Clinicoecon Outcomes Res 2011;3:89-104. [FREE Full text] [CrossRef] [Medline]
  2. Finkler SA. The distinction between cost and charges. Ann Intern Med 1982 Jan;96(1):102-109. [Medline]
  3. Brouwer W, Rutten F, Koopmanschap M. Costing in economic evaluations. In: Drummond M, McGuire A, editors. Economic Evaluation in Health Care: Merging Theory with Practice. Oxford, UK: Oxford University Press; 2001:68-93.
  4. Gold MR, Siegel JE, Russell LB, Weinstein MC, editors. Cost-Effectiveness in Health and Medicine. New York, NY: Oxford University Press; 1996.
  5. Weinstein MC, Siegel JE, Gold MR, Kamlet MS, Russell LB. Recommendations of the panel on cost-effectiveness in health and medicine. JAMA 1996 Oct 16;276(15):1253-1258. [Medline]
  6. Tan SS, Rutten FF, van Ineveld BM, Redekop WK, Hakkaart-van RL. Comparing methodologies for the cost estimation of hospital services. Eur J Health Econ 2009 Feb;10(1):39-45. [CrossRef] [Medline]
  7. Heerey A, McGowan B, Ryan M, Barry M. Microcosting versus DRGs in the provision of cost estimates for use in pharmacoeconomic evaluation. Expert Rev Pharmacoecon Outcomes Res 2002 Feb;2(1):29-33. [CrossRef] [Medline]
  8. Clement Nee Shrive FM, Ghali WA, Donaldson C, Manns BJ. The impact of using different costing methods on the results of an economic evaluation of cardiac care: microcosting vs gross-costing approaches. Health Econ 2009 Apr;18(4):377-388. [CrossRef] [Medline]
  9. Ruger JP, Emmons KM, Kearney MH, Weinstein MC. Measuring the costs of outreach motivational interviewing for smoking cessation and relapse prevention among low-income pregnant women. BMC Pregnancy Childbirth 2009;9:46. [FREE Full text] [CrossRef] [Medline]
  10. Barnett PG. An improved set of standards for finding cost for cost-effectiveness analysis. Med Care 2009 Jul;47(7 Suppl 1):S82-S88. [CrossRef] [Medline]
  11. Frick KD. Microcosting quantity data collection methods. Med Care 2009 Jul;47(7 Suppl 1):S76-81. [FREE Full text] [CrossRef] [Medline]
  12. Luce BR, Manning WG, Siegel JE, Lipscomb J. Estimating costs in cost-effectiveness analysis. In: Gold MR, Siegel JE, Russel LB, Weinstein MC, editors. Cost Effectiveness in Health and Medicine. New York, NY: Oxford University Press; 1996:176-212.
  13. Walker DG, Wilson RF, Sharma R, Bridges J, Niessen L, Bass EB, et al. Best Practices for Conducting Economic Evaluations in Health Care: A Systematic Review of Quality Assessment Tools. Rockville, MD: Agency for Healthcare Research and Quality (AHRQ); 2012.
  14. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Int J Technol Assess Health Care 2013 Apr;29(2):117-122. [CrossRef] [Medline]
  15. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, ISPOR Health Economic Evaluation Publication Guidelines-CHEERS Good Reporting Practices Task Force. Consolidated Health Economic Evaluation Reporting Standards (CHEERS)--explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value Health 2013;16(2):231-50. [FREE Full text] [CrossRef] [Medline]
  16. Fukuda H, Imanaka Y. Assessment of transparency of cost estimates in economic evaluations of patient safety programmes. J Eval Clin Pract 2009 Jun;15(3):451-459. [CrossRef] [Medline]
  17. Tan SS, Bakker J, Hoogendoorn ME, Kapila A, Martin J, Pezzi A, et al. Direct cost analysis of intensive care unit stay in four European countries: applying a standardized costing methodology. Value Health 2012 Jan;15(1):81-6. [FREE Full text] [CrossRef] [Medline]
  18. Charles JM, Edwards RT, Bywater T, Hutchings J. Micro-costing in public health economics: steps towards a standardized framework, using the incredible years toddler parenting program as a worked example. Prev Sci 2013 Aug;14(4):377-389. [CrossRef] [Medline]
  19. Xu X, Grossetta Nardini HK, Ruger JP. Micro-costing studies in the health and medical literature: protocol for a systematic review. Syst Rev 2014;3:47. [FREE Full text] [CrossRef] [Medline]
  20. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med 2010 Feb;7(2):e1000217. [FREE Full text] [CrossRef] [Medline]
  21. Moher D, Weeks L, Ocampo M, Seely D, Sampson M, Altman DG, et al. Describing reporting guidelines for health research: a systematic review. J Clin Epidemiol 2011 Jul;64(7):718-742. [CrossRef] [Medline]
  22. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, et al. CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLoS Med 2008 Jan 22;5(1):e20. [FREE Full text] [CrossRef] [Medline]
  23. Chiou C, Hay JW, Wallace JF, Bloom BS, Neumann PJ, Sullivan SD, et al. Development and validation of a grading system for the quality of cost-effectiveness studies. Med Care 2003 Jan;41(1):32-44. [CrossRef] [Medline]
  24. Ungar WJ, Santos MT. The Pediatric Quality Appraisal Questionnaire: an instrument for evaluation of the pediatric health economics literature. Value Health 2003;6(5):584-94. [FREE Full text] [CrossRef] [Medline]
  25. Nixon J, Rice S, Drummond M, Boulenger S, Ulmann P, de PG. Guidelines for completing the EURONHEED transferability information checklists. Eur J Health Econ 2009 May;10(2):157-165. [CrossRef] [Medline]
  26. Evers S, Goossens M, de VH, van TM, Ament A. Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria. Int J Technol Assess Health Care 2005;21(2):240-245. [Medline]
  27. Jacobs P, Ohinmaa A, Brady B. Providing systematic guidance in pharmacoeconomic guidelines for analysing costs. Pharmacoeconomics 2005;23(2):143-153. [Medline]
  28. Grutters JP, Seferina SC, Tjan-Heijnen VC, van Kampen RJ, Goettsch WG, Joore MA. Bridging trial and decision: a checklist to frame health technology assessments for resource allocation decisions. Value Health 2011;14(5):777-84. [FREE Full text] [CrossRef] [Medline]
  29. Phillips AC, Lewis LK, McEvoy MP, Galipeau J, Glasziou P, Hammick M, et al. Protocol for development of the guideline for reporting evidence based practice educational interventions and teaching (GREET) statement. BMC Med Educ 2013;13:9. [FREE Full text] [CrossRef] [Medline]
  30. Shemilt I, Mugford M, Byford S, Drummond M, Eisenstein E, Knapp M, et al. Incorporating economics evidence. In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. West Sussex, UK: Wiley; 2008:449-480.
  31. Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ 1996 Aug 3;313(7052):275-83. [FREE Full text] [Medline]
  32. Philips Z, Bojke L, Sculpher M, Claxton K, Golder S. Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment. Pharmacoeconomics 2006;24(4):355-371. [Medline]
  33. Ruger JP, Chawarski M, Mazlan M, Luekens C, Ng N, Schottenfeld R. Costs of addressing heroin addiction in Malaysia and 32 comparable countries worldwide. Health Serv Res 2012 Apr;47(2):865-87. [FREE Full text] [CrossRef] [Medline]
  34. Ruger JP, Lazar CM. Economic evaluation of drug abuse treatment and HIV prevention programs in pregnant women: a systematic review. Addict Behav 2012 Jan;37(1):1-10. [FREE Full text] [CrossRef] [Medline]
  35. Ruger JP, Lazar CM. Economic evaluation of pharmaco- and behavioral therapies for smoking cessation: a critical and systematic review of empirical research. Annu Rev Public Health 2012 Apr;33:279-305. [FREE Full text] [CrossRef] [Medline]
  36. Ruger JP, Ben AA, Cottler L. Costs of HIV prevention among out-of-treatment drug-using women: results of a randomized controlled trial. Public Health Rep 2010;125 Suppl 1:83-94. [FREE Full text] [Medline]
  37. Ruger JP, Emmons KM. Economic evaluations of smoking cessation and relapse prevention programs for pregnant women: a systematic review. Value Health 2008;11(2):180-90. [FREE Full text] [CrossRef] [Medline]
  38. Xu X, Yonkers KA, Ruger JP. Costs of a motivational enhancement therapy coupled with cognitive behavioral therapy versus brief advice for pregnant substance users. PLoS One 2014;9(4):e95264. [FREE Full text] [CrossRef] [Medline]
  39. Linstone HA, Turoff M. Introduction. In: Linstone HA, Turoff M, editors. The Delphi Method: Techniques and Applications. Boston, MA: Addison-Wesley Educational Publishers; 1975:3.
  40. de Meyrick J. The Delphi method and health research. Health Education 2003 Feb;103(1):7-16. [CrossRef]
  41. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000 Oct;32(4):1008-1015. [Medline]
  42. Leisner BA. Learning needs of hospitalized veteran patients: developing a tool for practice. Patient Educ Couns 1995 May;25(2):151-162. [Medline]
  43. de Loe RC. Exploring complex policy questions using the policy Delphi. Applied Geography 1995 Jan;15(1):53-68. [CrossRef]
  44. Paterson FM, Paterson RC, Watts A, Blinkhorn AS. Initial stages in the development of valid criteria for the replacement of amalgam restorations. J Dent 1995 Jun;23(3):137-143. [Medline]
  45. Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res 2004 Sep 29;6(3):e34. [FREE Full text] [CrossRef] [Medline]
  46. Finkler SA, Ward DM. Cost Accounting for Health Care Organizations: Concepts and Applications. Burlington, MA: Jones & Bartlett Learning; 1999.
  47. Tan SS, Hakkaart-van RL, Al MJ, Bouwmans CA, Hoogendoorn ME, Spronk PE, et al. A microcosting study of intensive care unit stay in the Netherlands. J Intensive Care Med 2008;23(4):250-257. [CrossRef] [Medline]
  48. Hendriks ME, Kundu P, Boers AC, Bolarinwa OA, Te Pas MJ, Akande TM, et al. Step-by-step guideline for disease-specific costing studies in low- and middle-income countries: a mixed methodology. Glob Health Action 2014;7:23573. [FREE Full text] [Medline]
  49. Smith MW, Barnett PG. Direct measurement of health care costs. Med Care Res Rev 2003 Sep;60(3 Suppl):74S-91S. [Medline]
  50. Swindle R, Lukas CV, Meyer DA, Barnett PG, Hendricks AM. Cost analysis in the Department of Veterans Affairs: consensus and future directions. Med Care 1999 Apr;37(4 Suppl Va):AS3-AS8. [Medline]
  51. Fishman PA, Hornbrook MC. Assigning resources to health care use for health services research: options and consequences. Med Care 2009 Jul;47(7 Suppl 1):S70-S5. [FREE Full text] [CrossRef] [Medline]
  52. Ritzwoller DP, Goodman MJ, Maciosek MV, Elston LJ, Meenan R, Hornbrook MC, et al. Creating standard cost measures across integrated health care delivery systems. J Natl Cancer Inst Monogr 2005(35):80-87. [CrossRef] [Medline]
  53. Mogyorosy Z, Smith P. The Main Methodological Issues in Costing Health Care Services: A Literature Review. York, UK: Centre for Health Economics; 2005.
  54. Wordsworth S, Ludbrook A, Caskey F, Macleod A. Collecting unit cost data in multicentre studies. Creating comparable methods. Eur J Health Econ 2005 Mar;6(1):38-44. [CrossRef] [Medline]
  55. Graves N, Walker D, Raine R, Hutchings A, Roberts JA. Cost data for individual patients included in clinical studies: no amount of statistical analysis can compensate for inadequate costing methods. Health Econ 2002 Dec;11(8):735-739. [CrossRef] [Medline]
  56. Raftery J. Costing in economic evaluation. BMJ 2000 Jun 10;320(7249):1597. [FREE Full text] [Medline]
  57. Sacristán JA, Soto J, Galende I. Evaluation of pharmacoeconomic studies: utilization of a checklist. Ann Pharmacother 1993 Sep;27(9):1126-1133. [Medline]
  58. Bergmo TS. How to measure costs and benefits of eHealth interventions: an overview of methods and frameworks. J Med Internet Res 2015;17(11):e254. [FREE Full text] [CrossRef] [Medline]


CHEERS: Consolidated Health Economic Evaluating Reporting Standards
CHERRIES: Checklist for Reporting Results of Internet E-Surveys


Edited by G Eysenbach; submitted 24.06.16; peer-reviewed by A Abdallah; comments to author 25.07.16; revised version received 08.08.16; accepted 08.08.16; published 05.10.16

Copyright

©Jennifer Prah Ruger, Marian Reiff. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 05.10.2016.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.