Background: The quality of care for tuberculosis (TB) is deficient in high-burden countries and urgently needs improvement. However, comprehensively identifying the required improvements is challenging. Providing high-quality TB care is an important step toward improving patients’ quality of life and decreasing TB morbidity and mortality. Effective tools for assessing the quality of TB services using international standards and guidelines can identify existing gaps in services and inform improvements to ensure high-quality inpatient TB services.
Objective: This study aimed to develop evaluation instruments for defining the quality of provision of TB services.
Methods: To assess quality of services in the largest TB hospital in Armenia, we developed instruments based on the Joint Commission International Accreditation Standards for Hospitals, International Standards for TB Care, TB Laboratories Bio-Safety Standards, and the World Health Organization framework for conducting TB program reviews. A mixed methods approach was utilized, triangulating quantitative (checklists) and qualitative (in-depth interviews) results. A scoring system and strengths, weaknesses, opportunities, and treats analysis was applied to detail results for each of the 122 standards assessed. A scaling approach was used to present overall performances of inpatient services for eight patient-centered functions and five organization management functions.
Results: Overall, 40 in-depth interviews and 91 checklists (21 observations, 16 policy papers, 20 staff qualification documents, and 34 medical records) were developed, utilized, and analyzed to explore practices of health care professionals, assess inpatient treatment experience of patients and their family members, evaluate facility environmental conditions, and define the degree of compliance to standards.
Conclusions: The effective comprehensive evaluation instruments and methods developed in this study for quality of inpatient TB services support the implementation of similar effective assessments in other countries. It may also become a platform to develop similar approaches for assessing ambulatory TB services in resource-limited countries.
International Registered Report Identifier (IRRID): DERR1-10.2196/13903
In 2014, the 67th World Health Assembly set a goal for the year 2035: The number of tuberculosis (TB) deaths will reduce by 95%, the TB incidence rates will reduce by 90% as compared with those in the year 2015, and no family will “face catastrophic costs due to tuberculosis” . The same report identified a stubborn persistence of the burden of TB disease in resource-limited countries, exasperated by multidrug resistant (MDR)-TB. Multidrug resistance threatens effective TB control and is a major threat to global health security. However, the effective management of MDR-TB is limited by health service barriers [ ]. Approximately 39% of the estimated TB cases and 75% of the estimated MDR-TB cases globally were either undiagnosed or underreported in 2017, indicating deficiencies in the quality of TB diagnostic and treatment services [ - ].
To achieve the goal of reducing the burden of TB worldwide, the special challenges of MDR-TB in high-burden resource-limited countries such as Armenia have provided impetus for new recommendations, moving from a focus on expanding coverage of free TB diagnostic and treatment services to a focus on quality of services . Provisions to achieve high-quality TB health care are essential steps toward improving TB medical practice and patient outcomes, leading to decreases in TB disease incidence [ , ]. The World Health Organization (WHO) defines high quality of health care as a patient-centered conjunction of six dimensions: effectiveness, efficiency, accessibility, acceptability, equitability, and safety. Best practices address all these dimensions [ ].
Standards Measuring Quality of Inpatient Services
The internationally adopted Joint Commission International (JCI) Accreditation Standards for Hospitals are evidence-based standards to measure the quality of services provided in hospitals in order to improve performance and outcomes of hospitals . The assessment of service quality provided in hospitals is conducted through utilization of functions, standards, and measurable elements [ ]. Functions consist of various standards, and the standards consist of measurable elements. Measurable elements measure the degree of compliance of hospital performance to their respective standards. Compliance to standards are, in turn, used to evaluate overall hospital performance for the respective function [ ].
The Quality Improvement Handbook for TB and MDR-TB Programs identifies three perspectives on the quality of TB care: the perspectives of patients, service providers, and health facility managers . Service providers’ perspective of quality [ , ] includes clinical competence, confidence, being respectful, educating patients, application of TB management core principles, team working ability, motivation, and proper documentation of treatment outcomes [ , ]. Facility managers’ perspective includes offering services that satisfy patients and community, succeeding on performing TB monitoring indicators, and assuring recognition of their health facility by other stakeholders [ ]. The International Standards for TB Care instrument addresses these perspectives, which are utilized to measure the quality of care for TB patients [ ].
Assessments of quality of TB health care have typically relied heavily on quantitative assessment methods, including survey instruments, checklists using simulations, checklists for direct observations in the health care facility, and chart reviews or audits . However, quantitative assessments alone provide an incomplete profile of the quality of health care facility services and do not fully address underlying factors influencing quality [ , ]. Qualitative assessments such as semistructured in-depth interviews and focus groups have long been touted for answering quality of health care questions that quantitative methods are ill suited to answer [ ], including questions on how health care services are actually operating [ , ]. Qualitative assessments are better at extricating reasons for questionable clinical practices and providing further clarification on how patients and caregivers experience and perceive their health care [ ]. However, these qualitative assessment methods are infrequently used to assess the quality of health care [ ]. They are especially absent in resource-limited high-burden countries for assessing the quality of TB health services, where there is a heavy reliance on quantitative assessment methods alone [ - ]. In addition, a large majority of the quality assurance tools for resource-limited settings are designed for national TB programs or for national- or district-level services and are not applicable to individual TB hospitals [ ].
The most comprehensive understanding of the quality of health care and underlying factors influencing quality requires a combination of both quantitative and qualitative assessment methods, a mixed methods approach [, ]. Mixed methods better inform the design and development of more successful interventions to improve the quality of health care [ , ]. Yet, no previous published literature integrated both qualitative and quantitative assessments to evaluate the quality of inpatient TB health care in resource-limited high-burden settings. We designed and applied a mixed methods assessment based on the WHO best practices for a comprehensive evaluation of TB inpatient health care services. This assessment was designed to inform systematic feasible improvements in quality and address the two pillars of integrated patient-centered care, and it intensified research and innovation of the Global End TB strategy [ ]. This study aimed to develop evaluation instruments using international experience and different assessment tools for defining the quality of care in the largest TB inpatient facility in Armenia.
Study Design and Instruments
For a larger, more comprehensive, valid evaluation of the quality of diagnostic and treatment services of the largest TB hospital of the National Tuberculosis Control Center (NTCC) in Armenia, we used a mixed methods study design. We integrated qualitative and quantitative methods by triangulating results from all data collection instruments and methods .
We developed the study instruments based on suitable JCI Standards for Hospital Accreditation, International Standards for TB Care, TB Laboratories Bio-Safety Standards, and WHO Framework for Conducting TB Programs Review [, ]. JCI standards and measurable elements were incorporated into the study instruments when appropriate for the assessment ( ). These standards were divided into two main sections consisting of eight patient-centered functions, including the TB-tobacco control function [ ] (using the WHO recommendations for integration of TB and tobacco control measures [ ]), and five health care organization management functions. All these functions have their specific standards, and each of these standards consist of several measurable elements ( ).
The study instruments include document review checklists for policy review, staff qualification review, and medical records; observation checklists for TB patients’ admission, access and continuity of care, laboratory services, medication storage management and use, infection prevention and control, and kitchen and food storage; and in-depth interview guides for 11 groups of key informants (). The JCI and WHO standards or measurable elements were used to develop open-ended questions for the in-depth interview guides. All the study instruments were developed in English and translated into Armenian. The quantitative checklists were pretested before data collection; the qualitative guides were continuously improved, as needed, during the process of data collection.
The Institutional Review Board of the American University of Armenia approved the study for compliance with locally and internationally accepted ethical standards (protocol number: AUA-2016-002).
All participants were informed of their rights; all those who chose to participate provided verbal informed consent. Audio recording and observations were possible only with permission of participants; if a participant did not want to be audio recorded, only written notes were taken.
Policy documents were reviewed to understand the formal documents that regulate the hospital’s daily practices in relation to TB treatment and diagnosis. Reviewed documents included organizational charter of the NTCC, internal disciplinary rules of the organization, and internal regulations of different structural units. We also reviewed several guidelines and national regulations, such as hand-hygiene guidelines, waste and expired materials’ disposal guidelines, methodological guides for TB infection control, and the national standards for TB treatment and diagnosis.
The staff’s qualification document review was designed to evaluate compliance of relevant professional experience, qualifications, and credentials required for job duties and responsibilities of the staff.
Medical records review included reviews of medical records and TB treatment cards of patients with TB admitted for TB inpatient treatment at the NTCC 2 months before the assessment. We selected 2 months before the assessment to have adequate time to review the full range of medical records per patient (medical history, TB treatment card, and MDR-TB treatment files). After discharge, patients’ TB treatment cards are transferred to TB outpatient centers. All clinical departments that did not admit new TB patients during the data collection period were asked to provide all records they had from 2016.
Observations were designed to explore several environmental conditions and daily practices of health care providers in TB treatment and diagnosis using standardized checklists. To evaluate the quality of diagnostic laboratory services, we observed daily practices in both bacteriological or microbiological and clinical laboratories of the hospital, considering the degree of compliance with the WHO biosafety standards of TB laboratories  and the radiology department. We used a standardized checklist to assess smoking practices and strategies in order to eliminate indoor smoking based on the observation of behaviors of health care providers and patients or family members.
The study team identified key informants from clinical departments and administrative units (based on experience and expertise on inpatient care) by using purposive sampling to optimize information acquisition and convenience sampling for those willing to participate, given the optimal utilization of available resources. All stakeholders of inpatient TB services were included to ensure validity. In-depth interviews included 11 key informant groups or sampling units to protect their confidentiality and provide data triangulation : (1) NTCC managers, (2) heads of departments, (3) TB physicians and nurses, (4) patients with TB and their family members, (5) laboratory staff, (6) pharmacists, (7) infection prevention and control staff, (8) radiologists, (9) monitoring and evaluation staff, (10) continuing education staff, and (11) human resource management staff. They were further categorized as (1) administration, (2) health care providers, and (3) patients with TB and their family members. TB health care providers were physicians and nurses with professional experience of working in the inpatient unit of the NTCC. To ensure full coverage, at least one representative from each inpatient department participated in the study. Participating TB patients (their family members) had completed their intensive phase of TB treatment in the NTCC hospital and were in the continuation phase of treatment in outpatient TB centers. For patients from the children’s TB department, only adult caregivers were contacted to participate. To recruit patients with TB, we collaborated with physicians from the TB outpatient center who made the initial contact with patients to share their contacts with the research team, and after they agreed, they passed the patients’ contact information to the research team.
The quantitative assessment checklists (legal and staff qualification documents review, medical records, and observations) data were entered in a Microsoft Excel 2013 worksheet for further analysis.
The qualitative in-depth interview data were analyzed utilizing deductive content analysis with a structured matrix [- ]. The research team used a predefined structure of initial coding, which comprised the measurable elements of the selected standards ( ). After verbatim transcription of the data, two researchers reviewed all transcripts and started the analysis using investigator triangulation. Data triangulation was applied across the different data sources [ ].
To integrate qualitative and quantitative results to measure compliance to the standards, we developed a single checklist comprising all assessed standards with their measurable elements. Using this checklist, we applied integrated methodological triangulation across both quantitative and qualitative results .
Next, we developed a scoring system, setting the maximum score for each standard to 10. Applying weighted scores to measurable elements within each standard, we calculated the score of the assessed standards by summing the scores of their measurable elements. The number of measurable elements for each standard ranged from 2 to 10 (average=4), depending on the standards’ complexity. We based our evaluation of compliance to the NTCC’s daily practices on the obtained scores (scored from 0 to 10) for each assessed standard.
After developing the coding scheme and the scoring system, we conducted a strengths, weaknesses, opportunities, and threats (SWOT) analysis, grouping all the findings into SWOT for each of the 122 standards . The findings include both a scoring table and a SWOT analysis for each of the standards. The scoring table and SWOT analysis were supported by direct quotes from respondents, which reduced the influence of biases of the study team and enhanced the findings for improved communication to a wider audience ( ).
Furthermore, to measure the overall performance of inpatient services of the NTCC in meeting the 13 assessed functions, we calculated the function mean score. We calculated this score based on a scoring system we developed that identifies the level to which standards of each function were met. The score ranges were defined using the SD calculated from the mean , with the minimum score equal to 0 and the maximum score equal to 10. These numeric values were further defined as a function scale, with categories of not met (0), minimally met (0.1-3.3), partially met (3.4-6.6), satisfactory met (6.7-9.9), and fully met (10). Regarding data on meeting the standards, we have converted the scores to corresponding percentage ( ).
Overall, 16 different internal policy papers and national regulations were reviewed to complete the policy review checklist. We used the policy review checklist to assess the presence or absence of certain policies and procedures that are recommended internationally. The staff’s qualification documents review utilized 20 personnel files, including descriptions of positions of all employees from all departments and the staffing plan of the organization for which standardized checklists were utilized. The medical records review included 34 medical records and TB treatment cards of patients with TB and utilized standardized checklists ().
Applying the standardized checklists, we conducted 21 observations in the hospital and in its vicinity ().
Overall, 40 in-depth interviews of key informants were conducted: NTCC managers (n=2), heads of departments (n=5), TB physicians and nurses (n=21), TB patients and their family members (n=4), laboratory staff (n=2), pharmacist (n=1), infection prevention and control staff (n=1), radiologist (n=1), monitoring and evaluation staff (n=1), continuing education staff (n=1), and human resource management staff (n=1;).
The calculated scores of the 122 assessed standards and their SWOT analysis contributed to understanding the details and the extent to which inpatient TB services in Armenia comply with local and international standards. The scaling approach, which was applied to evaluate the overall performance of inpatient services for its 13 functions, helped visualize existing gaps in patient-centered and organization management levels of the system and share findings with a wider audience.
Improving the quality of TB health services is possible through adherence to international standards adapted for the local resource-limited context. Modern mechanisms of patient safety and quality assurance in inpatient and diagnostic facilities will result in sustained improvements in operations and improved quality of care provision and will create a safer environment for patients with TB including those with drug-resistant TB. The suggested protocol for quality assessment could help identify gaps in quality of care and patient safety; addressing those gaps could strengthen the response of health systems in providing accessible, affordable, and acceptable services with patient-centered approaches in line with the WHO Roadmap to prevent and combat drug-resistant TB in the European region .
National TB programs in other countries (beside Armenia) can use similar innovative mixed methods and instruments to determine compliance of their TB care systems with the internal policies and procedures and national and international guidelines to improve TB care.
Moreover, this approach of inpatient assessment of TB services can be applied for developing and adopting mechanisms for ambulatory assessment of TB services, providing resource-limited national TB programs with a tool to comprehensively measure compliance of TB services with the international standards.
We are grateful to the Armenian Medical Fund for the financial support. We are thankful to the administration of the National TB Control Center of the Ministry of Health of the Republic of Armenia and the inpatient center clinical staff, who provided their continuous support throughout the study. This work was financially supported by the Armenian Medical Fund, United States.
NT, ZG, LM, and VP designed the study. NT, ZG, and LM implemented the study activities. NT and BC drafted the manuscript. NT, ZG, LM, BC, and VP reviewed, commented on, and approved the manuscript for publication.
Conflicts of Interest
Standards of patient-centered and health care organization management functions to evaluate the quality of inpatient diagnostic and treatment services.DOCX File , 155 KB
Examples of strengths, weaknesses, opportunities, and threats and overall performance analyses.DOCX File , 34 KB
- Raviglione M. World Health Organization. Global strategy and targets for tuberculosis prevention, care and control after 2015 URL: https://www.who.int/tb/post_2015_tb_presentation.pdf [accessed 2019-01-27]
- World Health Organization. Global Tuberculosis Report 2018. Geneva: World Health Organization; 2018.
- Lawn SD, Zumla AI. Tuberculosis. Lancet 2011 Jul 2;378(9785):57-72. [CrossRef] [Medline]
- Faustini A, Hall AJ, Perucci CA. Risk factors for multidrug resistant tuberculosis in Europe: a systematic review. Thorax 2006 Feb;61(2):158-163 [FREE Full text] [CrossRef] [Medline]
- Cazabon D, Alsdurf H, Satyanarayana S, Nathavitharana R, Subbaraman R, Daftary A, et al. Quality of tuberculosis care in high burden countries: the urgent need to address gaps in the care cascade. Int J Infect Dis 2017 Mar;56:111-116 [FREE Full text] [CrossRef] [Medline]
- Gebrekidan G, Tesfaye G, Hambisa MT, Deyessa N. Quality of tuberculosis care in private health facilities of addis ababa, ethiopia. Tuberc Res Treat 2014;2014:720432 [FREE Full text] [CrossRef] [Medline]
- Hopewell PC, Pai M, Maher D, Uplekar M, Raviglione MC. International standards for tuberculosis care. Lancet Infect Dis 2006 Nov;6(11):710-725. [CrossRef] [Medline]
- World Health Organization. Quality of Care: A Process for Making Strategic Choices in Health Systems. Geneva: World Health Organization; 2006.
- Joint Commission. Joint Commission International Accreditation Standards for Hospitals. Fourth Edition. Oakbrook Terrace: Joint Commission International; 2011.
- Sadaphal S, Kak N, Holschneider S, Smith-Arthur A, Matji R. Southern Africa HIV and AIDS Information Dissemination Service. 2013. TB CARE II. Quality Improvement Handbook for TB and MDR-TB Programs URL: http://catalogue.safaids.net/sites/default/files/publications/Quality%20Improvement%20Handbook.pdf [accessed 2019-01-21]
- Nikisi J, Kanjipite W, Sarkar S, Holschneider S, Matji R, Smith-Arthur A. ReproLinePlus. 2013. TB CARE II. Quality Improvement of TB Services: Assessment of Provider Adherence to TB Evidence-based Standards and Guidelines in Zambia URL: http://reprolineplus.org/system/files/resources/Adherence_Zambia.pdf [accessed 2019-02-03]
- Zannat F, Holschneider S, Matji R, Smith-Arthur A, Husain A. TB CARE II. 2013. TB CARE II. Quality improvement of TB services:Assessment of Provider Adherence to TB Evidence-based Standards and Guidelines in Bangladesh URL: https://tbcare2.org/wp-content/uploads/2018/04/TB-CARE-II-Adherence-report_Bangladesh.pdf [accessed 2019-02-12]
- Centers for Disease Control and Prevention. 2013. Core Curriculum on Tuberculosis: What the Clinician Should Know. Sixth Edition URL: https://www.cdc.gov/tb/education/corecurr/pdf/corecurr_all.pdf [accessed 2019-01-18]
- Gichangi P, Nyamu J, Kabue M, Gohole A, Malonza I, Sitienei J, et al. ReproLinePlus. 2013. TB CARE II. Quality Improvement of TB Services: Adherence with Evidence-Based TB Standards Guidelines in Selected Health Facilities in Kenya URL: http://reprolineplus.org/system/files/resources/Adherence_Kenya.pdf [accessed 2019-02-16]
- Fitzpatrick R, Boulton M. Qualitative methods for assessing health care. Qual Health Care 1994 Jun;3(2):107-113 [FREE Full text] [CrossRef] [Medline]
- McLaren ZM, Sharp AR, Zhou J, Wasserman S, Nanoo A. Assessing healthcare quality using routine data: evaluating the performance of the national tuberculosis programme in South Africa. Trop Med Int Health 2017 Feb;22(2):171-179 [FREE Full text] [CrossRef] [Medline]
- Silva-Sobrinho RA, Wysocki AD, Scatena LM, Pinto ES, Beraldo AA, Andrade RL, et al. Assessment of primary health care in the treatment of tuberculosis in a Brazilian locality of the international triple frontier. Open Nurs J 2017;11:124-134 [FREE Full text] [CrossRef] [Medline]
- Wysocki AD, Ponce MA, Brunello ME, Beraldo AA, Vendramini SH, Scatena LM, et al. Primary Health Care and tuberculosis: services evaluation. Rev Bras Epidemiol 2017;20(1):161-175 [FREE Full text] [CrossRef] [Medline]
- McQuade Billingsley K, Smith N, Shirley R, Achieng L, Keiser P. A quality assessment tool for tuberculosis control activities in resource limited settings. Tuberculosis (Edinb) 2011 Dec;91(Suppl 1):S49-S53 [FREE Full text] [CrossRef] [Medline]
- Das J, Hammer J, Leonard K. The quality of medical advice in low-income countries. J Econ Perspect 2008;22(2):93-114. [CrossRef] [Medline]
- Wisdom J, Creswell J. PCMH Resource Center - AHRQ. 2013 Mar. Mixed Methods: Integrating Quantitative and Qualitative Data Collection and Analysis While Studying Patient-Centered Medical Home Models URL: https://pcmh.ahrq.gov/page/mixed-methods-integrating-quantitative-and-qualitative-data-collection-and-analysis-while [accessed 2019-10-18]
- Creswell JW, Fetters MD, Ivankova NV. Designing a mixed methods study in primary care. Ann Fam Med 2004;2(1):7-12 [FREE Full text] [CrossRef] [Medline]
- Creswell JW, Clark VP. Designing And Conducting Mixed Methods Research. Second Edition. Thousand Oaks: Sage Publications Inc; 2011.
- Wisdom JP, Cavaleri MA, Onwuegbuzie AJ, Green CA. Methodological reporting in qualitative, quantitative, and mixed methods health services research articles. Health Serv Res 2012 Apr;47(2):721-745 [FREE Full text] [CrossRef] [Medline]
- World Health Organization. 2018 Sep 26. UN General Assembly High-Level Meeting on the fight against tuberculosis URL: https://www.who.int/news-room/events/un-general-assembly-high-level-meeting-on-ending-tb [accessed 2018-12-15]
- Patton MQ. Qualitative Evaluation Research Methods. Second Edition. Thousand Oaks, CA: Sage Publications Inc; 1990.
- World Health Organization. Framework for Conducting Reviews of Tuberculosis Programmes. Geneva: World health Organization; 2014.
- Alcaide J, Altet MN, Plans P, Parrón I, Folguera L, Saltó E, et al. Cigarette smoking as a risk factor for tuberculosis in young adults: a case-control study. Tuber Lung Dis 1996 Apr;77(2):112-116. [Medline]
- WHO, IUTLD. A WHO / the Union Monograph on TB and Tobacco Control: Joining Efforts to Control Two Related Global Epidemics. Geneva: World Health Organization; 2007.
- WHO. Tuberculosis Laboratory Biosafety Manual. Geneva: World Health Organization; 2012.
- Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res 2007 Aug;42(4):1758-1772 [FREE Full text] [CrossRef] [Medline]
- Meyer J. Qualitative research in health care. Using qualitative methods in health related action research. Br Med J 2000 Jan 15;320(7228):178-181 [FREE Full text] [CrossRef] [Medline]
- Curry LA, Nembhard IM, Bradley EH. Qualitative and mixed methods provide unique contributions to outcomes research. Circulation 2009 Mar 17;119(10):1442-1452. [CrossRef] [Medline]
- van Wijngaarden JD, Scholten GR, van Wijk KP. Strategic analysis for health care organizations: the suitability of the SWOT-analysis. Int J Health Plann Manage 2012;27(1):34-49. [CrossRef] [Medline]
- WHO. Roadmap to Prevent and Combat Drug‑resistant Tuberculosis. Geneva: World Health Organization; 2011.
|JCI: Joint Commission International|
|MDR: multidrug resistant|
|NTCC: National Tuberculosis Control Center|
|SWOT: strengths, weaknesses, opportunities, and threats|
|WHO: World Health Organization|
Edited by G Eysenbach; submitted 05.03.19; peer-reviewed by W LaMendola, M Schreier; comments to author 19.05.19; revised version received 01.08.19; accepted 13.08.19; published 07.01.20Copyright
©Nune Truzyan, Zaruhi Grigoryan, Lusine Musheghyan, Byron Crape, Varduhi Petrosyan. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 07.01.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.