This paper is in the following e-collection/theme issue:

Proposals (eHealth) (80) Clinical Information and Decision Making (1407) Methods and Feasibility Studies (379) Decision Support for Health Professionals (1185)

Published on in Vol 5, No 1 (2016): Jan-Mar

Predicting Appropriate Admission of Bronchiolitis Patients in the Emergency Department: Rationale and Methods

Predicting Appropriate Admission of Bronchiolitis Patients in the Emergency Department: Rationale and Methods

Predicting Appropriate Admission of Bronchiolitis Patients in the Emergency Department: Rationale and Methods

Authors of this article:

Gang Luo1 Author Orcid Image ;   Bryan L. Stone2 Author Orcid Image ;   Michael D. Johnson2 Author Orcid Image ;   Flory L. Nkoy2 Author Orcid Image
Article Authors Cited by (11) Tweetations (3) Metrics

    Protocol

    1School of Medicine, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, United States

    2School of Medicine, Department of Pediatrics, University of Utah, Salt Lake City, UT, United States

    Corresponding Author:

    Gang Luo, PhD

    School of Medicine

    Department of Biomedical Informatics

    University of Utah

    421 Wakara Way, Suite 140

    Salt Lake City, UT, 84108

    United States

    Phone: 1 801 213 3565

    Fax:1 801 581 4297

    Email: gangluo@cs.wisc.edu


    Background: In young children, bronchiolitis is the most common illness resulting in hospitalization. For children less than age 2, bronchiolitis incurs an annual total inpatient cost of $1.73 billion. Each year in the United States, 287,000 emergency department (ED) visits occur because of bronchiolitis, with a hospital admission rate of 32%-40%. Due to a lack of evidence and objective criteria for managing bronchiolitis, ED disposition decisions (hospital admission or discharge to home) are often made subjectively, resulting in significant practice variation. Studies reviewing admission need suggest that up to 29% of admissions from the ED are unnecessary. About 6% of ED discharges for bronchiolitis result in ED returns with admission. These inappropriate dispositions waste limited health care resources, increase patient and parental distress, expose patients to iatrogenic risks, and worsen outcomes. Existing clinical guidelines for bronchiolitis offer limited improvement in patient outcomes. Methodological shortcomings include that the guidelines provide no specific thresholds for ED decisions to admit or to discharge, have an insufficient level of detail, and do not account for differences in patient and illness characteristics including co-morbidities. Predictive models are frequently used to complement clinical guidelines, reduce practice variation, and improve clinicians’ decision making. Used in real time, predictive models can present objective criteria supported by historical data for an individualized disease management plan and guide admission decisions. However, existing predictive models for ED patients with bronchiolitis have limitations, including low accuracy and the assumption that the actual ED disposition decision was appropriate. To date, no operational definition of appropriate admission exists. No model has been built based on appropriate admissions, which include both actual admissions that were necessary and actual ED discharges that were unsafe.

    Objective: The goal of this study is to develop a predictive model to guide appropriate hospital admission for ED patients with bronchiolitis.

    Methods: This study will: (1) develop an operational definition of appropriate hospital admission for ED patients with bronchiolitis, (2) develop and test the accuracy of a new model to predict appropriate hospital admission for an ED patient with bronchiolitis, and (3) conduct simulations to estimate the impact of using the model on bronchiolitis outcomes.

    Results: We are currently extracting administrative and clinical data from the enterprise data warehouse of an integrated health care system. Our goal is to finish this study by the end of 2019.

    Conclusions: This study will produce a new predictive model that can be operationalized to guide and improve disposition decisions for ED patients with bronchiolitis. Broad use of the model would reduce iatrogenic risk, patient and parental distress, health care use, and costs and improve outcomes for bronchiolitis patients.

    JMIR Res Protoc 2016;5(1):e41

    doi:10.2196/resprot.5155

    Keywords

    Decision support techniques; forecasting; computer simulation; machine learning 


    Bronchiolitis is inflammation of the bronchioles, the smallest air passages in the lungs, primarily seen in children less than age 2. Within the first year of life, 10% of children are diagnosed with bronchiolitis [1]. By age 2, more than a third of children have had a bronchiolitis diagnosis [2]. Bronchiolitis causes about 71 hospitalizations and 77 emergency department (ED) visits per 1000 infant years [3]. In the United States, each year bronchiolitis incurs around 287,000 ED visits [4], 128,000 hospitalizations [5], and $1.73 billion in total inpatient costs (2009) [5]. For children under age 2, bronchiolitis is the most common cause of hospitalization and represents 16% of all hospitalizations [5-8].

    Despite the huge burden of bronchiolitis care, hospitalization decisions are made with insufficient evidence [7,9], resulting in variable admission rates [1,6,9-19]. About 32%-40% of ED patients with bronchiolitis are admitted to the hospital [20-22]. Studies suggest that 20%-29% of these admissions are unnecessary [23,24]. Unnecessary admissions waste health care resources, overwhelm hospital capacity, increase patient and parental distress, introduce iatrogenic risk such as exposure to other infectious diseases, and expose other hospitalized children to the respiratory pathogens of these patients [11,17,25]. As many as 10% of infants affected by bronchiolitis have adverse events while in the hospital [26]. Similarly, about 6% of ED discharges for bronchiolitis are unsafe, resulting in ED return with hospital admission [27] due to inadequate treatment [11]. New approaches are needed to improve ED disposition decision making and reduce unnecessary admissions and unsafe ED discharges.

    One method to reduce practice variation and improve clinicians’ decision making for bronchiolitis care is to use clinical guidelines [28-31]. However, existing clinical guidelines for bronchiolitis acknowledge that decisions to admit or to discharge are subjective and rely on variable clinical judgment due to a lack of specific objective thresholds with good evidence [30,31]. Clinical guidelines also do not account for differences in patient and illness characteristics including comorbidities [32] and offer limited improvement in determining ED disposition.

    Predictive models are an alternative method to reduce practice variation and improve decision making [20-22,33-35]. Predictive models can present objective criteria supported by historical data for an individualized disease management plan. Using data from previous patient encounters to identify patterns, predictive models [36] can overcome the limitations of clinical guidelines. Predictive models can be incorporated into electronic decision-support tools [37] to support the provisional judgment of clinicians or to trigger clinicians to reconsider their judgment in real time [20]. This is especially useful for physicians who see children infrequently or are junior. Typically when results of predictive models are provided, human experts can make better decisions [38].

    As reported in our previous paper [39], existing predictive models for ED patients with bronchiolitis are limited by low accuracy and the assumption that actual ED disposition decisions were appropriate. No operational definition of appropriate admission of ED patients with bronchiolitis exists and no model has been built on appropriate admissions (Figure 1). To fill the gap, we will meet the following 3 aims: (1) develop an operational definition of appropriate hospital admission for bronchiolitis, (2) develop a new model to accurately predict appropriate disposition for ED patients with bronchiolitis, and (3) conduct simulations to estimate the impact of using the model on outcomes.

    ‎
    Figure 1. The definition framework of appropriate admission versus appropriate ED discharge that was provided in our previous paper [39]. The details denoted by ""? will be determined by direct evidence in this current study.
    View this figure

    Innovations

    This study makes the following innovations within the context of bronchiolitis:

    1. We will develop a new approach to construct an operational definition of appropriate hospital admission in the ED based on objective data rather than clinical judgment. No such approach currently exists.
    2. We will figure out the most important attributes to put into the predictive model using a new simulation method. We will use various attribute combinations to ascertain the minimum requirement on performance and permit tradeoffs for the adaption of our model beyond our setting dependent upon available attributes. Current models are not generalizable beyond the study site because they rely on a certain set of attributes that may be nonexistent in different electronic medical records.
    3. We will build the first model to accurately predict appropriate admission for ED patients with bronchiolitis in real time. No such model currently exists. We will transform bronchiolitis care by developing a predictive model to guide appropriate admission for the first time.
    4. Our model will increase prediction accuracy by using a rich set of extracted attributes, including known predictors of hospital admission not used in existing models for ED patients with bronchiolitis.
    5. Our model will include environmental variables with a potential for a further increase in accuracy. Air quality environmental variables are associated with the daily number of hospitalizations for bronchiolitis [40] and a child’s risk of hospitalization for bronchiolitis within the first year of life [41]. The predictive power of air quality and respiratory virus environmental variables for appropriate admission has never been evaluated.
    6. Our study will evaluate the impact of using the model on outcomes. Previous predictive models focused only on accuracy. No impact estimate of using the model on bronchiolitis outcomes has ever been provided.
    7. We will use a large data set of 26,701 bronchiolitis patients with high potential to achieve high prediction accuracy. Previous studies are limited by small data sets with typically far fewer than 1000 patients. Many useful predictors of hospital admission cannot be identified in small data sets.

    In summary, this study is significant as it will fill gaps by developing a new model to guide and improve disposition decisions for ED patients with bronchiolitis. Broad use of the model will reduce iatrogenic risk, patient and parental distress, health care use and cost, and improve clinical outcomes for bronchiolitis patients. A future study will test the impact of using the model in a randomized controlled trial after implementing the model in an existing electronic medical record to facilitate real-time decision making.


    Machine learning is a field that studies the automatic improvement of computer algorithms with experience. Machine learning methods—such as support vector machine, neural network, and decision tree—are commonly used in predictive modeling [36] and will be adopted in our study. In comparison to statistical methods, machine learning can improve prediction accuracy, occasionally doubling it, with less stringent assumptions on data distribution [38,42,43].

    For all 3 aims, we will use the same patient population and data sets:

    1. Patient population: Our study cohort includes children under age 2 who had ED encounters at 22 Intermountain Healthcare facilities for bronchiolitis (ICD-9-CM discharge diagnosis code 466.1 [4]) in the past 10 years. Intermountain Healthcare is the biggest health care system in Utah, comprising 185 clinics and 22 hospitals.
    2. Data sets: A large administrative and clinical data set in the enterprise data warehouse (EDW) of Intermountain Healthcare will be used. The Intermountain Healthcare EDW contains a vast set of attributes [44]. Our Intermountain Healthcare data analyst will run SQL queries to obtain a data set that has been de-identified and encrypted and then securely transfer it to a computer that is encrypted and password-protected. Secondary analysis will be conducted on the computer. Intermountain Healthcare has dedicated tables to identify changes in procedure and diagnosis codes. The data set contains electronic documentation of about 85% of pediatric care delivered in Utah [45] and includes approximately 400 attributes. A partial list of categories of these attributes includes: admission date and time; age; orders (eg, medications, labs, exams, immunizations, imaging, counseling, etc), including order name, ordering provider, performing date, and result date; allergies; chief complaint; diagnoses; discharge date; exam result; facility seen for the patient visit; gender; health insurance; health care cost (billed charge, Intermountain Healthcare internal cost and reimbursed cost); height; home address; immunizations; lab test result; language(s) spoken; medication refills; primary care physician as listed in the electronic medical record; problem list; procedure date; procedures; provider involved in the visit; race/ethnicity; referrals; religion; visit type (inpatient, outpatient, urgent care, or emergency department); vital signs; and weight [46].

    For the last 5 years, data captured cover more than 2900 patients under age 2 and 3500 ED encounters for bronchiolitis per year. Due to its attribute richness and large size, the data set provides many advantages in the exploration of the proposed predictive models. Furthermore, we will use 21 environmental variables that regional monitoring stations recorded over the past decade within the Intermountain Healthcare region. These variables include carbon monoxide, nitrogen dioxide, particulate matter up to 2.5 μm in size and 10 μm in size, ozone, sulfur dioxide, relative humidity, temperature, precipitation, wind speed, dew point, and activities of each of the following viruses: enterovirus; adenovirus; parainfluenza virus types 1, 2, and 3; human metapneumovirus; influenza A and B viruses; rhinovirus; and respiratory syncytial virus. The data for all nonvirus environmental variables came from federal data sources [47,48], which provide such data throughout the United States. Observation unit admissions will be treated as hospital admissions since the only pediatric observation unit within Intermountain Healthcare has the same admission, coding, billing, and documentation requirements. Our analysis will consider various attribute combinations to ascertain the minimum requirement on performance and will permit tradeoffs for the adaption of our model beyond our setting dependent upon available attributes. Our analysis results will serve as the basis for future expansion of our models to other clinical data sets and diseases beyond bronchiolitis.

    Aim 1: Develop an Operational Definition of Appropriate Hospital Admission for ED Patients with Bronchiolitis

    In a recent paper [39], we provided a definition framework of appropriate hospital admissions. As shown in Figure 1, we equate appropriate admissions to necessary admissions and unsafe discharges. We equate appropriate ED discharges to safe discharges and unnecessary admissions. The definition uses several threshold values, such as the maximum number of hours for which major medical interventions are used. Using a data-driven approach, we will fill in these values and develop an operational definition to be used in Aims 2 and 3.

    For unsafe discharges, we will examine the distribution of the interval between discharge from the ED and a return visit resulting in admission for bronchiolitis within the period of 2 weeks [49,50]. The 95th percentile of the interval will cover most readmissions and define the return threshold for unsafe discharge. The distribution is highly skewed toward a short interval [27]. Thus, the return threshold will be insensitive to the length of the period chosen.

    For unnecessary admissions, we will examine the patients who stayed in the hospital for 12 hours or less and were discharged without readmission for bronchiolitis within 2 weeks. These patients are likely to have been admitted unnecessarily. Their median duration of using major medical interventions (Figure 1) will serve as a conservative threshold for use of major medical interventions in all admissions. Unnecessary admissions are those with major medical intervention exposures for no longer than the threshold. We will conduct sensitivity analysis to evaluate the impact of interactions between major medical interventions and other variables.

    If the operational definition for all bronchiolitis patients lacks face validity, we will examine data distributions for different age groups to obtain operational definitions by age group. Since the medical interventions for bronchiolitis have not changed over the last 10 years, we would expect the operational definition to remain the same during this period.

    Aim 2: Develop and Test the Accuracy of a New Model to Predict Appropriate Hospital Admission for an ED Patient with Bronchiolitis

    We will use clinical, administrative, and environmental variable attributes to build machine learning models to predict appropriate hospital admission for individual ED patients with bronchiolitis.

    Data Pre-Processing

    Traditional techniques like imputation will be used to handle missing values and identify and correct/remove invalid values [36,51]. In the case of environmental variables, classic methods [40,41] will be used to extract aggregate values (eg, daily average) from raw values. In the case of clinical and administrative attributes, grouper models like the diagnostic cost group system will be used to aggregate diseases, drugs, and procedures to reduce attributes [52].

    Input Variables

    For ED patients with bronchiolitis, predictors of hospital admission have not been exhaustively identified. We compiled in our recent paper [39] a comprehensive list of known predictors. Some of these known predictors (eg, atopic dermatitis [53], low dew point [54], duration of respiratory distress [7], absence of familial atopy [55], enterovirus infection [55], etc) have not been used in existing predictive models for ED patients with bronchiolitis. All known predictors stored in the Intermountain Healthcare EDW and environmental data sets will be used as input variables (ie, independent variables). In addition, our data sets contain attributes beyond known predictors. We will use classic feature selection techniques [56] like the information gain method to find attributes likely to be predictive of appropriate admission. Our team’s clinical experts will review attributes, select attributes with face validity, and add these as input variables. With more new predictors of appropriate hospital admission and larger sample size, we anticipate higher prediction accuracy.

    Predictive Models

    We will use Weka [56] to construct predictive models. Weka is a widely used open-source machine learning toolkit. It integrates a large set of standard machine learning algorithms and feature selection techniques. Both categorical and numerical variables exist in administrative, clinical, and environmental data. Supervised machine learning algorithms that can deal with both categorical and numerical variables, such as k-nearest neighbor and random forest, will be used. We will examine each applicable algorithm and tune hyper-parameters manually.

    The classic area under the receiver operating characteristic curve (AUROC) [56] performance metric will be used. Our target will be models achieving an AUROC larger than or equal to 0.9, which is considered outstanding discrimination [57]. Some machine learning models, such as decision tree and k-nearest neighbor (ie, similar patients), can be more easily interpreted [58,59]. Other machine learning models, such as random forest, are less straightforward to interpret. If accuracies of models are comparable (AUROC ≥0.9 and ≤0.02 worse for interpretable models compared to less interpretable models), we will favor those that clinicians can more easily interpret.

    Sample Size Justification and Performance Evaluation

    We have 10 years of data. We will train and test predictive models using a standard method. We will perform stratified 10-fold cross validation [56] on the initial 9 years of data to train predictive models and provide estimates of their accuracy. Data from the tenth year will be used to evaluate performance of the best-performing machine learning algorithm, reflecting use in practice. To figure out the environmental variable, administrative, and clinical attributes necessary for high accuracy, we will use backward elimination [36] to remove input variables so long as the AUROC does not decrease by more than 0.02 or go below 0.9.

    No AUROC achieved by current care has been reported before. By extrapolating from statistics reported in the literature (unnecessary admissions up to 29% and unsafe ED discharges of 6%), we anticipate the AUROC achieved by current care to be between 0.6 and 0.8+ [20-24,27]. We will test the hypothesis that the model’s prediction will be more accurate by a difference in AUROC of larger than or equal to 0.05. The dependent variable has 2 possible values: appropriate hospital admission and appropriate ED discharge. Assuming a correlation coefficient of 0.6 between the model’s prediction result and the actual disposition decision for both values and using a 1-sided Z-test at a 0.05 significance level, a sample size of 356 instances per possible value of the dependent variable will have 90% power to detect an AUROC increase of 0.05. Data from the tenth year include 3615 ED visits for bronchiolitis, which provides adequate power for testing our hypothesis.

    Based on 2 prior studies’ results, we anticipate that our model will achieve an AUROC larger than or equal to 0.9 and outperform current care in making disposition decisions. Neither prior study on predicting a bronchiolitis patient’s ED disposition is similar to our study, which uses appropriate admission as the gold standard. The first study [21] used actual admission as the gold standard and achieved an AUROC of 0.87. The second study [34] used judgment of an attending pediatrician as well as a length of stay longer than 1 day as the gold standard. The predictive model achieved 81% accuracy, better than an average admitting resident’s disposition decision.

    For ED patients with bronchiolitis, 17 known predictors of hospital admission (Table 1) are consistently recorded at Intermountain Healthcare facilities and available as structured attributes in our data sets, along with many other potential predictors. We will start building our model using structured attributes. If the model cannot achieve high prediction accuracy, we will extract additional de-identified input variables from ED clinical notes by conducting medical natural language processing on the HIPAA-compliant Homer computer cluster at the University of Utah [60]. For instance, additional input variables include the 8 known predictors of hospital admission (Table 1) that are inconsistently recorded in clinical notes at Intermountain Healthcare facilities.

    Table 1. The list of known predictors of hospital admission for ED patients with bronchiolitis recorded at Intermountain Healthcare facilities.
    CategoryPredictors
    The known predictors that are consistently recorded at Intermountain Healthcare facilities and available as structured attributes in our data setsSpO2, heart rate, respiratory rate, temperature, age, gender, prior hospitalization, prior intubation, abnormal chest x-ray, low dew point (from the environmental variable data set), rhinovirus infection, coinfection, dehydration, history of bronchopulmonary dysplasia, history of eczema, prematurity, maternal/passive smoking
    The known predictor that is rarely recorded as structured attributes at Intermountain Healthcare facilitiesenterovirus infection
    The known predictors that are inconsistently recorded in clinical notes at Intermountain Healthcare facilitiesincreased work of breathing, poor feedings, decreased feeding, breastfed, abnormalities on auscultation, retractions, family history of atopy, fewer albuterol in the first hour

    If our model still cannot reach high prediction accuracy on the entire group of ED patients with bronchiolitis, we will conduct subanalyses to identify subgroups of ED patients with bronchiolitis on which our model performs well. In this scenario, we will apply our final model only to the identified subgroups of patients. These subgroups are identified by certain characteristics, such as comorbidity, prematurity, age, or ED arrival time (eg, daytime vs night, weekday vs weekend) that are typically independent variables in the original model.

    We have large data sets. If scalability is a problem with Weka, a parallel machine learning toolkit like Spark’s MLlib [61] will be adopted to develop predictive models on the secure Homer computer cluster [60].

    Aim 3: Conduct Simulations to Estimate the Impact of Using the Model on Bronchiolitis Outcomes

    We will use a method similar to that in Luo et al [46] to establish the model’s utility for future use in clinical practice. More specifically, we will estimate the impact of using the model on bronchiolitis outcomes by applying the model to a retrospective cohort, and determine how the model can be generalized to different sites that collect differing sets of attributes. Our model will be developed using data from Intermountain Healthcare. Our simulations will help determine how to implement the model in other EDs. No prior study has either assessed the impact of using a predictive model on bronchiolitis outcomes or found the set of attributes most essential to generalize the model.

    Outcomes

    We will assess the outcomes of hospital admission, discharge to home, cost, and ED return. The primary outcome is cost. Other outcomes are indirectly reflected in cost and secondary. Each medical claim is companioned by a billed cost, a reimbursed cost, and an Intermountain Healthcare internal cost [52]. The Intermountain Healthcare internal cost [62] will be used because it is subject to less variation resulting from member cost-sharing [52] and more closely reflects actual cost. To deal with inflation, the medical consumer price index [63] will be used to standardize costs to 2014 US dollars. ED returns will be computed using the time interval defining unsafe discharge.

    Estimate a Model’s Impact

    Given a predictive model and a set of input variables, we will estimate the impact of using the model on each outcome. The same method in Aim 2 will be used to train the model on data from the first 9 years. Data from the tenth year have 4 groups: (1) necessary admissions, (2) unnecessary admissions, (3) unsafe discharges, and (4) safe discharges (Figure 1). For each group, we will obtain prediction results, then estimate the outcome if the model’s suggestions were followed. For example, consider necessary admissions. The model will erroneously predict that some of these patients should be discharged. We assume that in clinical application, every such patient will incur an unsafe discharge, an early return visit for bronchiolitis, and a cost equal to unsafe discharges’ average cost. The overall estimated outcome is the aggregate of outcome estimates in all 4 groups. Similarly, we can determine the minimum requirement of the model’s accuracy for the model to be valuable clinically.

    Sensitivity Analysis

    Intermountain Healthcare gathers a vast range of attributes. A different hospital may gather a portion of these attributes. To ensure that the model is generalizable, we will examine miscellaneous attribute combinations and estimate the outcomes of bronchiolitis when using the modified model. Our estimate will determine which attributes are important to include. In the case that an important attribute is nonexistent in a given ED, the estimate can advise substitute attributes that have a minor impact on bronchiolitis outcomes.

    Our complete model will include as many as 400 attributes. Conducting simulations for each possible combination of the attributes is not realistic because of the exponential growth of the number of combinations. As an alternative, an attribute grouping approach will be used. This approach associates attributes that commonly coexist based upon the judgment of our clinical experts. If an attribute in a group is not recorded by a hospital, related attributes in the group are also likely to be missing, such as attributes from the same lab test panel. Grouping will allow us to create and publish a table that lists the groups of possible attribute combinations, including bronchiolitis outcomes estimated via simulations and the trained parameters of the predictive model. If a hospital shows interest in implementing the model, the table can help assess expected outcomes in their environment, whether additional attributes need to be gathered, and if so, which ones. One row in the table will reflect the attributes in the PHIS+ [64] data model that standardizes administrative and clinical attributes from 6 major US children’s hospitals. The model of the row will apply directly to at least these 6 hospitals.

    Sample Size Justification and Performance Evaluation

    We will test 3 hypotheses: use of our predictive model will be linked to reduced (1) costs, (2) ED returns, and (3) hospital admissions. Due to their skewed distribution, cost data will be log-transformed [52]. The primary hypothesis will be accepted if the model lowers the log cost by at least 10% of its standard deviation. We will use 1-sided paired-sample t-test to assess the log cost difference between the model’s prediction result and the actual disposition decision. We will use McNemar’s test to assess the difference in ED returns and hospital admissions. A sample size of 857 data instances has 90% power to support the primary hypothesis at a 0.05 significance level. Data from the tenth year include 3615 ED visits for bronchiolitis, which offers sufficient power for testing the primary hypothesis.

    If performing simulations on 1 computer is too slow for the numerous combinations of attribute groups, we will conduct parallel simulations on the secure Homer computer cluster [60].


    We have secured institutional review board approvals from Intermountain Healthcare and the University of Utah for this study. At present, we are extracting administrative and clinical data from the Intermountain Healthcare EDW. Our goal is to finish this study by the end of 2019.


    The principle of our approach to developing an operational definition of appropriate hospital admission in the ED is general and can be used for other diseases beyond bronchiolitis. Our simulation method will ascertain how a predictive model can be generalized to different sites collecting various sets of attributes, as well as the group of attributes most essential for generalization. This study will use data from a big health care system with numerous heterogeneous facilities spread across a large area. These facilities include EDs at 22 hospitals, ranging from community metropolitan and rural hospitals attended by general practitioners and family doctors with constrained pediatric resources to tertiary care children’s and general hospitals in urban areas attended by subspecialists. Each of these facilities has a differing patient population, scope of services, geographic location, staff composition, and cultural background. This variation creates a realistic situation for identifying factors that are generalizable to other facilities across the United States. One of the models produced during simulation will directly apply to at least 6 large US children’s hospitals. Moreover, this study will produce a new modeling strategy that can be generalized to other clinical conditions where decision making is uncertain.

    In summary, our work will transform bronchiolitis care by developing a new predictive model to guide appropriate admission for ED patients with bronchiolitis. Broad use of the model will lower health care use and cost and improve clinical outcomes for bronchiolitis patients. We will have a new simulation method to estimate the impact of using a predictive model on outcomes in dissimilar data environments. The method can be useful for implementing other models.

    Acknowledgments

    We thank Maureen A. Murtaugh, Xiaoming Sheng, Michael Mundorff, Farrant Sakaguchi, Karen Valentine, Wendy W. Chapman, Adam B. Wilcox, Christopher G. Maloney, Chris Benitez, Bart Dodds, David E. Jones, Bret Heale, Katherine Sward, and John Holmen for helpful discussions.

    GL was mainly responsible for the paper. He performed literature review, formed the concept and developed the study, and wrote the paper. BS, MJ, and FN provided feedback on various medical issues, contributed to conceptualizing and designing the study, and revised the paper.

    Conflicts of Interest

    None declared.

    1. Ducharme FM. Management of acute bronchiolitis. BMJ 2011;342:d1658. [Medline]
    2. Zorc JJ, Hall CB. Bronchiolitis: recent evidence on diagnosis and management. Pediatrics 2010 Feb;125(2):342-349. [CrossRef] [Medline]
    3. Carroll KN, Gebretsadik T, Griffin MR, Wu P, Dupont WD, Mitchel EF, et al. Increasing burden and risk factors for bronchiolitis-related medical visits in infants enrolled in a state health care insurance plan. Pediatrics 2008 Jul;122(1):58-64 [FREE Full text] [CrossRef] [Medline]
    4. Hasegawa K, Tsugawa Y, Brown David F M, Mansbach JM, Camargo CA. Temporal trends in emergency department visits for bronchiolitis in the United States, 2006 to 2010. Pediatr Infect Dis J 2014 Jan;33(1):11-18 [FREE Full text] [CrossRef] [Medline]
    5. Hasegawa K, Tsugawa Y, Brown David F M, Mansbach JM, Camargo CA. Trends in bronchiolitis hospitalizations in the United States, 2000-2009. Pediatrics 2013 Jul;132(1):28-36 [FREE Full text] [CrossRef] [Medline]
    6. Mansbach JM, Emond JA, Camargo CA. Bronchiolitis in US emergency departments 1992 to 2000: epidemiology and practice variation. Pediatr Emerg Care 2005 Apr;21(4):242-247. [Medline]
    7. Parker MJ, Allen U, Stephens D, Lalani A, Schuh S. Predictors of major intervention in infants with bronchiolitis. Pediatr Pulmonol 2009 Apr;44(4):358-363. [CrossRef] [Medline]
    8. Shay DK, Holman RC, Newman RD, Liu LL, Stout JW, Anderson LJ. Bronchiolitis-associated hospitalizations among US children, 1980-1996. JAMA 1999 Oct 20;282(15):1440-1446. [Medline]
    9. Brand PL, Vaessen-Verberne AA. Differences in management of bronchiolitis between hospitals in The Netherlands. Dutch Paediatric Respiratory Society. Eur J Pediatr 2000 May;159(5):343-347. [Medline]
    10. Behrendt CE, Decker MD, Burch DJ, Watson PH. International variation in the management of infants hospitalized with respiratory syncytial virus. International RSV Study Group. Eur J Pediatr 1998 Mar;157(3):215-220. [Medline]
    11. Chamberlain JM, Patel KM, Pollack MM. Association of emergency department care factors with admission and discharge decisions for pediatric patients. J Pediatr 2006 Nov;149(5):644-649. [CrossRef] [Medline]
    12. Johnson DW, Adair C, Brant R, Holmwood J, Mitchell I. Differences in admission rates of children with bronchiolitis by pediatric and general emergency departments. Pediatrics 2002 Oct;110(4):e49. [Medline]
    13. Mallory MD, Shay DK, Garrett J, Bordley WC. Bronchiolitis management preferences and the influence of pulse oximetry and respiratory rate on the decision to admit. Pediatrics 2003 Jan;111(1):e45-e51. [Medline]
    14. Plint AC, Johnson DW, Wiebe N, Bulloch B, Pusic M, Joubert G, et al. Practice variation among pediatric emergency departments in the treatment of bronchiolitis. Acad Emerg Med 2004 Apr;11(4):353-360 [FREE Full text] [Medline]
    15. Vogel AM, Lennon DR, Harding JE, Pinnock RE, Graham DA, Grimwood K, et al. Variations in bronchiolitis management between five New Zealand hospitals: can we do better? J Paediatr Child Health 2003;39(1):40-45. [Medline]
    16. Willson DF, Horn SD, Hendley JO, Smout R, Gassaway J. Effect of practice variation on resource utilization in infants hospitalized for viral lower respiratory illness. Pediatrics 2001 Oct;108(4):851-855. [Medline]
    17. Willson DF, Jiao JH, Hendley JO, Donowitz L. Invasive monitoring in infants with respiratory syncytial virus infection. J Pediatr 1996 Mar;128(3):357-362. [Medline]
    18. Wang EE, Law BJ, Boucher FD, Stephens D, Robinson JL, Dobson S, et al. Pediatric Investigators Collaborative Network on Infections in Canada (PICNIC) study of admission and management variation in patients hospitalized with respiratory syncytial viral lower respiratory tract infection. J Pediatr 1996 Sep;129(3):390-395. [Medline]
    19. Christakis DA, Cowan CA, Garrison MM, Molteni R, Marcuse E, Zerr DM. Variation in inpatient diagnostic testing and management of bronchiolitis. Pediatrics 2005 Apr;115(4):878-884. [CrossRef] [Medline]
    20. Marlais M, Evans J, Abrahamson E. Clinical predictors of admission in infants with acute bronchiolitis. Arch Dis Child 2011 Jul;96(7):648-652. [CrossRef] [Medline]
    21. Laham FR, Trott AA, Bennett BL, Kozinetz CA, Jewell AM, Garofalo RP, et al. LDH concentration in nasal-wash fluid as a biochemical predictor of bronchiolitis severity. Pediatrics 2010 Feb;125(2):e225-e233 [FREE Full text] [CrossRef] [Medline]
    22. Corneli HM, Zorc JJ, Holubkov R, Bregstein JS, Brown KM, Mahajan P, Bronchiolitis Study Group for the Pediatric Emergency Care Applied Research Network. Bronchiolitis: clinical characteristics associated with hospitalization and length of stay. Pediatr Emerg Care 2012 Feb;28(2):99-103. [CrossRef] [Medline]
    23. Shaw KN, Bell LM, Sherman NH. Outpatient assessment of infants with bronchiolitis. Am J Dis Child 1991 Feb;145(2):151-155. [Medline]
    24. Shafik MH, Seoudi Tarek M M, Raway TS, Al Harbash Nowair Z, Ahmad Meshal M A, Al Mutairi Hanan F. Appropriateness of pediatric hospitalization in a general hospital in Kuwait. Med Princ Pract 2012;21(6):516-521. [CrossRef] [Medline]
    25. Mansbach JM, Clark S, Christopher NC, LoVecchio F, Kunz S, Acholonu U, et al. Prospective multicenter study of bronchiolitis: predicting safe discharges from the emergency department. Pediatrics 2008 Apr;121(4):680-688. [CrossRef] [Medline]
    26. McBride SC, Chiang VW, Goldmann DA, Landrigan CP. Preventable adverse events in infants hospitalized with bronchiolitis. Pediatrics 2005 Sep;116(3):603-608. [CrossRef] [Medline]
    27. Norwood A, Mansbach JM, Clark S, Waseem M, Camargo CA. Prospective multicenter study of bronchiolitis: predictors of an unscheduled visit after discharge from the emergency department. Acad Emerg Med 2010 Apr;17(4):376-382 [FREE Full text] [CrossRef] [Medline]
    28. Seiden JA, Scarfone RJ. Bronchiolitis: An Evidence-Based Approach to Management. Clinical Pediatric Emergency Medicine 2009 Jun;10(2):75-81. [CrossRef]
    29. Ralston SL, Lieberthal AS, Meissner HC, Alverson BK, Baley JE, Gadomski AM, American Academy of Pediatrics. Clinical practice guideline: the diagnosis, management, and prevention of bronchiolitis. Pediatrics 2014 Nov;134(5):e1474-e1502 [FREE Full text] [CrossRef] [Medline]
    30. Scottish Intercollegiate Guidelines Network. Bronchiolitis in children. A national clinical guideline.   URL: https://www.guidelinecentral.com/summaries/bronchiolitis-in-children-a-national-clinical-guideline/ [accessed 2015-09-21] [WebCite Cache]
    31. Bronchiolitis Guideline Team, Children's Hospital Medical Center: Evidence-based care guideline for management of first time episode bronchiolitis in infants less than 1 year of age. Cincinnati   URL: http://www.guideline.gov/content.aspx?id=34411 [accessed 2015-09-20] [WebCite Cache]
    32. Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA 2009 Feb 25;301(8):868-869. [CrossRef] [Medline]
    33. Walsh P, Rothenberg SJ, O'Doherty S, Hoey H, Healy R. A validated clinical model to predict the need for admission and length of stay in children with acute bronchiolitis. Eur J Emerg Med 2004 Oct;11(5):265-272. [Medline]
    34. Walsh P, Cunningham P, Rothenberg SJ, O'Doherty S, Hoey H, Healy R. An artificial neural network ensemble to predict disposition and length of stay in children presenting with bronchiolitis. Eur J Emerg Med 2004 Oct;11(5):259-264. [Medline]
    35. Destino L, Weisgerber MC, Soung P, Bakalarski D, Yan K, Rehborg R, et al. Validity of respiratory scores in bronchiolitis. Hosp Pediatr 2012 Oct;2(4):202-209 [FREE Full text] [Medline]
    36. Steyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating (Statistics for Biology and Health). New York, NY: Springer; 2009.
    37. Greenes R. Clinical decision support: the road to broad adoption, 2nd ed. London, UK: Academic Press; 2014.
    38. Kuhn M, Johnson K. Applied predictive modeling. New York, NY: Springer; 2013.
    39. Luo G, Nkoy FL, Gesteland PH, Glasgow TS, Stone BL. A systematic review of predictive modeling for bronchiolitis. Int J Med Inform 2014 Oct;83(10):691-714. [CrossRef] [Medline]
    40. Ségala C, Poizeau D, Mesbah M, Willems S, Maidenberg M. Winter air pollution and infant bronchiolitis in Paris. Environ Res 2008 Jan;106(1):96-100. [CrossRef] [Medline]
    41. Karr CJ, Demers PA, Koehoorn MW, Lencar CC, Tamburic L, Brauer M. Influence of ambient air pollutant sources on clinical encounters for infant bronchiolitis. Am J Respir Crit Care Med 2009 Nov 15;180(10):995-1001. [CrossRef] [Medline]
    42. Axelrod RC, Vogel D. Predictive Modeling in Health Plans. Disease Management & Health Outcomes 2003;11(12):779-787. [CrossRef]
    43. Asadi H, Dowling R, Yan B, Mitchell P. Machine learning for outcome prediction of acute ischemic stroke post intra-arterial therapy. PLoS One 2014;9(2):e88225 [FREE Full text] [CrossRef] [Medline]
    44. Evans RS, Lloyd JF, Pierce LA. Clinical use of an enterprise data warehouse. AMIA Annu Symp Proc 2012;2012:189-198 [FREE Full text] [Medline]
    45. Byington CL, Reynolds CC, Korgenski K, Sheng X, Valentine KJ, Nelson RE, et al. Costs and infant outcomes after implementation of a care process model for febrile infants. Pediatrics 2012 Jul;130(1):e16-e24 [FREE Full text] [CrossRef] [Medline]
    46. Luo G, Stone BL, Sakaguchi F, Sheng X, Murtaugh MA. Using Computational Approaches to Improve Risk-Stratified Patient Management: Rationale and Methods. JMIR Res Protoc 2015;4(4):e128 [FREE Full text] [CrossRef] [Medline]
    47. Air quality data homepage of the United States Environmental Protection Agency.   URL: http://www3.epa.gov/airdata/ [accessed 2016-01-04] [WebCite Cache]
    48. MesoWest homepage.   URL: http://mesowest.utah.edu/ [accessed 2016-01-04] [WebCite Cache]
    49. Halfon P, Eggli Y, van MG, Chevalier J, Wasserfallen JB, Burnand B. Measuring potentially avoidable hospital readmissions. J Clin Epidemiol 2002 Jun;55(6):573-587. [Medline]
    50. Heggestad T, Lilleeng SE. Measuring readmissions: focus on the time factor. Int J Qual Health Care 2003 Apr;15(2):147-154 [FREE Full text] [Medline]
    51. Pyle D. Data preparation for data mining. San Francisco, Calif: Morgan Kaufmann Publishers; 1999.
    52. Duncan I. Healthcare Risk Adjustment and Predictive Modeling (ACTEX Professional Series). Winsted, CT: ACTEX Publications, Inc; 2011.
    53. Al-Shehri MA, Sadeq A, Quli K. Bronchiolitis in Abha, Southwest Saudi Arabia: viral etiology and predictors for hospital admission. West Afr J Med 2005;24(4):299-304. [Medline]
    54. Wang VJ, Cavagnaro CS, Clark S, Camargo CA, Mansbach JM. Altitude and environmental climate effects on bronchiolitis severity among children presenting to the emergency department. J Environ Health 2012 Oct;75(3):8-15; quiz 54. [Medline]
    55. Nascimento MS, Souza Andréa Vieira de, Ferreira Adriana Vada de Souza, Rodrigues JC, Abramovici S, Silva Filho Luiz Vicente Ferreira da. High rate of viral identification and coinfections in infants with acute bronchiolitis. Clinics (Sao Paulo) 2010;65(11):1133-1137 [FREE Full text] [Medline]
    56. Witten I, Frank E, Hall M. Data Mining: Practical Machine Learning Tools and Techniques, Third Edition (The Morgan Kaufmann Series in Data Management Systems). Burlington, MA: Morgan Kaufmann; 2011.
    57. Hosmer JDW, Lemeshow S, Sturdivant RX. Applied Logistic Regression, 3rd ed. Hoboken, NJ: Wiley; 2013.
    58. Freitas AA. Comprehensible classification models: a position paper. SIGKDD Explor. Newsl 2014 Mar 17;15(1):1-10. [CrossRef]
    59. Vellido A, Martín-Guerrero J, Lisboa P. Making machine learning models interpretable. In: Proc. ESANN. 2012 Presented at: ESANN; April 25-27, 2012; Bruges, Belgium p. 163-172.
    60. Bradford W, Hurdle JF, LaSalle B, Facelli JC. Development of a HIPAA-compliant environment for translational research data and analytics. J Am Med Inform Assoc 2014;21(1):185-189 [FREE Full text] [CrossRef] [Medline]
    61. MLlib homepage.   URL: https://spark.apache.org/mllib/ [accessed 2015-08-10] [WebCite Cache]
    62. James BC, Savitz LA. How Intermountain trimmed health care costs through robust quality improvement efforts. Health Aff (Millwood) 2011 Jun;30(6):1185-1191 [FREE Full text] [CrossRef] [Medline]
    63. Consumer Price Index - Measuring price change for medical care in the CPI.   URL: http://www.bls.gov/cpi/cpifact4.htm [accessed 2015-08-10] [WebCite Cache]
    64. Narus SP, Srivastava R, Gouripeddi R, Livne OE, Mo P, Bickel JP, et al. Federating clinical data from six pediatric hospitals: process and initial results from the PHIS+ Consortium. AMIA Annu Symp Proc 2011;2011:994-1003 [FREE Full text] [Medline]

    ‎
    AUROC: area under the receiver operating characteristic curve
    ED: emergency department
    EDW: enterprise data warehouse

    Edited by G Eysenbach; submitted 22.09.15; peer-reviewed by A Zaki; comments to author 05.01.16; accepted 07.01.16; published 07.03.16

    Copyright

    ©Gang Luo, Bryan L. Stone, Michael D. Johnson, Flory L. Nkoy. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 07.03.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.