To use a large national comparative database to measure the internal effect of a set of evidence-based bronchiolitis/viral pneumonia clinical care guidelines on clinical practice at a children's hospital, and to compare these changes with those at other children's hospitals.
Prospective cohort study with retrospective and concurrent (other hospital) controls.
The Children's Hospital, Denver, Colo.
Hospitalized children with bronchiolitis and/or viral pneumonia.
Our clinical guidelines focused on clear admission and discharge criteria, individualized transition-anticipating orders, and "prove it or don't use it" criteria for the use of respiratory syncytial virus testing, bronchodilators, chest physiotherapy, and ribavirin.
Main Outcome Measures
The effect of guideline implementation was determined by comparative measurement of internal changes in utilization and outcome (nosocomial infection rate) across time and by external comparison with other children's hospitals using standardized data from the Pediatric Health Information System database of the Child Health Corporation of America (Shawnee Mission, Kan).
Overall, 10 636 bronchiolitis/viral pneumonia cases were studied: 1302 at the index hospital and 9334 at the 7 comparison hospitals. Internally, the index hospital's residents and attending physicians responded favorably to the bronchiolitis/viral pneumonia care guidelines, resulting in decreases in targeted resource utilization. There were no fatalities, and the number of days in the intensive care unit decreased even though the mean severity of admitted cases increased significantly. Targeted utilization was favorably affected, whereas untargeted utilization was not. Nosocomial infections did not increase with a decreased use of respiratory syncytial virus testing. The index hospital differed favorably from other children's hospitals in several categories.
Evidence-based care guidelines can successfully influence utilization and clinical outcome.
IT HAS BEEN widely assumed that clinical care guidelines can favorably influence utilization and clinical outcome by agreeing on and disseminating evidence-based, peer-recommended information for patient care.1,2 One common childhood disease that has been successfully improved using this process is bronchiolitis/viral pneumonia. Several investigators have reported improved compliance with recommendations for laboratory utilization and medication use in bronchiolitis using care guidelines.3- 5 The overall value of guidelines and whether observed reductions in utilization variation actually result in improved outcomes is debated.6 In addition, collaborative efforts to measure these effects may require the definition of new variables and labor-intensive, expensive data collection methods such as questionnaires and medical record review.7,8
Attempts to use internal administrative databases for the evaluation of guidelines can be frustrated by the lack of existing data for risk adjustment and/or outcome measurement as well as the use of charge-based variables influenced by changing pricing practices. Many hospitals subscribe to external comparative databases that may have the same limitations but can potentially compare their practices with similar institutions.9,10 This added comparative dimension is at the core of the Joint Commission on Accreditation of Healthcare Organizations'11 ORYX initiative, which requires hospitals to select and measure predefined clinical indicators on a population basis and compare them quarterly with a peer group of comparable hospitals. We employed a similar measurement process in the evaluation of a set of bronchiolitis/viral pneumonia clinical guidelines using a national collaborative, children's hospital database.
In 1994, we implemented a collaborative process for the development of evidence-based clinical care guidelines at The Children's Hospital of Denver (TCH), Denver, Colo. This system included wide representation from specialists and community physicians; it focused on areas of agreement, rather than forcing resolution of strong disagreement, and a commitment to develop practical means of measuring the effect of these guidelines on an ongoing basis. The bronchiolitis/viral pneumonia guidelines were implemented in 1995.
Bronchiolitis/viral pneumonia was operationally defined as an acute lower respiratory illness in a previously healthy child with no history of asthma associated with tachypnea or retractions, diffuse wheezing and/or crackles, and no evidence of bacterial infection. Admission criteria included a complicating condition, an oxygen requirement that could not be accommodated at home, apnea, poorly responsive neurological status, difficulty feeding, or suspicion of sepsis. Testing for respiratory syncytial virus (RSV) was discouraged for bed placement; instead, patients with similar symptoms were grouped, and droplet precautions (hand washing, gloves, gown, and mask) were strictly enforced for direct patient contact.12
A "prove it or don't use it" policy was encouraged, which stated that laboratory and imaging studies and therapies should be performed or continued only if they were anticipated and/or proved to make a difference in each patient's clinical course.4,13,14 As a result, the continued use of bronchodilators and/or chest physiotherapy required observed improvement as measured by a defined respiratory distress score that evaluated age-specific respiratory rate, retractions, wheezing, and aeration.15,16 The use of ribavirin was recommended only in patients with impending respiratory failure.17 Respiratory syncytial virus antigen testing or viral culture was advised only if it was anticipated that a positive test result would cause a change in antimicrobial therapy.14 No recommendations were made regarding chest imaging or antibiotic use, although more recent studies suggest a lack of general clinical usefulness in patients with bronchiolitis.13,18,19 These 2 variables were useful controls for the targeted variables, with the hypothesis that only targeted variables would be altered with the implementation of clinical guidelines. A protocol was provided for the weaning of oxygen based on the rate of clinical recovery, including the ability to feed, decreasing respiratory distress, and absence of apnea and cyanosis. Discharge criteria included improved work of breathing, discontinuation of oxygen or a referral for home oxygen therapy, adequate oral intake, parental education regarding home oxygen therapy, and follow-up arranged with the primary care physician.
We hypothesized that the use of an existing comparative database could facilitate the evaluation of complex clinical care guidelines. Specifically, this meant that unique effects could be inferred if consistent postintervention intra-institutional effects on targeted variables were documented during sequential years, with additional support if the index institution differed favorably in comparison with other institutions using standardized data from the Pediatric Health Information System (PHIS) database of the Child Health Corporation of America (Shawnee Mission, Kan). This database currently contains discharge data from 29 children's hospitals geographically distributed throughout the United States. Available data elements abstracted and coded by the collaborating medical records departments using PHIS data quality guidelines include the following: standard demographic data, dates of service, International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM)20 diagnostic and procedural codes, utilization data for the date and unit level, and charges. Cases were grouped using the All Patient Refined Diagnosis-Related Groups version 12 software (3M Company, Salt Lake City, Utah).10 This well-known software uses complex clinical algorithms based on age and ICD-9-CM diagnostic and procedural codes to classify patients into diagnosis-related groups and assign relative severity scores ranging from 1 (least severe) to 4 (most severe). Infection control statistics were reviewed to determine the rates of nosocomial infection with RSV from December through April of each study year, based on the Centers for Disease Control's definition of symptoms beginning more than 72 hours after admission; viral testing was encouraged in such cases.
The RSV season in Colorado predictably occurs during the winter, and bronchiolitis and viral pneumonia cases at other times of the year may have different causes and outcomes. Therefore, we restricted our analysis to cases from January through March for 1994 to 1998. Because RSV can cause bronchiolitis and/or viral pneumonia, because we did not encourage etiologic testing of cases, and because new relevant codes were added during the study period, we selected patients from birth to age 21 years based on the following ICD-9-CM principal diagnostic codes using data from the PHIS database: pneumonia, organism unspecified (486); acute bronchiolitis (4661); pneumonia due to RSV (4801); viral pneumonia of unspecified cause (4809); acute bronchiolitis due to RSV (46 611); and acute bronchiolitis due to other organisms (46 619). All patient and comparison hospital identifiers were removed before analysis. We compared 1 preguideline year (1994) with 3 postguideline seasons (1995-1997) for TCH as compared with 7 other children's hospitals that had complete data and consistent coding of key variables and services for those years. Several of these hospitals had implemented bronchiolitis guidelines during the study period.
A hospital survey for price changes from 1994 to 1997 revealed inconsistent practices both between and within individual hospitals. Fiscal years varied. Some hospitals implemented across-the-board increases, whereas others were more selective; increases varied from 0% to 7%. Because of the results of this survey, we elected not to use charges as a measure of utilization. Given that even the definition of drug units may differ between hospitals or within the same hospital across time, we determined that the 2 most reliable measures of utilization would be days of use and any use of a test or therapy during a particular hospitalization. Then we created dummy variables for these items for each patient. The estimated length of stay (LOS) in days was calculated by subtraction of the date of admission from the date of discharge, not accounting for fractional days. To account for LOS outliers that might bias continuous variable comparisons, we excluded cases that exceeded the Health Care Financing Administration's LOS outlier definition for 1997: the geometric mean length of stay plus the lesser of 24 days or 3 SDs. This equaled 19.81 days for this data set, so all patients with an LOS greater than 19 were excluded.21 Calculations for LOS excluded any patient who did not have a discharge status of home or home health.
All statistical analyses were conducted using SPSS version 10 statistical software (SPSS Inc, Chicago, Ill). We assumed that new clinical guidelines were effective if intra-institutionally, consistent statistically significant trends could be documented during sequential years and inter-institutionally, if the index hospital (TCH) differed significantly in a favorable direction from the comparison institutions. Given the assumption of index hospital admissions for bronchiolitis/viral pneumonia of 250 to 300 cases per year, the power to detect a change in utilization from 50% to 40% was 0.65 and from 50% to 35% was 0.90. The χ2 test was used to compare proportionate data, and because a normal data distribution could not be assumed, the Mann-Whitney test was used to compare continuous and categorical variables (LOS and severity score). Single-entry logistic regression was used to evaluate internal changes (dependent variable: use or nonuse of targeted variable; independent variable: year) during the 4 study years. Confidence intervals were calculated for the difference between 1994 and 1997 for the index hospital as compared with the other institutions. All P values are reported as 2-tailed probabilities.
A total of 10 636 cases of bronchiolitis (68.2%) and viral pneumonia (31.8%) were identified using ICD-9-CM codes (Table 1), with patients ranging in age from birth to 21 years (mean, 1.7 years). There were no associated fatalities at the index hospital during that time. Although an ICD-9-CM coding shift occurred in 1997, total case numbers were relatively comparable from year to year.
The effect of the bronchiolitis and viral pneumonia guidelines at the index hospital was first estimated by comparing the temporal influence on its targeted variables (Table 2). Between 1994 and 1997, there was a significant (P = .02 using the Mann-Whitney test) increase in mean severity, with the mean severity score rising from 1.49 to 1.70 (outlier-trimmed data). However, there was no significant change in mean LOS. Despite the increased mean severity score, there was a significant (P<.001) decrease (from 20.2% to 7.8%) in the proportion of patients admitted to the intensive care unit (ICU). This was accompanied by a significant increase in the percentage of ICU patients with bronchiolitis/viral pneumonia who had a severity score of 3 or higher (from 32.1% to 60.9%). There were significant decreases in utilization of the targeted variables of RSV testing, ribavirin treatment, and chest physiotherapy but no significant changes in the untargeted variables of antibiotic treatment and chest imaging.
As shown in Figure 1 with logistic regression, these changes in utilization were progressive during the 3 years following implementation of the guidelines. There was no overall decrease in the targeted use of bronchodilators; however, there was a significant decrease among patients treated for 1 day and an increase among those never treated or treated for 2 or more days (Figure 2) that was consistent with the guidelines' "prove it or don't use it" recommendations. Despite the decrease in targeted RSV testing, there was no increase in the rate of nosocomial RSV infections (Table 3).
Association of bronchiolitis/viral pneumonia care guidelines with individual patient use of targeted variables (chest physiotherapy [CPT], respiratory syncytial virus [RSV] testing, and ribavirin) and pediatric intensive care unit (ICU) admissions during sequential years. Guidelines were implemented in 1995. Odds ratios (ORs) and 95% confidence intervals indicate yearly decrease in utilization by single-variable entry logistic regression, with the dependent variable being use or nonuse of a listed test or therapy.
Association of bronchiolitis/viral pneumonia care guidelines with bronchodilator use at the index hospital during sequential years. Guidelines were implemented in 1995. Overall, P<.001 using the Pearson χ2 test.
As an additional method to evaluate the effect of our implementation of clinical care guidelines, the results of the comparison of targeted and untargeted variables from our index hospital with those from 7 other children's hospitals for the 1994-1997 period are presented in Table 2. Both the index and comparison hospitals demonstrated significant increases in severity score, although this score was significantly higher for the index hospital than for the comparison hospitals in 1997 (1.73 vs 1.59; P = .04), with a lower ICU admission rate (7.8% vs 13.5%; P = .006). The difference from 1994 to 1997 between the change in the percentage of ICU patients was significantly decreased for the index hospital compared with the other institutions (difference, −16.6%; 95% confidence interval, −10.8% to −22.4%). During this period, we were able to decrease our use of chest physiotherapy relative to the comparison hospitals (difference, −29.7%; 95% confidence interval, −21.6% to −37.8%), but our use of this treatment still exceeded the comparison hospitals in 1997. Ribavirin use decreased in both groups, consistent with changing national guidelines. Respiratory syncytial virus testing, which was comparable in 1994, decreased in both groups by 1997. The use of bronchodilators at any time during hospitalization actually increased at the comparison institutions to levels observed during both periods at the index hospital.
In this study, we used the Child Health Corporation of America's existing PHIS database to evaluate the effect of a set of bronchiolitis and viral pneumonia clinical care guidelines on patient care practices. We applied the Joint Commission on Accreditation of Healthcare Organizations' concept of performing both temporal intra-institutional and comparative inter-institutional analyses. Our results are comparable with other recent studies of bronchiolitis.3- 5 Intra-institutionally, our residents and attending physicians appeared to respond favorably to the bronchiolitis/viral pneumonia care guidelines, resulting in targeted resource changes with no fatalities, a reduction in ICU admissions despite increased severity, and no increase in nosocomial infection rates even though RSV testing was not used for bed placement.12,22 The latter observation suggests that cohorting based on symptoms rather than RSV testing may be sufficient to prevent nosocomial infection. The untargeted reduction in ICU admissions appeared to be due to a significant decrease in the number of patients in the ICU with severity scores lower than 3. Besides the observed temporal changes, evidence supporting the effectiveness of the guidelines included the fact that targeted utilization responded favorably, whereas untargeted utilization (chest x-rays and antibiotics) did not. More recent publications suggest that these latter treatments may be considered targets for improvement as well.13,23,24 Inter-institutional comparison with 7 other hospitals in the database supported the index hospital's observations. It is likely that many of these children's hospitals implemented similar clinical care guidelines during the study period, based on evolving evidence-based trends in the care of bronchiolitis and viral pneumonia as reflected by comparable significant decreases in RSV testing and ribavirin use (although the use of antibiotics and bronchodilators remains high). Our study suggests that old practice patterns (eg, chest physiotherapy and bronchodilators) may be slow to change, validating the need for ongoing reinforcement, evaluation, and updating of guidelines.
In the process of this study, the strengths and limitations of working with a large comparative, administrative database became apparent. Because ICD-9 codes and coding practices change, it is imperative to check for internal and external coding consistency. Temporal trends in charge data are confounded by variable price increases (intra-institutionally and inter-institutionally), making charge comparisons unsatisfactory for most purposes. Nonetheless, in large databases, dummy variables such as days of use and/or any use of tests or therapies may have sufficient statistical power to show significant and clinically relevant differences. A noticeable deficiency in these databases is the lack of true outcome data, so comparisons of utilization with evidence-based precedents become a necessary but incomplete surrogate for true outcome measurement.
Accepted for publication June 7, 2002.
Presented at the Pediatric Academic Societies Meeting, San Francisco, Calif, May 4, 1999.
Corresponding author: James Todd, MD, Department of Epidemiology, The Children's Hospital, 1056 E 19th Ave, Denver, CO 80218 (e-mail: firstname.lastname@example.org).
Bronchiolitis and viral pneumonia are currently the most common causes of pediatric hospital admission. Evidence-based clinical care guidelines have been shown to influence selected aspects of care.
The association of clinical care guidelines with temporal changes in resource utilization for bronchiolitis and viral pneumonia is confirmed. Respiratory syncytial virus testing may not be needed to determine bed placement as long as droplet precautions are maintained for cohorts of patients with similar symptoms. Local and national comparisons suggest that evidence-based changes in practice will require ongoing reinforcement and measurement to be sustained.
Todd J, Bertoch D, Dolan S. Use of a Large National Database for Comparative Evaluation of the Effect of a Bronchiolitis/Viral Pneumonia Clinical Care Guideline on Patient Outcome and Resource Utilization. Arch Pediatr Adolesc Med. 2002;156(11):1086–1090. doi:10.1001/archpedi.156.11.1086