Author Affiliations: Department of Pediatrics, Montreal Children's Hospital, McGill University Health Centre, Montreal, Quebec (Dr Li), and Division of Paediatric Medicine and the Pediatric Outcomes Research Team, Hospital for Sick Children (Drs Parkin and Guttmann), Institute for Clinical Evaluative Sciences (Drs To, Anderson, and Guttmann), Child Health Evaluative Sciences, Hospital for Sick Children Research Institute (Drs To and Parkin), and Departments of Paediatrics (Drs Parkin and Guttmann) and Health Policy, Management, and Evaluation (Drs To, Parkin, Anderson, and Guttmann), University of Toronto, Toronto, Ontario, Canada.
Objective To determine whether children treated in emergency departments (EDs) with evidence-based standardized protocols (EBSPs) containing evidence-based content and format had lower risk of hospital admission or ED return visit and greater follow-up than children treated in EDs with no standardized protocols in Ontario, Canada.
Design Retrospective population-based cohort study of children with asthma. We used multivariable logistic regression to estimate risk of outcomes.
Setting All EDs in Ontario (N = 146) treating childhood asthma from April 2006 to March 2009.
Participants Thirty-one thousand one hundred thirty-eight children (aged 2 to 17 years) with asthma.
Main Exposure Type of standardized protocol (EBSPs, other standardized protocols, or none).
Main Outcome Measures Hospital admission, high-acuity 7-day return visit to the ED, and 7-day outpatient follow-up visit.
Results The final cohort made 46 510 ED visits in 146 EDs. From the index ED visit, 4211 (9.1%) were admitted to the hospital. Of those discharged, 1778 (4.2%) and 7350 (17.4%) had ED return visits and outpatient follow-up visits, respectively. The EBSPs were not associated with hospitalizations, return visits, or follow-up (adjusted odds ratio, 1.17 [95% CI, 0.91-1.49]; adjusted odds ratio, 1.10 [95% CI, 0.86-1.41]; and adjusted odds ratio, 1.08 [95% CI, 0.87-1.35], respectively).
Conclusions The EBSPs were not associated with improvements in rates of hospital admissions, return visits to the ED, or follow-up. Our findings suggest the need to address gaps linking improved processes of asthma care with outcomes.
Despite effective primary care strategies and outpatient quality improvement interventions to reduce acute health care services use for children with asthma,1- 3 the emergency department (ED) and hospital continue to be important sources of asthma care for children. In 2004, an estimated 754 000 visits for asthma were made by children to EDs in the United States, with 204 700 hospital admissions in the prior year.4 In Canada, asthma is the most common cause for hospitalizations in children aged 1 to 14 years.5
Evidence-based interventions outlining the management of acute asthma exist within national and international guidelines.6- 10 However, several studies have highlighted ongoing gaps in the quality of care provided. A sample of EDs in Ontario, Canada, demonstrated that only 35.2% and 31.7% of children had documented systemic steroid use in the ED and on discharge, respectively. These are evidence-based strategies shown to decrease hospitalizations and return visits to the ED.11- 13 A study examining 63 EDs in the United States observed a 46% reduced risk of hospitalization among adolescents and adults treated with processes of care recommended with the best evidence (level A) by the National Institutes of Health guidelines compared with those managed differently.14
To address gaps in care, EDs have explored the use of standardized protocols (SPs). These tools translate evidence and guideline recommendations into practice and exist in different formats including clinical practice guidelines, clinical pathways, preprinted orders, and medical directives (sometimes referred to as standing orders). There is evidence supporting formats of SPs that are part of patient medical records (including preprinted orders and clinical pathways). In a systematic review, key features of clinical decision support tools such as SPs that were associated with improved processes and outcomes included integration into the workflow, providing a recommendation rather than indicating an assessment, and computer-based systems.15
Evidence of the real-world effectiveness of SPs for childhood asthma is limited. Various types of SPs for childhood asthma in the ED have been tested mainly in single-center, before-and-after trials producing inconsistent results in processes and outcomes of care, including hospital admissions and ED return visits.16- 24 Reducing the latter 2 outcomes are widely affirmed health care priorities in Canada and the United States, where rates are reported as quality indicators for hospital and health system performance.25- 28 Improving follow-up visits is another important outcome, since they may prevent subsequent exacerbations and acute care use by ensuring continuity of care and chronic disease management.6,29,30
The objective of this study was to determine whether SPs containing evidence-based content and format (EBSPs) were associated with improved short-term childhood asthma outcomes. We hypothesized that EBSPs would reduce hospitalizations at the index ED visit and ED return visits within 7 days, as well as increase 7-day outpatient follow-up.
This study was a retrospective population-based cohort of children 2 to 17 years old with asthma treated in an ED in Ontario between April 14, 2006, and February 28, 2009. We used 4 linked health administrative data sets available at the Institute for Clinical Evaluative Sciences and survey data to determine the association of SPs with short-term asthma outcomes. Research ethics board approval was granted from Sunnybrook Health Sciences Centre, the Hospital for Sick Children, and the University of Toronto.
We obtained available SPs by conducting a survey (October 2009-February 2010) of all Ontario EDs who manage children with asthma. We used population-based data from the Canadian Institute for Health Information National Ambulatory Care Reporting System and Discharge Abstract Database, as well as the Ontario Health Insurance Plan claims database, to identify ED visits, hospitalizations, and outpatient follow-up visits, respectively. We used the Ontario Asthma Surveillance Information System, a validated population-based registry using administrative data, to identify children with previously diagnosed asthma.31 We used survey data from previous work21 to ascertain the frontline ED physician staffing model.
Our cohort consisted of all children 2 to 17 years old with previously diagnosed asthma (from the Ontario Asthma Surveillance Information System) with an unplanned visit to an Ontario ED for asthma (International Statistical Classification of Diseases,10th Revision, Canada code J45) in the fiscal years starting April 2006 and ending March 2009. We ended enrollment February 28, 2009, so that we could assess 7-day outcomes to March 7, 2009. Records for hospitalizations are found in the fiscal year of the date of discharge so we allowed a 3-week period to include all admissions that would have initiated in the first week of March 2009. We included only children with known asthma so that at presentation to the ED the diagnosis of an acute exacerbation should be more evident and an SP initiated if available. The SPs also often require the patient has a prior diagnosis before therapy initiation, especially medical directives allowing nurses or respiratory therapists to give medications. We excluded those with a visit in the 14 days prior to ensure they were not return visits for unresolved exacerbations. In Ontario, most children admitted to the hospital for asthma go through the ED, so this exclusion also captured those with a hospitalization in the prior 14 days. Children seen in EDs with low annual volumes for pediatric asthma (≤10th percentile) were excluded. The latter EDs treated fewer than 32 children with asthma per year, which we considered to be an insufficient number of patients to contribute adequate data on their overall management of acute asthma.
For all Ontario EDs managing children with asthma during the study time frame (April 2006-March 2009), one of us (P.L.) contacted ED administrators and directors previously identified as best respondents for Ontario Hospital Report25 surveys to determine the use of pediatric asthma SPs and obtain a copy. Two of us (P.L. and A.G.) categorized SPs into 4 formats, preprinted orders, clinical practice guidelines, medical directives, and clinical pathways, using the following definitions.32,33 Preprinted orders have preset management suggestions used as orders requiring a physician signature. Clinical practice guidelines are “systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances.”34(p38) Medical directives contain standing orders that nurses or respiratory therapists can initiate autonomously. Clinical pathways are structured multidisciplinary plans of care with at least 3 other features (translated guidelines or evidence, detailed steps of treatment, provided time frames and criteria-based progression, and/or standardized care).33 Based on the literature for clinical decision-support tools and SP implementation, we defined formats that were embedded into workflow as evidence based (ie, preprinted orders and clinical pathways).15
Using evidence from a literature review and guidelines from the Global Initiative for Asthma2,3 and US National Asthma Education and Prevention Program,4 we examined whether SPs had the evidence-based content related to improving hospital admissions, ED return visits, and outpatient follow-up (designated as “required content” in Table 1). The SPs were categorized as having evidence-based content if they contained practices supported by the best level of evidence (A, if available).
The EDs were categorized as having EBSPs (containing evidence-based format and content), other SPs (lacking evidence-based format and/or content), or none. We took into account the date when EDs implemented the SPs so that patients were assigned to the SP type that existed at the date of their ED visit.
The primary outcome was hospital admission at the index ED visit. Secondary outcomes included ED return visits and outpatient follow-up within 7 days among nonadmitted children. For return visits, we included unplanned visits to any Ontario ED for asthma and any diagnoses related to the respiratory system. Only high-acuity visits identified with the validated Canadian Triage and Acuity Scale36- 40 triage score of 3, 2, or 1 (urgent, emergent, and resuscitation, respectively) were included. The time frame of 7 days was chosen to accurately capture most relapses without falsely including new exacerbations, although morbidity from an acute event may continue for 7 to 15 days after discharge.41
Outpatient follow-up visit within 7 days was defined either as an office-based physician visit or a planned nonurgent return visit to the ED. Although guidelines recommend follow-up within 1 to 4 weeks after an acute exacerbation for asthma,6- 8 some authors advocate for follow-up 1 week or sooner.42 We chose the time frame of 1 week to include follow-up for the current exacerbation only.
We adjusted for patient-level covariates that could affect asthma outcomes, including sex, age, and neighborhood income quintile. The latter approximated the socioeconomic status of each child by linking the postal code with the mean neighborhood income within the dissemination area (average population, 650) from the 2006 Canadian Census. To adjust for acute severity, we used the Canadian Triage and Acuity Scale triage score of the index ED visit. To adjust for chronic severity, we used the history of asthma ED visits and hospitalizations in the prior 2 years. As factors that may reflect access or propensity to use ED care, we examined the history of ED visits for any diagnoses within the past 2 years and the distance from the child's home to the hospital. For the analyses involving ED return visits, we also considered 2 additional covariates: (1) outpatient follow-up visits and (2) the interaction term involving outpatient follow-up and SP type, since follow-up could act as both a confounder and effect modifier on the main exposure (SP type).
We examined hospital-level covariates including rurality, hospital type, and the annual pediatric asthma patient volume. A rural ED was defined as being outside the commuting zone of larger urban centers with a population of 10 000 or more using the Statistics Canada definition.43 The 4 hospital types were defined using the Ontario Joint Policy and Planning Committee definitions of teaching and community hospitals and survey data on frontline ED physician staffing.21 The Joint Policy and Planning Committee–defined teaching hospitals were divided as “pediatric” if EDs were staffed by pediatric ED-trained physicians or pediatricians vs “teaching” hospitals. Community hospitals were divided by whether pediatricians were available for consultation.
Categorical variables were described as frequencies and proportions. For the distance from home to the hospital, the distribution was skewed and log-transformed. To model the relationship between SPs and outcomes adjusting for significant patient- and hospital-level covariates, we used logistic regression with generalized estimating equations that accounted for clustering of patient outcomes by ED.44 Statistical significance was defined as a 2-tailed P < .05.
A total of 165 EDs managed children with asthma in Ontario, of which 17 were excluded because of low volumes and 2 could not be contacted. A total of 43 (29.5%) of the 146 EDs had SPs. Table 2 shows the SPs grouped for each outcome into EBSPs, other SPs, and no SP.
Table 3 describes the characteristics of the children and the EDs they visited by SP type. The final cohort consisted of 46 510 visits from 31 138 unique children managed in 146 EDs. Younger children presenting with a higher acuity score were more likely to be treated in EDs with EBSPs.
Of the 46 510 index ED visits, there were 4211 hospital admissions (9.1%). Of those not admitted (n = 42 299), 1778 (4.2%) had a high-acuity ED return visit and 7350 (17.4%) had a follow-up visit within 7 days. No significant associations between SPs and outcomes were observed in the multivariable analysis (Table 4).
In this population-based study, we examined the use of SPs as a strategy to improve the quality of acute asthma care. We linked data from a survey with a high response rate to key outcomes allowing us to examine the real-world effectiveness of SPs for childhood asthma in Ontario EDs. As currently implemented, we observed no impact of EBSPs on hospital admissions, ED return visits, and follow-up. Although we observed low follow-up (17.4%), these visits were associated with reduced odds of ED recidivism within the first week, supporting the recommendations of published guidelines for post-ED care.6- 8
The failure to demonstrate a significant effect of SPs may be explained in part by the quality and implementation of EBSPs. There were no computerized SPs, for which the literature is increasingly demonstrating support.15,45,46 A small single-center preintervention and postintervention study using a computerized decision support tool for asthma in an adult ED showed improved documentation of clinical parameters and discharge plans.45 Cost and time efficiency were not reported, but this study lends promise to the potential of ED-based computerized systems. A cluster randomized trial of clinical decision support embedded in electronic health records for primary care physicians also showed significantly improved adherence to national asthma guidelines.46
Several nonrandomized trials for asthma SPs in the ED have demonstrated improved processes of care (such as prescribing steroids and discharge planning)17- 20,22- 24 but few have shown improved asthma outcomes.19,24 For the current study, the quality of care may have improved from adherence to evidence-based processes of care but these were not reflected in our measured outcomes. For example, although SPs may have improved physician reminders for follow-up care, other barriers must be overcome to achieve these visits, including parental (beliefs and barriers)47,48 and health systems (access to providers) factors.49,50 The missing link between processes and outcomes is further supported by a recent study examining pediatric quality indicators of asthma care in 14 US-based EDs, where receipt of 1 or all evidence-based treatments did not predict successful ED discharges (ie, not resulting in return visits or significant ongoing symptoms).51
Our baseline rates for hospitalizations and ED return visits, which were similar to those previously reported in Ontario,21 may have been too low to observe a further decrease resulting from a single intervention in the ED. Pediatric asthma is a chronic disease with complex interdisciplinary pre-ED and post-ED management factors affecting the outcomes of each acute episode, such as the availability of a primary care medical home.1,2 In previous single-center studies demonstrating significant effects using SPs, the baseline hospitalization rates were as high as 27.5% to 32.0% and ED return visit rates ranged from 8.0% to 8.4%,16,19 whereas in the current study, they were 9.1% and 4.2%, respectively.
Our study was limited by the observational design. Associations but not causality could be inferred. Two methods to adjust for confounding in observational studies include regression and propensity scores. We used regression because in a systematic review of studies using both methods, either produced similar results.52 The quality of propensity scores depend on the data available and scores developed using administrative data may not always balance patient characteristics found within clinical data.53 We did not have the data to adjust for patient factors including the degree of self-management skills and prior asthma education,54 ethnicity,55 allergen/environmental exposures, and controller medication use.6 In the absence of the latter, we used history of hospitalizations and ED visits to adjust for asthma severity similar to previous studies.21,56,57
Although we successfully retrieved and examined the majority of SPs for childhood asthma in Ontario, we were unable to assess the extent to which they were implemented. The decision to comply with protocols, as well as the ultimate decision to hospitalize a child, remained in the hands of individual ED physicians. There is some evidence that physicians' management of childhood asthma reflects recommendations for SPs produced within their institutions. One study across 11 large pediatric EDs in New Zealand and Australia found that there was generally good agreement with the ED-specific clinical practice guidelines and physician practice, although the divergence was observed with disease severity (where the evidence seemed less clear).58
To our best knowledge, this is the first study examining in detail all SPs available for childhood asthma in a population-based sample across a wide range of ED settings. Although the literature suggests that evidence-based, rigorously implemented SPs in the trial setting may improve processes of care and to a lesser extent, outcomes, as currently implemented in Ontario, we did not observe an impact of EBSPs on short-term asthma outcomes. Our study does not have sufficient evidence to dispute the potential importance of SPs in quality improvement activities. With increasing policies and programs mandated to enhance quality of care, our findings point to the need for mechanisms to update the content and ensure widespread dissemination of SPs, but likely, effective implementation will continue to be an important research and policy need. Future studies should also identify and address the gaps linking improved processes of acute asthma care with outcomes.
Correspondence: Astrid Guttmann, MDCM, MSc, Institute for Clinical Evaluative Sciences, G1 06, 2075 Bayview Ave, Toronto, ON M4N 3M5, Canada (email@example.com).
Accepted for Publication: April 12, 2012.
Published Online: July 9, 2012. doi:10.1001/archpediatrics.2012.1195
Author Contributions:Study concept and design: Li, To, Parkin, Anderson, and Guttmann. Acquisition of data: Li and Guttmann. Analysis and interpretation of data: Li, Parkin, and Guttmann. Drafting of the manuscript: Li and Guttmann. Critical revision of the manuscript for important intellectual content: Li, To, Parkin, Anderson, and Guttmann. Statistical analysis: Li, To, and Guttmann. Obtained funding: Li and Guttmann. Administrative, technical, and material support: Li and Guttmann. Study supervision: To, Anderson, and Guttmann.
Financial Disclosure: None reported.
Funding/Support: Dr Li received an Academic Pediatric Association/Agency for Healthcare Research and Quality Young Investigator Award for this study, Dr To is supported by the Dales Award, University of Toronto, Dr Guttmann is a recipient of a Canadian Institutes of Health Research New Investigator Award and a Canadian Child Health Clinician Scientist Career Enhancement Program Award, and the Pediatric Outcomes Research Team is supported by a grant from the Hospital for Sick Children Foundation. This study was performed at the Institute for Clinical Evaluative Sciences, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care.
Disclaimer: The opinions, results, and conclusions reported in this article are those of the authors and are independent from all funding sources. No endorsement by the Institute for Clinical Evaluative Sciences or the Ontario Ministry of Health and Long-Term Care is intended or should be inferred.
Additional Contributions: We thank Kelvin Lam, MSc, data analyst at the Institute for Clinical Evaluative Sciences, for extracting the preliminary health services data.
Li P, To T, Parkin PC, Anderson GM, Guttmann A. Association Between Evidence-Based Standardized Protocols in Emergency Departments With Childhood Asthma OutcomesA Canadian Population-Based Study. Arch Pediatr Adolesc Med. 2012;166(9):834-840. doi:10.1001/archpediatrics.2012.1195