The overall and sensitivity analyses, excluding outlier population, are shown here. Weights are from random effects analysis.
The subgroup analysis by imbalance in day of illness (DOI) at presentation is shown here. Weights are from random effects analysis.
Subgroup analysis postulating a threshold response for LOS of 3 days or more. A, Admission to discharge. No correction for imbalance in day of illness at presentation. B, First symptoms to discharge. Corrected for imbalance in day of illness at presentation. WMD indicates weighted mean difference. Weights are from random effects analysis.
eTable. Results of Meta-regression Between Length of Stay (Days) and Study-, Treatment-, and Patient-Level Explanatory Variables
Customize your JAMA Network experience by selecting one or more topics from the list below.
Brooks CG, Harrison WN, Ralston SL. Association Between Hypertonic Saline and Hospital Length of Stay in Acute Viral Bronchiolitis: A Reanalysis of 2 Meta-analyses. JAMA Pediatr. 2016;170(6):577–584. doi:10.1001/jamapediatrics.2016.0079
Two previous meta-analyses of nebulized hypertonic saline (HS) on hospital length of stay (LOS) in acute viral bronchiolitis have suggested benefit. Neither study fully addressed the issue of excessive heterogeneity in the cohort of studies, indicating that it may be inappropriate to combine such dissimilar studies to estimate a common treatment effect.
To reanalyze the existing data set for sources of heterogeneity to delineate the population most likely to benefit from HS.
We used the previously analyzed cohort of randomized trials from 2 published meta-analyses comparing HS with normal saline (or, in 1 case, with standard of care) in infants hospitalized for bronchiolitis. We also repeated the search strategy used by the most recent Cochrane Review in the Medline database through September 2015.
Eighteen randomized clinical trials of HS in infants with bronchiolitis reporting LOS as an outcome measure were included.
Data Extraction and Synthesis
The guidelines used for abstracting data included LOS, study year, setting, sample size, type of control, admission/discharge criteria, adjunct medications, treatment frequency, mean day of illness at study enrollment, mean severity of illness scores, and mean age.
Main Outcomes and Measures
Weighted mean difference in LOS and study heterogeneity as measured by the I2 statistic.
There were 18 studies included of 2063 infants (63% male), with a mean age of 4.2 months. The mean LOS was 3.6 days. Two main sources of heterogeneity were identified. First, the effect of HS on LOS was entirely sensitive to the removal of one study population, noted to have a widely divergent definition of the primary outcome. Second, there was a baseline imbalance in mean day of illness at presentation between treatment groups. Controlling for either of these factors resolved the heterogeneity (I2 = reduced from 78% to 45% and 0%, respectively) and produced summary estimates in support of the null hypothesis (that HS does not affect LOS). There was a weighted mean difference in LOS of −0.21 days (95% CI, −0.43 to +0.02) for the sensitivity analysis and +0.02 days (95% CI, −0.14 to +0.17) for studies without unbalanced treatment groups on presentation.
Conclusions and Relevance
Prior analyses were driven by an outlier population and unbalanced treatment groups in positive trials. Once heterogeneity was accounted for, the data did not support the use of HS to decrease LOS in infants hospitalized with bronchiolitis.
Viral bronchiolitis represents a major burden on hospitals caring for children as one of the most common illnesses resulting in hospital admission in infants in the United States.1,2 Current clinical practice guidelines in the United States do not recommend the routine use of any medication for bronchiolitis therapy based on systematic review and meta-analysis of the available literature.3
Recently, however, nebulized hypertonic saline (HS) has been proposed to shorten hospital length of stay (LOS) in several meta-analyses, although significant heterogeneity was reported across the included studies (I2 = 82.1%), with mean LOS in the control groups varying from 1.8 to 7.4 days across 15 studies.4-6 In meta-analyses, some variability in treatment effect is to be expected between studies, whether owing to chance (ie, sampling effects) or meaningful differences between trials (eg, patient age, inclusion criteria, or dosing). The term heterogeneity refers to a statistical measure of the observed variability between study results, specifically intended to measure the likelihood that this variability is owing to something more than chance. Excessive heterogeneity indicates that care should be taken to further investigate the potential for differences in study design, population, or characteristics of the intervention. Random effects modeling is a statistical response to excess heterogeneity in meta-analysis; however, many experts believe that random effects modeling alone does not solve the problem of study heterogeneity and that one of the greatest benefits of conducting a meta-analysis is to more carefully examine sources of heterogeneity when present.7
None of the published analyses completely addresses and accounts for the large amount of study heterogeneity in the cohort of studies reporting the effect of HS on hospital LOS in bronchiolitis. Hence, we undertook a reanalysis of this data set to fully investigate potential study-, treatment-, or patient-level factors contributing to the heterogeneity to delineate any population most likely to benefit from HS.
Question Does hypertonic saline affect hospital length of stay in acute viral bronchiolitis once heterogeneity was resolved from recent meta-analyses?
Findings In this reanalysis, the appearance of a reduction in hospital length of stay for patients with bronchiolitis treated with hypertonic saline was entirely sensitive to the removal of an outlier study population as well as a baseline imbalance in day of illness at presentation between treatment groups.
Meaning Hypertonic saline cannot be expected to reduce hospital length of stay in acute viral bronchiolitis in a typical US population.
Included studies were randomized clinical trials of HS (3% or 5%) vs 0.9% saline, with or without adjunct medication use (ie, bronchodilators), or of HS vs standard care in hospitalized children, reporting our primary outcome of LOS and identified from previously published meta-analyses.4-6 We also repeated the search strategy used by the most recent Cochrane Review in the Medline database through September 2015, revealing no previously unidentified studies. The search terms included the following: (1) exp Bronchiolitis/; (2) (bronchiolit* or wheez*).tw.; (3) respiratory syncytial viruses/ or respiratory syncytial virus, human/; (4) Respiratory Syncytial Virus Infections/; (5) (respiratory syncytial virus* or rsv).tw.; (6) parainfluenza virus 1, human/ or parainfluenza virus 3, human/; (7) Parainfluenza Virus 2, Human/; (8) Respirovirus Infections/; (9) Adenovirus Infections, Human/; (10) Rhinovirus/; (11) Influenza, Human/; (12) exp influenzavirus a/ or exp influenzavirus b/; (13) (parainfluenza* or respirovirus* or adenovirus* or rhinovirus* or influenza*).tw.; (14) or/1-13; (15) Saline Solution, Hypertonic/; (16) (hypertonic adj3 (saline or solution*)).tw.; (17) Sodium Chloride/; (18) (sodium chloride or saline).tw.; (19) or/15-18; (20) exp “Nebulizers and Vaporizers”/; (21) (nebuli* or vapor* or vapour* or atomi*).tw.; (22) Administration, Inhalation/; (23) inhal*.tw.; (24) Aerosols/; (25) aerosol*.tw.; (26) or/20-25; and (27) 14 and 19 and 26.
Data extracted included study year, setting, sample size, control group, and the mean (SD) LOS. If a study reported outcomes as median LOS, we abstracted a mean using publicly available software.8 Additionally, potential drivers of study heterogeneity were extracted, which included study-level factors (admission/discharge criteria, adjunct medications used, and treatment frequency), individual-level factors (day of illness [DOI] at study enrollment, severity of illness scores, and age), and treatment group–level factors (imbalance between treatment groups in individual patient factors). As different respiratory distress severity scores were used, scores were standardized prior to analysis.
We preplanned the following steps: (1) qualitative survey of the articles for potentially important differences in study design and associated sensitivity or subgroup analyses on those factors; (2) single-variable meta-regression against each of the following variables: control group LOS, use of adjunct medications, treatment frequency, DOI at study enrollment, clinical score, and mean age; and (3) multivariable meta-regression against variables found to be significant contributors to heterogeneity in the previous step. Meta-regression extends standard regression techniques to meta-analyses to examine heterogeneity and the association between potential moderator variables and treatment effect (similar to subgroup analysis) but allows the inclusion of continuous variables.
Weighted mean difference in LOS between treatment arms represented the summary statistic in our analyses and were calculated using the random effects model for meta-analysis. Study heterogeneity was assessed using the Cochran Q test (homogeneity assumption rejected if P < .10) and quantified using the I2 statistic. Consistent with guidance given by the Cochrane Collaboration, heterogeneity is interpreted as potentially unimportant (0%-40%), moderate (30%-60%), substantial (50%-90%), and considerable (75%-100%).9 Funnel plot symmetry was assessed visually and by the Egger test. Statistical analysis was performed using Stata version 14 (StataCorp).
Eighteen total studies were included in this reanalysis (Table).10-27 The studies included 2063 infants (63% male), with a mean age of 4.2 months. The mean LOS was 3.6 days. Initial review of studies for qualitative study-level differences revealed that the 2 studies performed in the same center in China had strikingly different criteria for hospital discharge—12 hours without respiratory symptoms—a far more stringent requirement than any other trial. Comparing the mean (SD) LOS for the control groups between these 2 studies (6.8 [1.4] days) and all others (2.8 [1.6] days) further confirms this population’s status as a statistical outlier for LOS. We verified that there were no appreciable differences in severity of illness in this population through a comparison of normalized severity scores at study entry by t test (P = .72). Additionally, it was noted that most (>70%) of the children enrolled in these studies were receiving systemic corticosteroids prior to admission, rates not seen in other settings. Sensitivity analysis revealed that removal of this study population improved heterogeneity to a level considered moderate to acceptable (I2 decreased from 78% to 45%) and rendered the summary weighted mean difference statistically insignificant (−0.21 days; 95% CI, −0.43 to +0.02); thus, the results are sensitive to inclusion/exclusion of this population (Figure 1).
Quantitative assessment of study-level factors that might account for heterogeneity included expected LOS (as defined by LOS in control groups), use of adjunct medications (eg, β-agonists), and frequency of HS treatments. Adjustment for these factors did not substantially resolve heterogeneity with 2 exceptions. First, as reported by Badgett et al,5 meta-regression against expected LOS was significant (coefficient, −0.25 days; 95% CI, −0.37 to −0.12; P = .001) but the significance was sensitive to the removal of the studies from China (−0.14 days; 95% CI, −0.32 to +0.04; P = .13). Second, epinephrine use appeared to potentiate the effect of HS (−0.61 days; 95% CI, −0.94 to −0.25; P = .003) (eTable in the Supplement).
Figure 2 presents a funnel plot, which further supports the identification of the Chinese studies as statistical outliers. Visual inspection reveals the presence of asymmetry with the absence of smaller negative studies and is confirmed by a positive Egger test result (all studies: P < .001; with the outlier population removed: P = .045). Such asymmetry may be owing to small study effects, publication bias, or true heterogeneity.
Assessment of individual patient–level factors that might account for heterogeneity focused on patient age, DOI at study entry, and clinical severity score. Adjustment for these factors did not substantially resolve heterogeneity (eTable in the Supplement).
Because many of the trials of HS were small, there remained the possibility that, despite randomization, important baseline differences between infants assigned to each treatment arm could be influencing results. We assessed the effect on heterogeneity of within-study differences between the treatment groups in mean age, DOI at study entry, and clinical severity score. There was a difference in DOI between treatment arms in studies reporting the information (N = 1661; summary difference, +0.32 days; 95% CI, −0.04 to +0.68) with HS treatment arm patients admitted later in illness. Later DOI was correlated with shorter LOS (coefficient, −0.54 days; 95% CI, −0.97 to −0.10; P = .02; eTable in the Supplement). Subgrouping by baseline imbalance in DOIs resolved the remaining heterogeneity (I2 = 0; Figure 3). Studies with balanced or unreported DOIs demonstrated no significant effect of treatment (+0.02 days; 95% CI, −0.14 to +0.17), while those with unbalanced treatment groups did show a benefit from HS (−0.67 days; 95% CI, −0.98 to −0.36). The apparent potentiating effect of epinephrine disappeared after baseline DOI imbalance was corrected (coefficient, −0.07 days; 95% CI, −0.64 to +0.50; P = .79).
As noted previously, Badgett et al5 identified a threshold response in which HS was effective when LOS in the control group exceeded 3 days. Our sensitivity analysis excluding the 2 outlier studies attenuates this threshold effect, decreasing the effect on LOS from −0.65 days (95% CI, −1.04 to −0.26) to −0.45 days (95% CI, −0.76 to −0.15) (Figure 4A). Adjusting for total duration of illness removed the appearance of a threshold effect (−0.11 days; 95% CI, −0.44 to 0.22; Figure 4B), further demonstrating that the apparent benefit of HS is sensitive to the bias introduced by unbalanced treatment arms.
Our reanalysis revealed significant concerns about the generalizability of the published summary statistics for this cohort of studies on HS in bronchiolitis. One study population was a significant outlier using both quantitative and qualitative assessments, with very different criteria for discharge and substantially longer expected LOS. The statistical significance of the weighted mean difference in LOS attributed to HS is sensitive to the removal of this study population. Furthermore, heterogeneity resolves to moderate to acceptable levels when this outlier population is removed, suggesting that removal improves the likelihood that the summary statistic reflects a generalizable treatment effect (or lack thereof in this case).
We also found baseline differences between treatment arms in DOIs at study enrollment resulting in the presence of systematic bias favoring the treatment group. Patients presenting later in their illness were more likely to be allocated to the HS treatment arm in 6 of the 18 studies including most of the small positive studies, making DOI imbalance a probable mechanism of bias representing small study effects. Presenting at a later DOI has biologic plausibility for an association with shorter LOS in this self-limited viral illness, hence the reason for collecting and reporting the information in most of the study tables of baseline characteristics.
A significant correlation between LOS/duration of therapy and HS efficacy was noted by Badgett et al,5 with the suggestion of a threshold response at longer than 3 days of treatment. Our analysis does not rule out the possibility of a threshold effect, although such an effect is substantially attenuated with the outlier population excluded. Moreover, the hypothesis that a threshold exists in this data set is sensitive to the correction of systematic bias in baseline characteristics in treatment allocation groups.
The outlying patient population and the baseline characteristic imbalance account thoroughly for the heterogeneity in this cohort of studies. While our intention in investigating heterogeneity in this study cohort was to identify which patient populations benefitted from HS and thereby to contribute to the design of future studies, exploring the heterogeneity ultimately resolved any appearance of a treatment effect. Furthermore, the studies representative of a US population (where LOS is typically <3 days)2 did not show any benefit from hypertonic saline (difference, −0.01 days; 95% CI, −0.22 to +0.20). We also note a trend of regression to the mean on the summary statistic as the newer US and European studies emerged (decrease in published summary statistic for reduction of LOS from −1.15 in the earliest meta-analysis4 to −0.44 days in the most recent).6 The addition of our finding of imbalanced treatment arm allocations on DOI in a significant proportion of studies, a factor clearly correlated with LOS, further supports a hypothesis that early small study effects were the major driver of the appearance of a treatment effect.
The chief limitation of our findings was related to the ecological fallacy, in that a meta-analysis lacks the granularity of individual patient–level data to investigate the possibility that other factors, such as patient age or adjunctive treatments, have a potentiating effect on any efficacy of HS. Further studies, or reanalysis of the data from existing studies, could help definitively answer this question, although our own assessment of the state of the evidence is that further clinical trials are likely unwarranted.
The appearance of a meaningful summary treatment effect on LOS in the cohort of studies on HS in acute viral bronchiolitis is a result of inappropriately combining studies with meaningful differences in outcome definitions and previously unnoticed systematic bias in treatment group allocation. Hypertonic saline cannot be expected to shorten LOS in bronchiolitis in typical US hospital settings.
Accepted for Publication: January 3, 2016.
Corresponding Author: Shawn L. Ralston, MD, MS, Children’s Hospital at Dartmouth, 1 Medical Dr, Lebanon, NH 03745 (firstname.lastname@example.org).
Published Online: April 18, 2016. doi:10.1001/jamapediatrics.2016.0079.
Author Contributions: Dr Brooks had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: All authors.
Acquisition, analysis, or interpretation of data: Brooks, Harrison.
Drafting of the manuscript: Brooks, Ralston.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: All authors.
Study supervision: Ralston.
Conflict of Interest Disclosures: None reported.