van Nieuwenhoven CA, Buskens E, van Tiel FH, Bonten MJM. Relationship Between Methodological Trial Quality and the Effects of Selective Digestive Decontamination on Pneumonia and Mortality in Critically Ill Patients. JAMA. 2001;286(3):335-340. doi:10.1001/jama.286.3.335
Author Affiliations: Department of Medical Microbiology, University Hospital Maastricht (Drs van Nieuwenhoven and van Tiel), Julius Center for General Practice and Patient Oriented Research (Dr Buskens) and Department of Internal Medicine, Division of Infectious Diseases and AIDS (Dr Bonten), University Hospital Heidelberglaan, Utrecht, the Netherlands.
Caring for the Critically Ill Patient Section Editor: Deborah J. Cook, MD, Consulting Editor, JAMA.
Context Although meta-analyses of randomized trials have shown that selective
digestive decontamination (SDD) prevents nosocomial pneumonia in critically
ill patients, the influence of trial quality on the effectiveness of SDD has
not been rigorously evaluated.
Objective To assess the methodological quality of individual studies of SDD and
its relation to the reported effects on pneumonia and mortality.
Design Thirty-two studies were identified in a MEDLINE and reference list search
and their methodological quality was assessed using a scoring system (range,
0-13 points) based on allocation and concealment, patient selection, patient
characteristics, blinding of the intervention, and the definition of pneumonia.
Main Outcome Measure Methodological quality of the primary trials and its effect on the relative
risk reductions (RRRs) of SDD on pneumonia and mortality.
Results The mean (SD) methodological quality score was 7.8 (2.9) (range, 1-11).
The RRRs ranged from –0.1 to 1.0 for pneumonia and from –0.1 to
0.6 for mortality. The methodological quality score was associated with the
RRR for pneumonia so that for each quality-point added, the RRR decreased
by 5.8% (95% confidence interval, 2.4%-9.3%). No association between trial
quality and the impact of SDD was found on mortality. Of the individual trial
quality characteristics, patient selection, allocation of intervention, and
blinding most strongly influenced the treatment effect.
Conclusions The inverse relationship between methodological quality score and the
benefit of SDD on the incidence of pneumonia may have resulted in overly optimistic
estimates of SDD in prior meta-analyses. This emphasizes the importance of
rigorous trial design in evaluating preventive interventions in the intensive
Nosocomial pneumonia is the most frequent infection among mechanically
ventilated patients treated in intensive care units (ICUs) and has been associated
with increased morbidity, antibiotic use, and prolonged length of stay. Prevention
of nosocomial pneumonia, therefore, remains a challenge for intensive care
Selective decontamination of the digestive tract (SDD) is the most extensively
studied method of infection prevention for critically ill patients. The SDD
method consists of eradicating potentially pathogenic microorganisms in the
oral cavity by application of nonabsorbable antibiotic paste and decontamination
of the rest of the gastrointestinal tract by local administration of the same
antibiotics, conducting a short course of systemic prophylaxis for respiratory
tract infections that may occur during the first days of intubation, regulary
taking cultures of throat swabs and feces to monitor the effectiveness of
SDD, and maintaining optimal hygiene to prevent cross-infection. More than
30 studies and 6 meta-analyses2- 7
have been published. Although all meta-analyses showed significant reductions
in the incidence of pneumonia among patients receiving SDD, the effects on
mortality are less clear. A recent meta-analysis2
suggested an overall 20% reduction in ICU mortality. In another meta-analysis,3 a significant 40% reduction in ICU-mortality was found
among surgical patients, but no significant difference was found for medical
patients.3 Interestingly, reductions in mortality
were found only for patients who had received systemic and topical antibiotic
prophylaxis.2,3 Although some
have argued that SDD is now an evidence-based intervention with impressive
positive results for patient care,8 others
question the value of meta-analyses as a method for hypothesis testing.9- 11 However, the quantity
of studies does not necessarily reflect quality of studies. An inadequate
approach to controlled trial design and execution may be associated with bias
in estimating treatment effects, and incorporating such studies in meta-analyses
may bias the effect of interventions.12,13
In our study, we analyzed the relationship between the methodological quality
of individual SDD studies and the reported effects on pneumonia and mortality.
We searched MEDLINE and reference lists of previous meta-analyses for
relevant trials on SDD. We searched for randomized trials using the initialism SDD. Studies that were only published as abstracts were
excluded because data were insufficient for a thorough quality assessment.
Criteria to assess the methodological quality of individual trials were partly
derived from a previously published scoring method for prevention of nosocomial
pneumonia, which includes aspects of treatment allocation and concealment,
patient selection, comparability of patient groups, blinding, and diagnosis
Zero, 1, or 2 points were given for each of 5 criteria. We slightly
modified scoring criteria for diagnosing pneumonia by adding a point when
pneumonia was diagnosed with quantitative cultures of endotracheal aspirates
or samples obtained via bronchoscopic techniques. Several studies have demonstrated
that adding these techniques to standard diagnostic criteria improves the
specificity for diagnosing pneumonia, and their use has, therefore, been advised.15 Furthermore, randomization was separated into the
method of allocation and the method of concealment (ie, how the assignment
sequence is blinded and therefore protected against manipulation). As a result,
the maximum score for study quality was 13 (Table 1). When analyzing the relationship between mortality and
study quality, the category of diagnosing pneumonia was not included. Studies
were considered high-quality studies when the total score was 7 points or
more. Low-quality studies scored fewer than 7 points. A cutoff of 5 was used
for this analysis on patient mortality (without points for diagnosing pneumonia).
Since the quality scoring system gives equal weight to each characteristic
summed in a total score, an additional analysis of individual quality characteristics
was performed. For this analysis, components of trial quality were dichotomized:
high quality for any characteristic was defined as the highest possible score
of 2, and scores of 0 or 1 were considered low quality.
Three researchers (C.A.v.N., F.H.v.T., and M.J.M.B) independently reviewed
all studies. In cases of disagreement about any criteria, agreement was reached
by consensus. The relative risk reductions (RRRs) with 95% confidence intervals
(CIs) were calculated for pneumonia and mortality for each of the trials.16 Linear regression analysis was performed with methodological
quality score (Table 1) as the
independent variable; pneumonia and mortality were the dependent variables.
Since SDD has been reported to be more efficacious in surgical than in medical
patients,3 relationships between quality scores
and proportion of surgical patients in each study were also analyzed using
linear regression. Subsequently, we entered dichotomized characteristics in
the univariable analysis to asses the influence of individual characteristics
on outcomes. Characteristics with significant associations were then included
in multivariate linear regression analysis. A P value
of less than .05 was considered statistically significant.
Thirty-two trials were included (Table 2). In 1 study,31 2 SDD-schedules
were compared with a single control group, and both analyses have been included
separately. In another study in which SDD was compared with a simultaneously
studied as well as a historical control group, only the comparison with the
simultaneously studied control group was included.41
The mean (SD) methodological quality score was 7.8 (2.9) (range, 1-11).
Reviewers reached agreement in 61% of the studies. They differed by 1 point
in 31% of the studies. In the remaining studies, scores from 1 reviewer differed
by 2 points (5%), or all 3 reviewers arrived at different scores (3%).
Relative risk reductions for pneumonia ranged from –0.1 (incidence
higher in patients receiving SDD) to 1 (complete prevention of pneumonia).
Significant reductions in the incidence of pneumonia were reported in 21 studies.
When comparing all patients receiving SDD (n = 2400; incidence, 12%) with
control patients (n = 2404; incidence, 29%), the RRR for pneumonia was 0.57
(95% CI, 0.49-0.65). Relative risk reductions for the 12 studies with high
quality scores were 0.68 (95% CI, 0.57-0.95), and for the 18 studies with
low quality scores 0.47 (95% CI, 0.36-0.58).
Relative risk reductions for mortality ranged from –0.15 to 0.59,
but a significant decrease in mortality was not found in any individual trial.
When all the studies were grouped, mortality rates were 25% for SDD patients
and 28% for controls, resulting in an RRR of 0.12 (95% CI, 0.03-0.21). Relative
risk reductions for mortality were − 0.14 (95% CI, − 1.14 to 0.87)
for studies with low quality scores and 0.14 (95% CI, 0.03-0.25) for studies
with high quality scores. Relative risk reduction for mortality was 0.18 (95%
CI, 0.04-0.32) when only studies using systemic prophylaxis as part of SDD
with quality scores of 5 or more were analyzed.
A significant association was found for methodological quality scores
and RRRs for pneumonia (Figure 1).
For each point added to the methodological quality score, the RRR decreased
by 5.8% (95% CI, 2.4% to 9.3%). Interestingly, the variance in RRRs was largest
for studies with high quality scores. No such association was found for RRRs
of mortality. The calculated reduction in mortality per point reduction of
trial quality was –1.7% (95% CI, –3.7% to 0.03%). When studies
with and without systemic prophylaxis were analyzed separately, results remained
unchanged (data not shown).
Of the trial quality features we considered, patient selection, allocation
of the intervention and blinding were significantly associated with the RRRs
of pneumonia. Blinding remained significantly associated in multivariate analysis
(Table 3). For mortality, none
of the trial quality characteristics were significantly associated with outcome.
Our analysis demonstrated an inverse relationship between the methodological
quality of SDD studies and the observed effects on the incidence of nosocomial
pneumonia: the higher the quality score, the smaller the RRR of pneumonia.
No such association was found for mortality. The trial quality characteristics
associated with the outcome of pneumonia were patient selection, allocation
of the intervention, and blinding. When creating new methodological quality
scores, or adapting currently existing scores for meta-analysis, the relative
contributions of each characteristic should be taken into account.
Studies on the prevention of nosocomial pneumonia in ICU are methodologically
challenging. Most importantly, to date, no criterion standard test exists
for the primary end point: nosocomial pneumonia. Therefore, vigorous attempts
to minimize bias through other means is required. For example, the many cointerventions
that may influence the development of pneumonia should be described if not
controlled to aid in trial interpretation; such cointerventions were rarely
reported in this literature. Attempts to minimize bias through duplicate blinded
outcome assessment would be desirable.
Large numbers of patients from a homogenous patient population should
be included in prospective, randomized, placebo-controlled trials. Blinding
of whom for what purpose should be reported. Most of these trials included
a small number of patients leading to the possibility of type I and type II
errors, which can be magnified in meta-analyses. Our finding of an inverse
relationship between study quality and reported benefits on the incidence
of pneumonia in patients receiving SDD strongly suggests that the quality
of study design and execution has influenced reported benefits of SDD. The
fact that study quality influenced the effectiveness of pneumonia prevention
is supported by our findings that study quality did not influence the reported
effects on mortality. Interestingly, we found a small but significant reduction
in mortality when analyzing studies of high quality. No reduction in mortality
was found in low-quality studies although the reduction in incidences of pneumonia
was 68%. If pneumonia and ICU mortality are associated, similar trends between
both end points and study quality were to be expected. The absence of this
association could be explained by inaccurate estimates of the incidence of
pneumonia, particularly in unblinded trials.
In addition to possible bias in the trials on the prevention of nosocomial
pneumonia and difficulties in interpretation of meta-analyses, the use of
a single quality score may also influence study outcome.19
Using 25 different quality scales, Juni et al48
demonstrated that different quality scoring scales may lead to different conclusions
from meta-analyses. They concluded that relevant methodological aspects should
be assessed separately and that their influences on effect sizes should be
explored.48 We screened most of the quality
scoring scales used by Juni et al,48 and almost
all included variables such as patient selection, randomization, and blinding
procedures but not specific criteria for diagnosing pneumonia. Therefore,
we decided to use the scoring system designed for this patient population
and this specific infection.14 When assessing
the importance of different methodological variables separately, we found
differences in their relative importance. This suggests that by simply adding
up scores based on perceived relevance may not be appropriate and that informed
weighting of quality scores according to methodological importance may be
An association between study quality on the benefits of an intervention
summarized in a meta-analysis, as reported herein, confirms the results of
previous analyses in other research areas. Analyzing 250 controlled trials
from 33 meta-analyses, Schulz et al12 determined
that the exaggerated estimates of treatment effects were 30% to 41% for trials
in which concealment was either unclear or inadequate and 17% for trials that
were not double-blind. In addition, Moher et al13
reanalyzed 127 randomized controlled trials included in 11 meta-analyses and
quantitatively determined study quality. Compared with high-quality trials,
low-quality trials were associated with an increased estimate of benefit of
34%, and trials that used inadequate allocation concealment were associated
with an increased estimate of benefit of 37% compared with trials using adequate
The routine use of SDD as a measure of infection prevention in the ICU
remains controversial. Whether SDD significantly reduces ICU mortality would
be most convincing if it were confirmed in a large, prospective, randomized,
double-blind study. Moreover, beneficial effects on secondary end points,
such as duration of mechanical ventilation and ICU stay, and amount of antibiotic
use remain unproven, whereas selection of antibiotic-resistant pathogens remains
a cause for concern and must be determined from longitudinal studies or modelled
in a cost-effectiveness analysis. Unfortunately, the short time horizon of
these SDD trials means we have insufficient information about long-term microbial
consequences of SDD. Our findings have implications in practice and research.
We have shown the importance of considering randomization trial quality in
making appropriate inferences about using SDD for pneumonia prevention in
the ICU. The results for this methodological analysis emphasize the need for
rigorous methods to avoid biased randomized trials and also biased meta-analyses
of these trials.