Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of Funding and Conclusions in Randomized Drug TrialsA Reflection of Treatment Effect or Adverse Events?. JAMA. 2003;290(7):921–928. doi:10.1001/jama.290.7.921
Author Affiliations: The Copenhagen Trial Unit, Center for Clinical Intervention Research, Copenhagen University Hospital, Copenhagen, Denmark.
Context Previous studies indicate that industry-sponsored trials tend to draw
Objective To explore whether the association between funding and conclusions in
randomized drug trials reflects treatment effects or adverse events.
Design Observational study of 370 randomized drug trials included in meta-analyses
from Cochrane reviews selected from the Cochrane Library, May 2001. From a
random sample of 167 Cochrane reviews, 25 contained eligible meta-analyses
(assessed a binary outcome; pooled at least 5 full-paper trials of which at
least 1 reported adequate and 1 reported inadequate allocation concealment).
The primary binary outcome from each meta-analysis was considered the primary
outcome for all trials included in each meta-analysis. The association between
funding and conclusions was analyzed by logistic regression with adjustment
for treatment effect, adverse events, and additional confounding factors (methodological
quality, control intervention, sample size, publication year, and place of
Main Outcome Measure Conclusions in trials, classified into whether the experimental drug
was recommended as the treatment of choice or not.
Results The experimental drug was recommended as treatment of choice in 16%
of trials funded by nonprofit organizations, 30% of trials not reporting funding,
35% of trials funded by both nonprofit and for-profit organizations, and 51%
of trials funded by for-profit organizations (P<.001; χ2 test). Logistic regression analyses indicated that funding, treatment
effect, and double blinding were the only significant predictors of conclusions.
Adjusted analyses showed that trials funded by for-profit organizations were
significantly more likely to recommend the experimental drug as treatment
of choice (odds ratio, 5.3; 95% confidence interval, 2.0-14.4) compared with
trials funded by nonprofit organizations. This association did not appear
to reflect treatment effect or adverse events.
Conclusions Conclusions in trials funded by for-profit organizations may be more
positive due to biased interpretation of trial results. Readers should carefully
evaluate whether conclusions in randomized trials are supported by data.
Empirical evidence shows that conclusions in randomized trials are more
positive toward experimental interventions if funded by for-profit organizations.1- 7 Three
studies found this association in randomized trials published in high-impact
journals.1- 3 Two
studies reached similar results in randomized trials on arthritis4 and myeloma.5 Two recent
systematic reviews6,7 highlight
the external validity of these findings. It is not known whether this association
reflects the quantitative trial results.8 More
positive conclusions in trials funded by for-profit organizations could reflect
either more beneficial treatment effects or less frequent occurrence of adverse
events. None of the previous studies1- 7 assessed
these aspects. Furthermore, previous studies included relatively heterogeneous
trial cohorts. This case mix could confound the findings. It is possible that
the association simply reflects that trials funded by for-profit organizations
assess the most effective interventions.
The influence of methodological quality, type of control intervention,
sample size, and disease area also could be important. These variables were
assessed only in 1 study but did not appear to explain the association between
funding and conclusions.3 Moreover, financial
interests may influence the decision to submit trials with positive results
to high-impact journals.
We assessed whether an association between funding and conclusions in
randomized drug trials reflects the magnitude of the treatment effect or occurrence
of adverse events. Secondary objectives were to explore the impact of methodological
quality, type of control intervention, size of the trial, year of publication,
or publication in high-impact journals on this association.
We selected all randomized trials included in eligible meta-analyses
from a random sample of Cochrane reviews obtained in May 2001 and extracted
data on conclusions, funding, treatment effect, adverse events, and additional
confounding factors. The primary outcome for each trial was selected according
to the meta-analysis including the trial. We analyzed whether an association
between funding and conclusions reflected treatment effects or adverse events.
We selected meta-analyses of randomized trials because within each meta-analysis,
trials assessed comparable treatments for specific diseases. This allowed
us to adjust for disease areas and type of experimental and control intervention.
Meta-analyses of binary outcomes were selected to adjust our analyses for
the treatment effect estimated by an odds ratio (OR). To reduce the number
of empty cells in our analyses, the meta-analyses had to include at least
5 full-paper trials. To examine the impact of trial quality, the meta-analyses
had to include at least 1 trial with and 1 trial without adequate allocation
concealment.9- 11 Our
inclusion criteria were adopted from previous methodological studies.9,11,12 We included all full-paper
trials that were contained in eligible meta-analyses.
Based on a pilot study of 100 systematic reviews, we estimated that
about 25% of all reviews published in the Cochrane Library would contain an
eligible meta-analysis. We also estimated that we had to include about 40
reviews to obtain a sample of about 500 trials. Previous studies1- 3,5 have
indicated that this number would provide an acceptable risk of type II error.
The number of systematic reviews in the Cochrane Library 2001, Issue 2 was
1081.13 To obtain about 40 reviews, we used
a computer-generated list of random numbers in blocks of 26 and randomly selected
4 trials from each block. This resulted in 167 systematic reviews, which were
screened by one author (B.A.-N.) for eligible meta-analyses.
Two of the authors (B.A.-N., L.L.K.) independently selected the primary
binary outcome for each included meta-analysis. The primary binary outcome
from each meta-analysis was considered the primary outcome for all trials
included in each meta-analysis. If it was not explicitly reported or several
primary outcomes were listed, we chose the most clinically relevant outcome,
assessed by the largest number of trials.
All data were extracted from the original trial reports. We defined
conclusions as the interpretation of the extent to which the overall trial
results favored the experimental over the control intervention. We graded
conclusions according to the phrasing in the abstract and the summarized conclusion
on a scale of 1 to 6 points (Box).14 The higher the score, the more
positive the conclusion toward the experimental intervention. Because the
score did not conform to a normal distribution, we divided conclusions into
whether the experimental drug was recommended as the treatment of choice without
disclaimers (6 points) or not (1-5 points).
Experimental intervention highly preferred and should now be considered
the standard intervention in all patients, or similar
Experimental intervention preferred to control, but further trials still
indicated, experimental may be more costly, or similar
Experimental and control intervention about equal, but the experimental
cheaper, easier to administer, or similar minor advantage
Experimental and control intervention about equal, but the control may
be cheaper, easier to administer, or similar minor advantage
Control intervention preferred to experimental intervention, but experimental
intervention might be promising under some circumstances, or similar
Control intervention highly preferred and should now be considered the
standard intervention in all patients, or similar
We extracted the sources of funding from the text, statements of sources
of support, authors' affiliations, and acknowledgments. Funding sources were
classified as nonprofit organizations, not reported, both nonprofit and for-profit
organizations, or for-profit organizations. For-profit organizations were
defined as companies that might acquire financial gain or loss depending on
the outcome of the trial. Funding included provision of grants, study materials
(drug, placebo, assay kits, or similar materials), or manpower (authorship,
statistical analysis, or other assistance). For the primary binary outcome
measure, we extracted the number of outcomes and participants in the experimental
and control groups.
We classified the occurrence of adverse events as (1) no significant
difference between experimental and control group, (2) significantly more
frequent in the control group, (3) significantly more frequent in the experimental
group, or (4) not reported.
We assessed methodological quality from the original trial reports and
any additional information provided in the Cochrane review. We assessed the
following 3 components10,11: generation
of the allocation sequence (classified as adequate if based on a table of
random numbers, computer-generated, or similar), allocation concealment (classified
as adequate if based on central randomization, identical coded drug boxes,
sealed envelopes, or similar), and double blinding (classified as adequate
if the trial was described as double blind). We extracted the type of control
intervention (placebo/no intervention or active intervention), the number
of patients randomized, and whether a preset sample size was estimated and
reached. We registered the meta-analysis in which the trial was included,
the year of publication, and whether the trial was published in a high-impact
journal (impact factor ≥6).1- 3
Two authors (B.A.-N., W.C.) independently extracted data from each trial
in an unblinded manner. Consensus was achieved before data entry. A third
author (C.G.) arbitrated disagreements. A fourth author (L.L.K.), who was
blinded with regard to funding, extracted conclusions in a random sample of
60 trials. The intraclass correlation coefficient between blinded and unblinded
assessment of conclusions was 0.93 (95% confidence interval [CI], 0.89-0.96).
For each trial, we estimated the OR of an unfavorable outcome (eg, mortality).
The SE of the logarithm of the OR was calculated as a measure of uncertainty.15 We calculated a z score
(log OR/SE to log OR) as a measure of treatment effect.15 The z score combines the magnitude of the point estimate (log
OR) with the level of uncertainty (SE).
We used the Kruskal-Wallis test for testing the overall null hypothesis
of no association between funding and conclusions assessed on the continuous
scale. We used a cutoff value between 5 and 6 points to divide trials into
whether the conclusions recommended the experimental drug as the treatment
of choice or not. We used logistic regression to assess the association between
funding and conclusions while adjusting for treatment effect, adverse events,
and other potentially confounding trial variables (methodological quality,
sample size, whether preset sample size was estimated and reached, meta-analysis,
year of publication, and journal impact factor). The logistic regression model
was fit using conclusions as the dependent variable and including trial variables
in a forward stepwise procedure. Meta-analysis was kept in the model irrespective
of statistical significance to adjust for the disease area and type of drug
and control intervention. All other variables were excluded if P>.05. The appropriateness of the logistic regression models was confirmed
by the Hosmer-Lemeshow test.16 All P values were 2-tailed and significance was defined as P<.05. Analyses were performed in SPSS version 11.0 for Windows
(SPSS Inc, Chicago, Ill).
From our random sample of 167 Cochrane reviews, we excluded 126 that
included fewer than 5 full-paper randomized trials in a meta-analysis (n =
105), included only trials with adequate (n = 6) or inadequate allocation
concealment (n = 13), or did not assess a binary outcome (n = 2). The remaining
41 reviews contained meta-analyses, which included 523 trials, that met our
inclusion criteria. Sixteen of these reviews, which included 153 trials, assessed
nonpharmacological interventions. Initial analyses revealed that only 4 (3%)
of these trials were funded by for-profit organizations. We therefore limited
our analyses to the 370 drug trials (references available on request from
the authors) from 25 reviews.17- 41
In most trials, conclusions favored the experimental drug (median score
[interquartile range], 5 [4-6]). In 36% of trials (n = 135), the experimental
drug was recommended as the treatment of choice (6 points).
Eighteen percent of trials (n = 67) were funded by nonprofit organizations
and in 29% (n = 106) funding was not reported. Fourteen percent of trials
(n = 51) were funded by both nonprofit and for-profit organizations and 39%
(n = 146) by for-profit organizations alone. The treatment effect assessed
by the mean (SD) z score was –1.39 (1.90) (range,
–10.99 to 5.45). In 50% of the trials (n = 185), the occurrence of adverse
events did not differ significantly between the intervention groups, and in
5% (n = 20) the occurrence was significantly higher in the control than that
in the experimental group. In 16% of trials (n = 60), the occurrence of adverse
events was significantly higher in the experimental group, and in 28% (n =
105) adverse events were not reported.
Adequate generation of the allocation sequence was reported in 28% of
trials (n = 105), adequate allocation concealment in 22% (n = 82), and 63%
(n = 234) were double blind. In 76% of trials (n = 283), the control intervention
was placebo or no intervention. The median number of patients randomized was
98 (range, 10-82 892). Preset sample size was estimated and reached in
21% of trials (n = 76).
The disease areas were intensive care (n = 85), smoking cessation (n
= 78), respiratory diseases (n = 54), gynecology/obstetrics (n = 48), gastroenterology
(n = 33), neurology (n = 26), psychiatry (n = 13), infectious diseases (n
= 12), rheumatology (n = 9), nephrology (n = 6), and dermatology (n = 6).
The primary outcome measures were smoking cessation (n = 78), mortality (n
= 64), blood transfusion (n = 61), withdrawals (n = 33), dysphagia (n = 23),
endometritis (n = 20), parasitemia (n = 17), depression (n = 13), admission
to hospital (n = 11), bronchiolitis (n = 8), neurologic deficit (n = 8), cesarean
delivery (n = 7), warts (n = 6), cytomegalovirus disease (n = 6), pregnancy
(n = 5), bacterial vaginosis (n = 5), and asthma (n = 5).
The year of publication ranged from 1971 to 2000 with 1990 (5.6 years)
as the mean (SD) publication year. Eighteen percent of trials (n = 65) were
published in high-impact journals.
The conclusions in trials stratified by funding are shown in Table 1. Conclusions were significantly
more favorable toward experimental drugs in trials funded by for-profit organizations
compared with those of trials funded by other sources (P<.001, Kruskal-Wallis test). The proportion of trials in which
conclusions recommended the experimental drug as the treatment of choice (6
points) was significantly higher among trials with for-profit funding compared
with that of other trials (P<.001, χ2 test).
The distributions of the potential confounding variables stratified
by funding are shown in Table 2.
Funding by for-profit organizations alone or by for-profit and nonprofit organizations
was associated with more complete reporting of adverse events, more adverse
events in the experimental group, more frequent report of adequate allocation
concealment and double blinding, and more frequent use of placebo or no treatment
as control intervention. Funding by both nonprofit and for-profit organizations
also was associated with a larger sample size and publication in high-impact
journals. No significant difference was observed between the groups regarding
the treatment effect, adequate generation of the allocation sequence, or whether
a preset sample size had been estimated and reached.
The logistic regression analyses showed that funding, treatment effect,
and double blinding were significantly associated with conclusions (Table 3). None of the remaining variables
were associated significantly with conclusions. After adjusting for the treatment
effect and double blinding, conclusions were significantly more likely to
recommend the experimental drug as treatment of choice in trials funded by
for-profit organizations alone compared with trials funded by nonprofit organizations
(OR, 5.3; 95% CI, 2.0-14.4). Compared with trials funded by nonprofit organizations,
conclusions were not significantly different in trials not reporting funding
or trials funded by both nonprofit and for-profit organizations. The likelihood
of recommending the experimental drug as the treatment of choice decreased
with decreasing treatment effect (OR, 0.6; 95% CI, 0.5-0.7) and increased
with adequate double blinding (OR, 2.9; 95% CI 1.4-6.0) (Table 3).
In randomized drug trials from a randomly selected sample of reviews
published in the Cochrane Library,13 we found
that conclusions of trials were significantly more likely to recommend the
experimental drug as the treatment of choice if trials were funded by for-profit
organizations. This result is in accordance with previous studies.1- 7 The
present study adds to previous evidence by showing that this association does
not reflect the quantitative trial results; neither the magnitude of the treatment
effect nor the occurrence of adverse events could explain the association.
We selected meta-analyses from a random sample of Cochrane reviews because
the Cochrane Collaboration is a nonprofit organization that aims to minimize
the influence of financial and other competing interests. Furthermore, Cochrane
reviews appear to be of higher quality and are less prone to bias than reviews
published in traditional medical journals.42- 44 We
excluded a high (75%) but expected number of the randomly selected reviews
because they did not contain a meta-analysis that fulfilled our inclusion
criteria. Furthermore, only 3% of the nonpharmacological trials in our original
sample were funded by for-profit organizations. Because of the low frequency
of for-profit funding, these trials would not contribute valuable information,
but rather introduce noise into the analyses.45 Therefore,
these trials also were excluded. Considering that more than 360 000 randomized
clinical trials have been published,46 our
results are based on a small sample. These factors may reduce the external
validity of our study. However, our sample was obtained randomly from the
Cochrane Library, which covers all medical areas.13 The
included trials covered a variety of medical areas, drugs, publication years,
and journals. This variability supports the external validity of our findings.
Within each meta-analysis, trials addressed the same clinical question,
compared the same treatments, and provided data on the same outcome measure.
We therefore were able to adjust our analyses for both disease areas and type
of experimental and control treatments. Therefore, the clinical importance
of the treatment corresponds to the magnitude of the treatment effect. Together
the point estimate (OR) and the level of uncertainty (SE) provide information
about the treatment effect. We used a z score to
capture these 2 aspects in 1 variable. We found similar results when including
the OR and the SE of the treatment effect separately in the logistic regression
model (data not shown). The magnitude of the treatment effect did not explain
the association between funding and conclusions.
To estimate the treatment effect, we selected the primary binary outcome
specified in the meta-analysis including the trial. We did this to obtain
clinically important and homogeneous data. Many randomized trial reports do
not specify the primary outcome measures,47 use
surrogate outcomes,48- 51 or
use multiple primary outcomes.52 Cochrane reviews
are based on prespecified, peer-reviewed, and published protocols in which
the outcome measures most clinically relevant to patients are selected.53 Accordingly, the selection of the primary outcome
measure in Cochrane reviews aims to be unbiased and not data driven.52
The favoring of experimental drugs in trials funded by for-profit organizations
did not appear to reflect the occurrence of adverse events. Compared with
trials funded by nonprofit organizations, trials funded by for-profit organizations
reported a significantly higher number of adverse events in the experimental
arm. This might reflect differences in the quality of reporting. Possibly,
drug companies are more focused on reporting of adverse events because of
use of Good Clinical Practice guidelines.54
We assessed the effect of several potential confounders. These were
selected according to previous evidence and theoretical considerations. Several
of the confounders had some overlap (eg, the treatment effect and the use
of inactive control, double blinding and the use of placebo as control intervention).
However, this overlap only increased the SEs of the logistic regression model
but did not invalidate the model. Our results indicated that the methodological
quality, type of control intervention, sample size, whether a preset sample
size had been estimated and reached, year of publication, or publication in
high-impact journals did not explain the association between funding and conclusions.
Trials funded by for-profit organizations had better methodological
quality than trials funded by nonprofit organizations regarding allocation
concealment and double blinding. This finding is in accordance with previous
are aware that discrepancies might occur between report and conduct of trials.56 The Cochrane Collaboration recommends that reviewers
correspond with the primary investigators of trials and companies to obtain
information about central methodological aspects.53 We
based our assessment of quality on both the trial reports and any additional
information provided in the Cochrane reviews.17- 41
We assessed the reported conclusions using a scale developed by Gilbert
et al.14 The lack of validation of this scale
may be criticized.20 However, the scale has
been used in 3 studies,3,5,57 and
we found that it had high face and content validity as well as high reliability.
One study analyzed the reported conclusions on the continuous scale.3 Other studies divided the reported conclusions into
"positive" or "negative."1,2,4,5 In
the present study, only 20% of the trials scored 1 to 3 points. This skewed
distribution of data supported the use of a cutoff value between 5 and 6 points.
This allowed us to explore whether conclusions were more likely to recommend
the experimental drug as the treatment of choice without disclaimers. Such
conclusions must have a considerable impact on clinical decision making. Sensitivity
analyses revealed that selecting another cutoff (eg, between 4 and 5 points)
gave similar results. This increases the robustness of the evidence and supports
the existence of an association between funding and conclusions.1- 7
A potential weakness of the present study is that the reported conclusions
were assessed unblinded with regard to the source of funding. However, we
found high interobserver agreement between blinded and unblinded assessment.
This concurs with previous findings3,58 and
suggests that blinding would not significantly change our conclusions.
Sponsor involvement and influence on the conduct and reporting of a
trial varies. The degree of influence is difficult to assess from trial reports.
We combined trials funded entirely by for-profit organizations with trials
in which only drugs and placebo were provided. Sensitivity analyses showed
that the significant association between funding and conclusions was present
both in the group of trials having only obtained for-profit funding in the
form of drug and placebo and in the group of trials having obtained more substantial
for-profit funding (data not shown).
A large proportion of trials did not report the sources of funding.
It is likely that funding by for-profit organizations is underreported in
trials. However, the degree of underreporting is not possible to assess. We
found that about 50% of the included trials were funded solely or in part
by for-profit organizations. This is in accordance with a recent review, in
which the median proportion of trials receiving for-profit funding was 39%
(interquartile range, 23%-64%).7
Conclusions reflect a trade-off between efficacy, safety, and cost-effectiveness.
We did not assess the impact of cost-effectiveness on conclusions. Economic
analyses are rarely included in randomized trials59,60 and
new interventions are generally more expensive than conventional ones. Friedberg
et al61 showed that studies in oncology funded
by for-profit organizations were nearly 8 times less likely to reach unfavorable
conclusions regarding economic assessments of experimental interventions than
studies funded by nonprofit organizations.
We based our analyses on all full-paper trials that were included in
the eligible meta-analyses. Trials that were only published as abstracts or
letters were excluded as they very rarely contained information on funding
and/or trial quality. We did not assess the studies that the authors of the
Cochrane reviews had excluded from meta-analyses. The reasons for exclusion
were described in all reviews. The majority of studies were excluded because
they were not randomized trials. It is possible that these studies estimated
a greater treatment effect because lack of randomization increases the risk
of selection bias. However, it is uncertain whether the design of a study
(randomized or nonrandomized) has a significant effect on the association
between funding and conclusions. Our study was not designed to address this
The present study cannot show the causes or consequences of the association
between for-profit funding and conclusions favoring the experimental drug.
Our study was designed to assess if an association reflected the quantitative
trial results. We found a significant association between conclusions and
the estimated treatment effect. The likelihood of recommending the experimental
drug as the treatment of choice increased significantly with increasing treatment
effect (ie, decreasing z scores). However, this did
not appear to explain the association between funding and conclusions. We
found no significant difference in treatment effect between trials stratified
according to funding. This is in accordance with a recent pilot study.62 Our findings oppose the suggestion that conclusions
are more likely to be positive if funded by for-profit organizations because
these trials should be more likely to reach positive results.8
Likely explanations for the association could be violation of the uncertainty
principle, publication bias, emphasis on subgroup or secondary outcome analyses,
or bias in drawing conclusions. Violation of the uncertainty principle could
occur if for-profit organizations were more prone to sponsor trials that were
likely to favor the experimental drug.5 A main
objective of the pharmaceutical industry is to acquire financial gain. After
having conducted exploratory randomized trials in phase 2 drug development,
several confirmatory randomized phase 3 (proof of concept) trials usually
are launched. Such trials may have a higher likelihood of favoring the experimental
drug. Publication bias also has been suggested as a possible explanation.1- 3,5,7,63 Concern
has been raised that for-profit organizations might discontinue ongoing studies
if accumulating results appear negative or if they avoid the publication of
negative studies.1,63,64 Because
of the design of the present study, we cannot refute or confirm these queries.
We found no significant difference in the estimated intervention effect of
the primary outcome measure between groups of trials stratified according
to funding. It is possible, that the favoring of experimental interventions
in conclusions of trials funded by for-profit organizations was due to emphasis
of results from surrogate outcomes,49 secondary
outcome measure analyses, or subgroup analyses.65,66 Finally,
the association between funding and conclusions might reflect a biased interpretation
of the overall trial results. This potential bias could be due to financial
conflicts of interest.67
In principle, about half of all trials should favor the control intervention.5 We found that trial results and conclusions rarely
favored the control. This finding concurs with previous studies.1- 4,6,57,62,64,68 The
combined evidence underlines the need for an international register of all
initiated randomized clinical trials. Registration would enable the public
to follow the development of drugs from the beginning of phase 2 trials. Furthermore,
data from unpublished randomized trials irrespective of funding source or
results could be contained in an international register available to the public.69,70 The Consolidated Standards of Reporting
Trials (CONSORT) statement71 could consider
the importance of reporting of funding, as suggested by the International
Committee of Medical Journal Editors.72 Our
study suggests that editors, peer reviewers, and readers of trial reports
should evaluate carefully the trial data to determine if the reported conclusions
are supported by data.