Background
The evaluation of research output, such as estimation of the proportion of treatment successes, is of ethical, scientific, and public importance but has rarely been evaluated systematically. We assessed how often experimental cancer treatments that undergo testing in randomized clinical trials (RCTs) result in discovery of successful new interventions.
Methods
We extracted data from all completed (published and unpublished) phase 3 RCTs conducted by the National Cancer Institute cooperative groups since their inception in 1955. Therapeutic successes were determined by (1) assessing the proportion of statistically significant trials favoring new or standard treatments, (2) determining the proportion of the trials in which new treatments were considered superior to standard treatments according to the original researchers, and (3) quantitatively synthesizing data for main clinical outcomes (overall and event-free survival).
Results
Data from 624 trials (781 randomized comparisons) involving 216 451 patients were analyzed. In all, 30% of trials had statistically significant results, of which new interventions were superior to established treatments in 80% of trials. The original researchers judged that the risk-benefit profile favored new treatments in 41% of comparisons (316 of 766). Hazard ratios for overall and event-free survival, available for 614 comparisons, were 0.95 (99% confidence interval [CI], 0.93-0.98) and 0.90 (99% CI, 0.87- 0.93), respectively, slightly favoring new treatments. Breakthrough interventions were discovered in 15% of trials.
Conclusions
Approximately 25% to 50% of new cancer treatments that reach the stage of assessment in RCTs will prove successful. The pattern of successes has become more stable over time. The results are consistent with the hypothesis that the ethical principle of equipoise defines limits of discoverability in clinical research and ultimately drives therapeutic advances in clinical medicine.
Although cancer remains the second leading cause of deaths in the United States,^{1} there have been continuous improvements in survival and other outcomes in patients with cancer over time.^{1} To a large extent, this has occurred through the introduction of new treatments tested in clinical trials,^{2} with randomized controlled trials (RCTs) widely considered to be the most reliable method of assessing differences between the effects of health care interventions.^{3}^{,4} Cancer is the only disease for which the National Institutes of Health has consistently funded a cooperative clinical trial infrastructure.^{4} Despite this investment,^{5} little is known about the proportion of clinical trials that have led to the discovery of successful new treatments.^{6}
There are ethical, public policy, and scientific reasons for evaluating the success of new treatments in RCTs. New discoveries cannot be made without the willingness of patients to enroll in clinical trials.^{7} Because they are being asked to volunteer for these trials, it has been suggested that patients be given all relevant details about the situation in which they find themselves, including the track record of experimental treatments studied in earlier trials.^{6} Similarly, researchers, policymakers, funding agencies, and the public share an interest in knowing what payback has been achieved from the research that society supports.^{8} Herein we address this question by systematically evaluating the treatment successes in RCTs conducted by cooperative oncology groups (COGs) under the aegis of the National Cancer Institute (NCI). We assessed how often new treatments for cancer evaluated in phase 3 RCTs are superior to standard treatments in a cohort of consecutive trials conducted by a common funder (NCI), which has the same framework for the development of preventive and therapeutic advances in oncology.
We evaluated a consecutive series of all phase 3 RCTs completed between 1955 and 2000 under the aegis of the 8 NCI-sponsored COGs. Since it typically takes several years to complete follow-up and publish an RCT,^{9} we included only completed trials up to the year 2000 that had been published by December 2006.
We obtained a list of trials from the NCI that was verified by the headquarters of the relevant COG. Each COG provided a copy of the research protocol(s) for each study. We analyzed data from published and unpublished trials. December 31, 2006, was used as the cutoff date for determination of each RCT publication status. If the trial had not been published by this date, it was classified as unpublished. For 1 COG (Cancer and Leukemia Group B), data were available for the trials that began in 1980 and later. This study was approved by the institutional review board (No. 100449) of the University of South Florida.
EVALUATION OF TREATMENT SUCCESSES: OUTCOME MEASURES
For each trial we identified the new/experimental and the standard treatment by means of information provided in the background section of each article and/or the related research protocol. Trials that compared 2 experimental arms against each other were excluded. Similarly, our protocol specified exclusion of trials that had been designed to assess equivalence rather than superiority.
We determined the superiority of experimental or standard treatment by using 3 outcome measures to capture all the aspects of treatment successes^{10}^{,11}: (1) the proportion of the trials that were statistically significant according to the primary outcomes specified in the null hypothesis; (2) the proportion of trials for which the original researchers claimed that the new or the standard treatment was better; and (3) pooled outcome data from the trials obtained by quantitative meta-analytic techniques.^{12}
The second outcome measure was necessary because a simple determination of the proportion of trials that achieved statistical significance on their primary outcome does not capture subtleties involved in identifying advances in treatments or the trade-off between benefits and harms of competing interventions. For example, treatments with shorter duration or a convenient method of application can be considered successful even if no statistically significant findings are identified. To deduce investigators' judgments about treatment success, we used the method that, in previous studies, was found to have high face and content validity and high reliability^{10}^{,11,13}^{,14} and therefore likely reflected the accurate judgments of the investigators about the merits of the treatments.
The third outcome measure, pooled outcome data quantitatively, is important because the above mentioned “vote counting” methods^{15} are based on assessment of a proportion of successes or failures and do not take into account effect sizes in individual trials, the number of patients, or time to event data. Consequently, we synthesized data on the most important events (deaths, disease progression, or relapses) to explore the distribution of the outcomes between experimental and standard treatments.^{12} Summary effects were expressed as hazard ratios (HRs)^{12} with 99% confidence intervals (CIs). A random-effects model was used.^{12} Odds ratios (ORs) were used to summarize data on response rate and treatment-related mortality. The unit of analysis was the comparison within each trial. For trials/reports that included more than 1 experimental treatment group, we pooled the data from all such groups if the results suggested that these interventions were not better than the standard treatment. We also dealt with trials in which more than 1 new treatment was compared with standard treatment by repeating the analysis using 1 of the new intervention groups only, selecting it at random, and by splitting the control group into the relevant number of subsets and using each of these for the statistical comparison with each new treatment. Our results were not different in any important way, regardless of which analytic method was used; therefore, we report only the main analyses.
To assess the efficiency of the clinical trial system to successfully address scientific hypotheses that led to initiation of the trials, we also determined a proportion of “conclusive” and “inconclusive” trials. We operationally defined inconclusive trials as those in which the 95% CI for the treatment effect included an HR and OR of 0.8 and 1.2, respectively, as a surrogate for a clinically important effect.
We determined the proportion of discoveries that were “breakthrough interventions.” This was arbitrarily defined as interventions judged by the original researchers to be so beneficial that they should immediately become the new standard of care or that had an effect size so large that they reduced the death rate by 50% or more (ie, the HR for death was 0.5 or less).
Two investigators (A.K. and H.P.S.) extracted the data, and the principal investigator (B.D.) checked 1 in 4 trials at random. Consensus meetings were held to resolve any interobserver differences. Interobserver agreements for quality appraisal and for assessment of treatment success were high (0.90-0.97).
FACTORS OTHER THAN TREATMENT THAT MAY AFFECT OUTCOMES
Factors other than the true differences between the effects of treatments may affect the results of clinical trials. Research during the past decade has identified publication bias,^{16} methodologic quality,^{17} and the choice of control intervention^{18} as key factors affecting trials' results.
We used the NCI definition of completed studies^{19} to determine the publication rate. If a study had more than 1 publication, we extracted data from the most recent report. Studies that were initiated but were closed early because of poor patient accrual or that had not yet completed follow-up were excluded. Trials that were stopped early because the results clearly favored one treatment over another were included in our analysis.
We extracted data on the methodologic domains relevant to minimizing bias and random error in the conduct and analysis of the trials.^{17}^{,20} To ensure the accuracy of quality assessment of the trials, we used both the research protocols and the most recent publication.^{10}^{,11,21}
Classification of Comparator
The results of a trial may be affected by the use of an inappropriate comparator,^{18} even if the trial adheres to all contemporary standards of good design.^{20} In light of the evidence that trials using placebos or no therapy as controls may produce misleading results in favor of new treatments,^{13}^{,22}^{,23} we classified comparators in trials as either active treatment or placebo/no active treatment. We analyzed these trials separately.
ASSESSMENT OF PATTERN OF TREATMENT SUCCESSES OVER TIME
Two approaches were used: time series analysis and meta-analysis stratified by time periods. For time series analysis, we postulated that if one treatment success affects the outcome of another, we would see significant correlation between experimental treatments at time t and preceding times. However, if testing in each trial is independent of another, we would expect to see a “white noise” pattern with no significant autocorrelation in time series. Since time series may miss a trend toward improvement in outcomes due to various exogenous factors, such as increase in sample size of trials over time, we performed the subgroup meta-analyses testing for differences of treatments during 4 time periods. We tested whether treatment effect in the trials was, on average, different between trials conducted from 1950 to 1970, 1971 to 1980, 1981 to 1990, and 1991 to 2000.
Finally, we performed mathematical modeling of treatment successes in an attempt to deduce whether there is a pattern in the distribution of therapeutic advances in cancer.
All analyses were done with Stata statistical software.^{24} Curve fitting was performed with TableCurve 2D software.^{25}
Details of trial characteristics and number of trials conducted by each cooperative group are summarized in Table 1. There were 781 randomized comparisons, involving a total of 216 451 patients in 624 trials. Overall, the methodologic quality of trials conducted by the COGs was judged to be high (Table 2).
Figure 1 is a flow diagram for the inclusion of trials in our analyses and the publication rate of the trials. Of 671 completed RCTs, 602 (90%) were published. We were able to obtain data from 22 unpublished studies. However, data from 47 unpublished trials were not available for the analysis. These were early trials, records of which appear to have been lost owing to reorganization of COGs. By all accounts, the loss of these records appeared to be a random event. Thus, it is unlikely that the absence of the records on 7% (47 of 671) of the trials had a significant effect on our analysis.
We were able to assess statistical significance in 743 of the 781 randomized comparisons; investigators' judgments about overall treatment value were deduced for 766 comparisons, and survival data were available for 614 comparisons.
EVALUATION OF TREATMENT SUCCESSES
Thirty percent of the results (n = 221) were statistically significant. Of these, 20% (n = 45) favored the standard treatment and 80% (n = 176) favored the experimental treatment. Among nonsignificant findings, 3% (n = 12) of the results were true negative, 50% (n = 218) were inconclusive, 19% (n = 84) were highly unlikely to have been consistent with standard treatments being superior, and 27% (n = 119) of interventions were also deemed inconclusive but highly unlikely to have favored experimental treatments (Figure 2).
The trial investigators concluded that experimental treatments were superior in 41% (n = 316) of comparisons, while standard treatments were favored in 59% (n = 450) of comparisons (N = 766) (Figure 3). In addition, investigators judged that 116 (15%) of trials resulted in discovery of breakthrough interventions. Twelve of 614 trials (2%) identified experimental interventions that reduced death rate by 50% or more.
When data were quantitatively pooled, summary estimates for overall survival, event-free survival, and response rate slightly favored the new treatments. The HR for death from any cause was 0.95 (99% CI, 0.93-0.98); for relapse, progression, or death, 0.90 (99% CI, 0.87-0.93); and for response rate, 0.89 (99% CI, 0.81-0.98). Restricting our evaluation to primary outcomes, we also found a slight benefit in favor of new treatments. For the primary outcomes survival (death from any cause) and event-free survival (relapse, progression, or death), HRs were 0.97 (99% CI, 0.94-1.00) and 0.87 (99% CI, 0.83-0.92), respectively. For the primary outcome response rate, OR was 0.84 (99% CI, 0.74-0.96) (Figure 4A). However, new treatments were associated with increased treatment-related mortality (OR, 1.43; 99% CI, 1.26-1.62).
We investigated the robustness of our results as a function of the most important factors that may have affected the overall findings.
Effect of Cooperative Group
For 4 cooperative groups, experimental treatment resulted in slightly improved overall survival, and for 4 cooperative groups, on average, experimental treatments were as likely to be inferior as they were to be superior to standard treatments. Event-free survival from 6 of the 8 cooperative groups, were, on average, better with the use of new treatments (Figure 4B)
Effect of Methodologic Quality and Type of Treatment
As expected, sensitivity analysis according to the type of treatment and disease indicated that some areas were more successful than others. For example, we found that the use of experimental curative and adjuvant therapies were, on average, associated with largest survival benefits. Similarly, the largest survival improvements were seen in gastrointestinal and hematologic malignant neoplasms (Figure 4C).
Figure 5 shows the sensitivity analysis for the outcome of overall survival, according to the most important methodologic domains. In general, the methodologic quality of the trials was high and did not influence our findings.
PATTERN OF TREATMENT SUCCESSES OVER TIME
Figure 6 and Figure 7 show a result of time series analysis consistent with a “white noise” pattern, indicating no significant autocorrelation between studies carried out at various time intervals. This shows that each trial represents an independent experiment in a given time. The analysis also shows more consistent effects with fewer outliers in both directions and a slight weighting toward positive findings in more recent times. As suspected, this could be because the sample size over time has increased, resulting in improved precisions of estimates (Spearman ρ = 0.32; P < .001).
Subgroup analysis stratified by time periods shows a slight improvement in overall survival over time, unlikely to be clinically meaningful (P for trend = .005) (Figure 6B). Figure 7 shows subgroup analysis stratified by time according to treatment and diseases, respectively. In general, the analysis indicated consistent average treatment effect with slight improvement over time.
MODELING OF THE DISTRIBUTION OF TREATMENT SUCCESS
In an attempt to explain the observed results, we hypothesized that investigators cannot predict what they are going to discover in any individual trial that they undertake, ie, sometimes new treatments will be superior to standard treatments, sometimes the reverse, and sometimes no difference will be detected between new and standard treatments.^{10}^{,11,26}^{,27} Theoretically, the distribution of hazard ratios should then follow a normal distribution (Figure 8A). However, the Shapiro-Wilk test indicated that the results deviated from normality (z = 7.97;P < .001). We found that the best fit (r^{2} = 95%) to data was obtained by using a simple power law function: y = A/[1 + (B × x^{2})], where y represents the percentage of treatment success with new or standard treatment, respectively, and x is expressed as the natural logarithm of HR (Figure 8B).
We present a comprehensive evaluation of treatment success identified in RCTs conducted by the NCI-funded COG program. Previous research evaluating treatment successes has been limited in a variety of ways. It was typically based on a convenience sample of published studies,^{2}^{,28} did not evaluate factors that may have affected results such as publication bias^{14} or methodologic quality,^{29} was based on a small number of studies,^{30} had not used a multidimensional methodologic approach in the assessment of treatment success,^{31} or focused on success rates of individual COGs.^{10}^{,11,31}
Experimental treatments were statistically superior in 24%, and standard treatments were statistically superior in 6% of comparisons (Figure 2). The original researchers, however, concluded that the new treatments were better in 41% of comparisons (Figure 3). The real effect of new treatments compared with standard treatments in terms of patient outcomes such as survival is best measured by quantitative pooling of data. When done this way, new treatments are, on average, found to be slightly superior to standard treatments, with a 5% relative reduction in the death rate (HR, 0.95; 99% CI, 0.93-0.98). This, of course, should not be understood as the average effects of new discoveries being equally spread among all patients. This average 5% relative reduction in death from cancer due to discoveries of new treatments is a combination of some spectacular advances, which in 2% of cases cut the death rate by more than 50%, as well as smaller and moderate improvement in survival (median HR among successful treatments was 0.83, which translates into 17% average death rate reduction). However, successes were offset by an almost symmetrical number of the results favoring standard treatments, including increased treatment-related mortality associated with the use of experimental drugs.
When measured in terms of decrease in death rate, the findings show that the main advances have occurred in the management of gastrointestinal cancer and hematologic malignant neoplasms. This is presumably due to introduction of adjuvant and intensive chemotherapies, respectively. However, a number of other significant advances, which did not necessarily improve survival, were captured by our second measurement method. A typical example was introduction of lumpectomy instead of mastectomy, which resulted in dramatic improvement in the quality of life of women with breast cancer.
A critical issue in interpreting our results is whether they occurred by chance or because of some underlying reasons. We believe that the prediction of therapeutic success can be simplified by considering some of the forces that underlie clinical research.^{27} On one hand, a continuing increase in scientific discoveries will be possible only if the drive, enthusiasm, and knowledge of the researchers who invest in lengthy, time-consuming, and expensive experiments such as RCTs are maintained, and this means that those researchers probably need to have some belief in the likely success of the new treatments they assess.^{32} On the other hand, researchers cannot test all their ideas in RCTs. They are constrained by ethical precepts and pragmatic limitations. A key ethical precept is that an RCT should be done only if the physicians and the patients are uncertain about the relative effects of the new and standard treatments to be compared.^{26}^{,27,33}^{,34} This requirement is referred to as the “uncertainty principle”^{35} or “equipoise.”^{36} We have previously postulated the so-called equipoise hypothesis, arguing that treatment successes are a consequence of the predictable relationship between this ethical principle, which underlies study design and conduct, and outcomes of RCTs.^{10}^{,11,26}^{,27} According to this hypothesis, if the investigators are truly and equally uncertain or indifferent about 2 alternative options,^{27} theoretically treatment successes should follow a normal distribution.^{26}^{,27} However, the observed results did not fit a normal distribution. The reason for the deviation of the results from the normal curve is that the previous model had only focused on equipoise (ie, what is the most ethical and rational decision for patients) but did not take into account preferences of researchers toward one of the alternative treatments that are being tested. Both mechanisms are important. This unpredictability in the result of any individual trial and the expectation that there will be a proportionally larger success for new treatments means that the curve describing the probability of treatment success rate is expected to be skewed with a “heavy tail.”^{37}^{,38} Several mathematical functions can satisfy these qualitative requirements of our model, but a particularly popular one is the power law.^{37}^{,38} Indeed, the power law had an excellent fit, explaining 95% of model data (r^{2} = 0.95) (Figure 8). The power law appears to be ubiquitous and has been found to be applicable to a wide range of natural, economic, and social phenomena, including a distribution of wealth, species size, word frequencies, scientific citation counts, the sizes of various natural events, business firms, and cities.^{37}^{,38} According to the power law, the majority of successes in cancer will result in small or moderate advances, with occasional discovery of breakthrough interventions.
How efficient is our current system of RCTs in cancer at generating clinical discoveries? In the early 1980s, Mosteller^{39} estimated that we can expect that innovations will be successful about 50% of the time, which he called a “good investment.” Our results are consistent with Mosteller's estimates. This is also ethically a welcome finding because, if every RCT found that the new treatments were better, it would destroy the system of RCTs because people would not accept randomization if it meant that they would only get the better treatment if they were in the 50% of patients allocated to the new treatment. Patients would not enroll into such studies; previous research also found that it is unlikely that the public would support an RCT system in which there is greater than a 70% to 80% chance that one of the treatments will be better.^{40}^{,41} This expected probability of treatment success appears to describe the limits within which we could operate to advance treatment of cancer. For example, when people were asked under what probability of treatment success they would enroll in an RCT (with a 50:50 chance of allocation to successful treatment), only 3% of them stated that they would participate in a trial in which the probability of success is 80:20.^{40} Most people, however, would accept an RCT to be ethical if the probability of success of experimental treatment is between 50% and 70%.^{40} Similarly, we found that no institutional review board member would approve RCTs in which the probability of success of one treatment over another is 90% or more, while most of the members would approve the trial with expected probability of success between 40% and 60%.^{41} Therefore, an estimate of the probability of treatment success is fundamental to the sustenance of the RCT system, because the trials are possible to conduct—and hence bring about the new clinical discoveries—only within a certain range of the expected treatment benefits. Our data indicate the probability of such a treatment success, which seems to be optimal for most people to accept in order to continue to support the clinical trial system. Of note is that the size of treatment effect and proportion of successes have remained within a constant but limited range of plausible values, with more consistent effects seen over time (Figures 6 and 7). Therefore, there are limits to what can be discovered in RCTs: unearthing of new therapeutic success will remain within the constraints of the ethical principle of equipoise.
It can, however, be argued that the pattern of treatment successes described herein pertains to the publicly sponsored trials only, and that industry has higher success owing to more detailed knowledge of the drugs it develops.^{29} This is a testable hypothesis, which to date has not been satisfactorily answered because of the industry's unwillingness to share unpublished data.^{42} Indirectly, one can assess industry drug success rates by determining a proportion of regulatory approval for all phase 3 trials. Measured this way, the technical success rate for drug development is about 50%.^{43}
We also found that, estimating conservatively, 218 of the 743 randomized comparisons (29%) (Figure 2) produced inconclusive findings, ie, failed to answer the question asked. The reasons for such a large number of inconclusive trials are not entirely clear, but they appear not to be related to difficulties in the accrual of patients or other logistical problems, but rather to the researchers' overly optimistic assessment of the size of treatment effects that they designed their trials to measure.^{30}^{,44}^{,45} Is this the “right” price for discovery we are making, or can the system be more efficient? When we excluded inconclusive trials from the analysis, the overall success rate did not change, but obviously the number of discoveries in absolute terms decreased. Strikingly, however, the number of inconclusive studies has not changed over time (not shown).
In conclusion, society has received a good return on its investment in the COG system. The public can expect that about 25% to 50% of new cancer treatments that reach the stage of assessment in RCTs will prove to be successful. This pattern of successes has become more consistent over time. However, our results also indicate that the absolute number of discoveries might be improved if the proportion of inconclusive trials is reduced.
Correspondence: Benjamin Djulbegovic, MD, PhD, H. Lee Moffitt Cancer Center, 12902 Magnolia Dr, MRC, Room 2067H, Tampa, FL 33612 (benjamin.djulbegovic@moffitt.org).
Accepted for Publication: September 23, 2007.
Author Contributions: Dr Djulbegovic had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Djulbegovic. Acquisition of data: Djulbegovic, Kumar, and Soares. Analysis and interpretation of data: Djulbegovic, Kumar, Soares, Hozo, Bepler, Clarke, and Bennett. Drafting of the manuscript: Djulbegovic, Kumar, Soares, Hozo, Bepler, Clarke, and Bennett. Critical revision of the manuscript for important intellectual content: Djulbegovic, Kumar, Soares, Hozo, Bepler, Clarke, and Bennett. Statistical analysis: Djulbegovic, Kumar, Soares, and Hozo. Obtained funding: Djulbegovic. Administrative, technical, or material support: Djulbegovic, Kumar, and Soares. Study supervision: Djulbegovic.
Financial Disclosure: Dr Bennett received consulting fees and grant support from Sanofi-Aventis and AMGEN. Dr Bepler received consulting fees and grant support from Eli Lilly and Company and Sanofi-Aventis.
Funding/Support: This study was supported by Research Program on Research Integrity, Office of Research Integrity, and National Institutes of Health (grants 1R01NS044417-01 and 5 R01 NS052956-02).
Role of the Sponsor: The sponsor had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Additional Contributions: We sincerely thank the cooperative groups that have openly shared their data with us. Without their help, this project would not have been possible.
1.Jemal
ASiegel
RWard
EMurray
TXu
JThun
MJ Cancer statistics, 2007.
CA Cancer J Clin 2007;57
(1)
43- 66
PubMed 2.Wittes
RE Therapies for cancer in children: past successes, future challenges.
N Engl J Med 2003;348
(8)
747- 749
PubMed 3.Collins
R McMahon
S Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials.
Lancet 2001;357
(9253)
373- 380
PubMed 4.Grann
AGrann
VR The case for randomized trials in cancer treatment: new is not always better.
JAMA 2005;293
(8)
1001- 1003
PubMed 6.Chalmers
I What is the prior probability of a proposed new treatment being superior to established treatments?
BMJ 1997;314
(7073)
74- 75
PubMed 7.World Medical Association, World Medical Association Declaration of Helsinki.
Ethical Principles for Medical Research Involving Human Subjects. http://www.wma.net/e/policy/b3.htm. Accessed April 17, 2007
8.Johnston
SCRootenberg
JDKatrak
SSmith
WSElkins
JS Effect of a US National Institutes of Health programme of clinical trials on public health and costs.
Lancet 2006;367
(9519)
1319- 1327
PubMed 9.Soares
HKumar
AClarke
MDjulbegovic
B How long does it takes to publish a high quality trial in oncology? Paper presented at: XIII Cochrane Colloquium October 24, 2005 Melbourne, Australia
10.Kumar
ASoares
HWells
R
et al. Are experimental treatments for cancer in children superior to established treatments? observational study of randomised controlled trials by the Children's Oncology Group.
BMJ 2005;331
(7528)
1295
PubMed 11.Soares
HPKumar
ADaniels
S
et al. Evaluation of new treatments in radiation oncology: are they better than standard treatments?
JAMA 2005;293
(8)
970- 978
PubMed 12.Egger
MSDAltman
D Systematic Reviews in Health Care: Meta-analysis in Context. 2nd ed. London, England BMJ2001;
13.Djulbegovic
BLacevic
MCantor
A
et al. The uncertainty principle and industry-sponsored research.
Lancet 2000;356
(9230)
635- 638
PubMed 14.Als-Nielsen
BChen
WGluud
CKjaergard
LL Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events?
JAMA 2003;290
(7)
921- 928
PubMed 15.Hedges
LVOlkin
I Statistical Methods for Meta-analysis. San Diego, CA Academic Press1985;
16.Dickersin
K How important is publication bias? a synthesis of available data.
AIDS Educ Prev 1997;9
(1)
((suppl))
15- 21
PubMed 17.Jüni
PAltman
DGEgger
M Systematic reviews in health care: assessing the quality of controlled clinical trials.
BMJ 2001;323
(7303)
42- 46
PubMed 18.Mann
HDjulbegovic
B Choosing a control intervention for a randomised clinical trial.
BMC Med Res Methodol 2003;3
(1)
7
PubMed 20.Altman
DGSchulz
KFMoher
D
et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration.
Ann Intern Med 2001;134
(8)
663- 694
PubMed 21.Soares
HPDaniels
SKumar
A
et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group.
BMJ 2004;328
(7430)
22- 24
PubMed 22.Djulbegovic
BCantor
AClarke
M The importance of preservation of the ethical principle of equipoise in the design of clinical trials: relative impact of the methodological quality domains on the treatment effect in randomized controlled trials.
Account Res 2003;10
(4)
301- 315
PubMed 23.Rothman
KJMichels
KB The continuing unethical use of placebo controls.
N Engl J Med 1994;331
(6)
394- 398
PubMed 24.StataCorp, Stata Statistical Software, Release 9. College Station, TX StataCorp LP2006;
25. TableCurve 2D [computer program]. San Jose, CA Systat Software Inc2002;
26.Djulbegovic
B Acknowledgment of uncertainty: a fundamental means to ensure scientific and ethical validity in clinical research.
Curr Oncol Rep 2001;3
(5)
389- 395
PubMed 27.Djulbegovic
B Articulating and responding to uncertainties in clinical research.
J Med Philos 2007;32
(2)
79- 98
PubMed 28.Chlebowski
RTLillington
LM A decade of breast cancer clinical investigation: results as reported in the Program/Proceedings of the American Society of Clinical Oncology.
J Clin Oncol 1994;12
(9)
1789- 1795
PubMed 29.Peppercorn
JBlood
EWiner
EPartridge
A Association between pharmaceutical involvement and outcomes in breast cancer clinical trials.
Cancer 2007;109
(7)
1239- 1246
PubMed 30.Machin
DStenning
SParmar
M
et al. Thirty years of Medical Research Council randomized trials in solid tumours.
Clin Oncol (R Coll Radiol) 1997;9
(2)
100- 114
PubMed 31.Joffe
SHarrington
DPGeorge
SLEmanuel
EJBudzinski
LAWeeks
JC Satisfaction of the uncertainty principle in cancer clinical trials: retrospective cohort analysis.
BMJ 2004;328
(7454)
1463
PubMed 32.Murray
C Human Accomplishment: The Pursuit of Excellence in the Arts and Sciences, 800 BC to 1950. New York, NY HarperCollins Inc2003;
33.Chalmers
I Well informed uncertainties about the effects of treatments.
BMJ 2004;328
(7438)
475- 476
PubMed 34.Edwards
SJLLilford
RJBraunholtz
DAJackson
JCHewison
JThornton
J Ethical issues in the design and conduct of randomised controlled trials.
Health Technol Assess 1998;2
(15)
i- vi, 1-132
PubMed 35.Peto
RBaigent
C Trials: the next 50 years.
BMJ 1998;317
(7167)
1170- 1171
PubMed 36.Freedman
B Equipoise and the ethics of clinical research.
N Engl J Med 1987;317
(3)
141- 145
PubMed 37.Watts
DJ The “new” science of networks. Annu Rev Sociol August2004;30243- 27010.1146/annurev.soc.30.020404.104342
38.Newman
MEJ Power laws, Pareto distributions and Zipf's law. Contemp Phys 2005;46
(5)
323- 351
39.Mosteller
F Innovation and evaluation.
Science 1981;211
(4485)
881- 886
PubMed 40.Johnson
NLilford
JRBrazier
W At what level of collective equipoise does a clinical trial become ethical?
J Med Ethics 1991;17
(1)
30- 34
PubMed 41.Djulbegovic
BBercu
B At what level of collective equipoise does a clinical trial become ethical for the IRB members? Paper presented at: USF Third National Symposium, Bioethical Considerations in Human Subject Research March 8, 2002 Clearwater, FL
42.Lexchin
JBero
LADjulbegovic
BClark
O Pharmaceutical industry sponsorship and research outcome and quality: systematic review.
BMJ 2003;326
(7400)
1167- 1170
PubMed 43.DiMasi
JAGrabowski
HG Economics of new oncology drug development.
J Clin Oncol 2007;25
(2)
209- 216
PubMed 44.Chalmers
IMatthews
R What are the implications of optimism bias in clinical research?
Lancet 2006;367
(9509)
449- 450
PubMed 45.Kumar
ASoares
HDjulbegovic
B High proportion of high quality trials conducted by the NCI are negative or inconclusive. Paper presented at: XIII Cochrane Colloquium October 25, 2005 Melbourne, Australia