Context Reliable interpretation of the results of a controlled trial entails
setting its results in the context of similar research. A previous study showed
that most reports of controlled trials published in 5 general medical journals
in May 1997 were deficient in this respect. We assessed the extent to which
reports of controlled trials published in the same 5 journals discussed new
results in light of the totality of evidence from other controlled trials.
Methods Assessment of the discussion sections in all 33 reports of randomized
trials published during May 2001 in Annals of Internal Medicine, BMJ, JAMA, The Lancet, and The New England Journal of Medicine.
Results Three reports appeared to have been the first published trials to address
the questions studied. In none of the remaining 30 reports were the results
of the new trial discussed in the context of an updated systematic review
of other trials. Although reference was made to relevant systematic reviews
in 3 of these 30 reports, there was no integration, quantitative or qualitative,
of the results of the new trials in an update of these reviews. In the remaining
27 reports, there was no evidence that any systematic attempt had been made
to set the new results in the context of previous trials.
Conclusions Between 1997 and 2001, there was no evidence of progress in the proportion
of reports of trials published in general medical journals that discussed
the new results within the context of, or with reference to, up-to-date systematic
reviews of relevant evidence from other controlled trials.
Anyone wishing to interpret a trial needs to know how its results compare
with those of similar studies, a fact recognized by the original CONSORT (Consolidated
Standards of Reporting Trials) statement, which recommended that the report
of a randomized trial discuss its findings in light of the totality of relevant
evidence.1 In May 1997, Annals of Internal Medicine, BMJ, JAMA, The Lancet, and The New England Journal
of Medicine published 26 reports of randomized trials. Reports of apparently
similar trials were found for 25 of these. In only 2 were a trial's results
placed in the context of an up-to-date systematic review of relevant studies.
Thus, only a small proportion of the reports provided sufficient information
to permit reliable interpretation of the new results.2
We repeated our study in May 2001 to assess whether there had been any detectable
improvement since then. In 1997, and again in this study, it was not our aim
to assess the overall quality of the discussion sections of trial reports
but simply to assess how well the results of the new trial had been placed
in the context of other relevant research.
A report was eligible for inclusion as a trial if it met the following
criteria. First, it was published during May 2001 as a full report or article
(that is, not in the editorials, news, research letters, short reports, or
correspondence sections) in Annals of Internal Medicine, BMJ, JAMA, The Lancet, or The New England Journal of Medicine.
Second, on the basis of the best available information, the individuals (or
other units) observed in the trial were assigned prospectively to 1 of 2 or
more forms of health care by using random allocation or some quasi-random
method of allocation, such as alternation, date of birth, or case record number,
that is, randomized and quasi-randomized trials, as defined by the Cochrane
Collaboration.3
If the discussion section of an eligible report contained an explicit
attempt to identify and discuss all other similar trials, whether or not an
attempt was made to combine their results quantitatively with those of the
new trial, it was classified as a systematic review.
Each of us searched the relevant issues of the journals in different
random order. Any reports judged to be eligible by at least 1 of us were considered
for inclusion. We resolved any outstanding disagreements by discussion and
included reports on which all 3 of us agreed.
Other trials that seem to have addressed the question concerned in the
index report were sought by searching the Cochrane Controlled Trials Register
(CENTRAL). We did not systematically search for all such trials or judge whether
there was sufficient similarity between the new trial and other trials to
combine them in a formal meta-analysis.
We independently assessed the discussion section of each eligible report
to decide whether an attempt had been made within it to integrate the results
of the new trial within a systematic review, either qualitatively or quantitatively.
Such a review could have been one done previously or one done especially by
the authors of the report. We resolved any disagreements in our assessments
by discussion.
Thirty-three reports of randomized trials, available in the online reference list (available in PDF format),
were identified in the 19 issues of these 5 journals published in May
2001. In 4 reports, the authors claimed that their study was the first to
have addressed the question concerned. Reports of apparently similar trials
were found for 1 of these 4.
None of the 30 reports for which there appear to be similar trials contained
a discussion of the trial's results in the context of an up-to-date systematic
review of earlier trials. Relevant systematic reviews were mentioned in the
discussion section of 3 reports,4-6
but the results of the new trial had not been integrated either qualitatively
or quantitatively into an update of these reviews. However, one trial6 was reported in the same issue of The Lancet as a systematic review of other relevant trials.7 In the remaining 27 reports, there was no evidence
that any systematic attempt had been made to set the results of the new trials
in context (Table 1).
Table. Classification of Discussion Sections in Reports of Randomized Controlled Trials Published in May 1997 and May 2001 in 5 General Medical Journals
More than 35 years ago, Austin Bradford Hill suggested that the structure
of a scientific paper could usefully be conceptualized in terms of 4 questions:
Why did you start? What did you do? What answer did you get? And what does
it mean anyway?8 These questions are reflected
in a common structure for scientific reports (sometimes called IMRaD) comprising an introduction, a description of the materials and
methods, the results, and a discussion of the findings. The introduction should
clarify why the study was worth starting. The discussion should indicate the
contribution of the new findings to the evidence available at reporting.
The public would be served better if systematic reviews were always
available before new clinical studies began, which would reduce unwanted duplication
of research and help ensure that new research had been designed to build on
lessons from earlier research. Another consequence would be the automatic
establishment of the basis for preparing a more informative discussion section
when the new study is reported.9,10
In our 2 studies, separated by 4 years, we have been unable to detect
any increase in the extent to which the results of new randomized trials published
in 5 prestigous general medical journals have been presented within the context
of updated systematic reviews of other relevant studies (Table 1). We have not identified any other reports of empirical
research assessing the extent to which this issue has been addressed. Some
articles have considered other issues about the quality of discussion sections,11 but the fundamental and important issue addressed
in our studies seems to have been ignored. Perhaps even more worrying, the
CONSORT group appears to have weakened its initial guidance on this matter
by deleting their earlier reference to the need to interpret the data from
a trial "in the light of the totality of the available evidence."12
The expectation that the results of a new randomized trial will be reported
in the context of an up-to-date systematic review of earlier trials does not
imply that the discussion section of every report of a randomized trial should
contain a full account of the material, methods, and findings of such a review.
The technology already exists to enable a brief review to be included in the
discussion section, with links to relevant, up-to-date, systematic reviews
published elsewhere. The encouraging development by the BMJ of including a summary with each report of new research to show
what is already known on a topic and what the new study adds could be extended
to provide links to the evidence, such as systematic reviews, upon which these
summaries are based.
Because our expectations imply radical changes in the way that research
is done and reported, we expect that not all researchers, journal editors,
or publishers will agree with them. However, science is cumulative, and everyone,
including the public, has a right to expect that this principle will be reflected
more effectively in the way that science is conducted and reported. We feel
that this imposes a duty on researchers to present their results in proper
context and on journal editors to require researchers to do so.
1.Begg C, Cho M, Eastwood S.
et al. Improving the quality of reporting of randomized controlled trials:
the CONSORT statement.
JAMA.1996;276:637-639.Google Scholar 2.Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general
medical journals: islands in search of continents?
JAMA.1998;280:280-282.Google Scholar 3.Clarke M, Oxman AD. Definitions of RCTs and CCTS Cochrane Reviewers Handbook 4.1.3, appendix
5b. Oxford, England: Cochrane Library, Update Software; 2002;issue 1.
4.CAPRICORN Investigators. Effect of carvedilol on outcome after myocardial infarction in patients
with left-ventricular dysfunction: the CAPRICORN randomised trial.
Lancet.2001;357:1385-1390.Google Scholar 5.Eccles M, Steen N, Grimshaw J.
et al. Effect of audit and feedback, and reminder messages on primary-care
radiology referrals: a randomised trial.
Lancet.2001;357:1406-1409.Google Scholar 6.Villar J, Ba'aqeel H, Piaggio G.
et al. WHO antenatal care randomised trial for the evaluation of a new model
of routine antenatal care.
Lancet.2001;357:1551-1564.Google Scholar 7.Carroli G, Villar J, Piaggio G.
et al. WHO systematic review of randomised controlled trials of routine antenatal
care.
Lancet.2001;357:1565-1570.Google Scholar 9.Chalmers I. Improving the quality and dissemination of reviews of clinical research. In: Lock S, ed. The Future of Medical Journals:
In Commemoration of 150 Years of the British Medical Journal. London,
England: British Medical Journal Books; 1991:127-146.
10.Chalmers I, Altman DG. How can medical journals help prevent poor medical research? some opportunities
presented by electronic publishing.
Lancet.1999;353:490-493.Google Scholar 11.Skelton JR, Edwards SJ. The function of the discussion section in academic medical writing.
BMJ.2000;320:1269-1270.Google Scholar 12.Moher D, Schulz KF, Altman DG, Lepage L. The CONSORT statement: revised recommendations for improving the quality
of reports of parallel-group randomised trials.
Lancet.2001;357:1191-1194.Google Scholar