[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.158.173.184. Please contact the publisher to request reinstatement.
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Quality Issues and Standards
June 5, 2002

Discussion Sections in Reports of Controlled Trials Published in General Medical Journals

Author Affiliations

Author Affiliations: UK Cochrane Centre, Oxford, England.

JAMA. 2002;287(21):2799-2801. doi:10.1001/jama.287.21.2799
Context

Context Reliable interpretation of the results of a controlled trial entails setting its results in the context of similar research. A previous study showed that most reports of controlled trials published in 5 general medical journals in May 1997 were deficient in this respect. We assessed the extent to which reports of controlled trials published in the same 5 journals discussed new results in light of the totality of evidence from other controlled trials.

Methods Assessment of the discussion sections in all 33 reports of randomized trials published during May 2001 in Annals of Internal Medicine, BMJ, JAMA, The Lancet, and The New England Journal of Medicine.

Results Three reports appeared to have been the first published trials to address the questions studied. In none of the remaining 30 reports were the results of the new trial discussed in the context of an updated systematic review of other trials. Although reference was made to relevant systematic reviews in 3 of these 30 reports, there was no integration, quantitative or qualitative, of the results of the new trials in an update of these reviews. In the remaining 27 reports, there was no evidence that any systematic attempt had been made to set the new results in the context of previous trials.

Conclusions Between 1997 and 2001, there was no evidence of progress in the proportion of reports of trials published in general medical journals that discussed the new results within the context of, or with reference to, up-to-date systematic reviews of relevant evidence from other controlled trials.

Anyone wishing to interpret a trial needs to know how its results compare with those of similar studies, a fact recognized by the original CONSORT (Consolidated Standards of Reporting Trials) statement, which recommended that the report of a randomized trial discuss its findings in light of the totality of relevant evidence.1 In May 1997, Annals of Internal Medicine, BMJ, JAMA, The Lancet, and The New England Journal of Medicine published 26 reports of randomized trials. Reports of apparently similar trials were found for 25 of these. In only 2 were a trial's results placed in the context of an up-to-date systematic review of relevant studies. Thus, only a small proportion of the reports provided sufficient information to permit reliable interpretation of the new results.2 We repeated our study in May 2001 to assess whether there had been any detectable improvement since then. In 1997, and again in this study, it was not our aim to assess the overall quality of the discussion sections of trial reports but simply to assess how well the results of the new trial had been placed in the context of other relevant research.

METHODS

A report was eligible for inclusion as a trial if it met the following criteria. First, it was published during May 2001 as a full report or article (that is, not in the editorials, news, research letters, short reports, or correspondence sections) in Annals of Internal Medicine, BMJ, JAMA, The Lancet, or The New England Journal of Medicine. Second, on the basis of the best available information, the individuals (or other units) observed in the trial were assigned prospectively to 1 of 2 or more forms of health care by using random allocation or some quasi-random method of allocation, such as alternation, date of birth, or case record number, that is, randomized and quasi-randomized trials, as defined by the Cochrane Collaboration.3

If the discussion section of an eligible report contained an explicit attempt to identify and discuss all other similar trials, whether or not an attempt was made to combine their results quantitatively with those of the new trial, it was classified as a systematic review.

Each of us searched the relevant issues of the journals in different random order. Any reports judged to be eligible by at least 1 of us were considered for inclusion. We resolved any outstanding disagreements by discussion and included reports on which all 3 of us agreed.

Other trials that seem to have addressed the question concerned in the index report were sought by searching the Cochrane Controlled Trials Register (CENTRAL). We did not systematically search for all such trials or judge whether there was sufficient similarity between the new trial and other trials to combine them in a formal meta-analysis.

We independently assessed the discussion section of each eligible report to decide whether an attempt had been made within it to integrate the results of the new trial within a systematic review, either qualitatively or quantitatively. Such a review could have been one done previously or one done especially by the authors of the report. We resolved any disagreements in our assessments by discussion.

RESULTS

Thirty-three reports of randomized trials, available in the online reference list (available in PDF format), were identified in the 19 issues of these 5 journals published in May 2001. In 4 reports, the authors claimed that their study was the first to have addressed the question concerned. Reports of apparently similar trials were found for 1 of these 4.

None of the 30 reports for which there appear to be similar trials contained a discussion of the trial's results in the context of an up-to-date systematic review of earlier trials. Relevant systematic reviews were mentioned in the discussion section of 3 reports,46 but the results of the new trial had not been integrated either qualitatively or quantitatively into an update of these reviews. However, one trial6 was reported in the same issue of The Lancet as a systematic review of other relevant trials.7 In the remaining 27 reports, there was no evidence that any systematic attempt had been made to set the results of the new trials in context (Table 1).

Table. Classification of Discussion Sections in Reports of Randomized Controlled Trials Published in May 1997 and May 2001 in 5 General Medical Journals
Table. Classification of Discussion Sections in Reports of Randomized Controlled Trials Published in May 1997 and May 2001 in 5 General Medical Journals
Image description not available.
COMMENT

More than 35 years ago, Austin Bradford Hill suggested that the structure of a scientific paper could usefully be conceptualized in terms of 4 questions: Why did you start? What did you do? What answer did you get? And what does it mean anyway?8 These questions are reflected in a common structure for scientific reports (sometimes called IMRaD) comprising an introduction, a description of the materials and methods, the results, and a discussion of the findings. The introduction should clarify why the study was worth starting. The discussion should indicate the contribution of the new findings to the evidence available at reporting.

The public would be served better if systematic reviews were always available before new clinical studies began, which would reduce unwanted duplication of research and help ensure that new research had been designed to build on lessons from earlier research. Another consequence would be the automatic establishment of the basis for preparing a more informative discussion section when the new study is reported.9,10

In our 2 studies, separated by 4 years, we have been unable to detect any increase in the extent to which the results of new randomized trials published in 5 prestigous general medical journals have been presented within the context of updated systematic reviews of other relevant studies (Table 1). We have not identified any other reports of empirical research assessing the extent to which this issue has been addressed. Some articles have considered other issues about the quality of discussion sections,11 but the fundamental and important issue addressed in our studies seems to have been ignored. Perhaps even more worrying, the CONSORT group appears to have weakened its initial guidance on this matter by deleting their earlier reference to the need to interpret the data from a trial "in the light of the totality of the available evidence."12

The expectation that the results of a new randomized trial will be reported in the context of an up-to-date systematic review of earlier trials does not imply that the discussion section of every report of a randomized trial should contain a full account of the material, methods, and findings of such a review. The technology already exists to enable a brief review to be included in the discussion section, with links to relevant, up-to-date, systematic reviews published elsewhere. The encouraging development by the BMJ of including a summary with each report of new research to show what is already known on a topic and what the new study adds could be extended to provide links to the evidence, such as systematic reviews, upon which these summaries are based.

Because our expectations imply radical changes in the way that research is done and reported, we expect that not all researchers, journal editors, or publishers will agree with them. However, science is cumulative, and everyone, including the public, has a right to expect that this principle will be reflected more effectively in the way that science is conducted and reported. We feel that this imposes a duty on researchers to present their results in proper context and on journal editors to require researchers to do so.

References
1.
Begg C, Cho M, Eastwood S.  et al.  Improving the quality of reporting of randomized controlled trials: the CONSORT statement.  JAMA.1996;276:637-639.
2.
Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents?  JAMA.1998;280:280-282.
3.
Clarke M, Oxman AD. Definitions of RCTs and CCTS Cochrane Reviewers Handbook 4.1.3, appendix 5b. Oxford, England: Cochrane Library, Update Software; 2002;issue 1.
4.
CAPRICORN Investigators.  Effect of carvedilol on outcome after myocardial infarction in patients with left-ventricular dysfunction: the CAPRICORN randomised trial.  Lancet.2001;357:1385-1390.
5.
Eccles M, Steen N, Grimshaw J.  et al.  Effect of audit and feedback, and reminder messages on primary-care radiology referrals: a randomised trial.  Lancet.2001;357:1406-1409.
6.
Villar J, Ba'aqeel H, Piaggio G.  et al.  WHO antenatal care randomised trial for the evaluation of a new model of routine antenatal care.  Lancet.2001;357:1551-1564.
7.
Carroli G, Villar J, Piaggio G.  et al.  WHO systematic review of randomised controlled trials of routine antenatal care.  Lancet.2001;357:1565-1570.
8.
Hill AB. The reasons for writing.  BMJ.1965;4:870.
9.
Chalmers I. Improving the quality and dissemination of reviews of clinical research. In: Lock S, ed. The Future of Medical Journals: In Commemoration of 150 Years of the British Medical Journal. London, England: British Medical Journal Books; 1991:127-146.
10.
Chalmers I, Altman DG. How can medical journals help prevent poor medical research? some opportunities presented by electronic publishing.  Lancet.1999;353:490-493.
11.
Skelton JR, Edwards SJ. The function of the discussion section in academic medical writing.  BMJ.2000;320:1269-1270.
12.
Moher D, Schulz KF, Altman DG, Lepage L. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials.  Lancet.2001;357:1191-1194.
×