Discussion Sections in Reports of Controlled Trials Published in General Medical Journals

Context Reliable interpretation of the results of a controlled trial entails setting its results in the context of similar research. A previous study showed that most reports of controlled trials published in 5 general medical journals in May 1997 were deficient in this respect. We assessed the extent to which reports of controlled trials published in the same 5 journals discussed new results in light of the totality of evidence from other controlled trials.

Methods Assessment of the discussion sections in all 33 reports of randomized trials published during May 2001 in Annals of Internal Medicine, BMJ, JAMA, The Lancet, and The New England Journal of Medicine.

Results Three reports appeared to have been the first published trials to address the questions studied. In none of the remaining 30 reports were the results of the new trial discussed in the context of an updated systematic review of other trials. Although reference was made to relevant systematic reviews in 3 of these 30 reports, there was no integration, quantitative or qualitative, of the results of the new trials in an update of these reviews. In the remaining 27 reports, there was no evidence that any systematic attempt had been made to set the new results in the context of previous trials.

Conclusions Between 1997 and 2001, there was no evidence of progress in the proportion of reports of trials published in general medical journals that discussed the new results within the context of, or with reference to, up-to-date systematic reviews of relevant evidence from other controlled trials.

Anyone wishing to interpret a trial needs to know how its results compare with those of similar studies, a fact recognized by the original CONSORT (Consolidated Standards of Reporting Trials) statement, which recommended that the report of a randomized trial discuss its findings in light of the totality of relevant evidence.¹ In May 1997, Annals of Internal Medicine, BMJ, JAMA, The Lancet, and The New England Journal of Medicine published 26 reports of randomized trials. Reports of apparently similar trials were found for 25 of these. In only 2 were a trial's results placed in the context of an up-to-date systematic review of relevant studies. Thus, only a small proportion of the reports provided sufficient information to permit reliable interpretation of the new results.² We repeated our study in May 2001 to assess whether there had been any detectable improvement since then. In 1997, and again in this study, it was not our aim to assess the overall quality of the discussion sections of trial reports but simply to assess how well the results of the new trial had been placed in the context of other relevant research.

A report was eligible for inclusion as a trial if it met the following criteria. First, it was published during May 2001 as a full report or article (that is, not in the editorials, news, research letters, short reports, or correspondence sections) in Annals of Internal Medicine, BMJ, JAMA, The Lancet, or The New England Journal of Medicine. Second, on the basis of the best available information, the individuals (or other units) observed in the trial were assigned prospectively to 1 of 2 or more forms of health care by using random allocation or some quasi-random method of allocation, such as alternation, date of birth, or case record number, that is, randomized and quasi-randomized trials, as defined by the Cochrane Collaboration.³

If the discussion section of an eligible report contained an explicit attempt to identify and discuss all other similar trials, whether or not an attempt was made to combine their results quantitatively with those of the new trial, it was classified as a systematic review.

Each of us searched the relevant issues of the journals in different random order. Any reports judged to be eligible by at least 1 of us were considered for inclusion. We resolved any outstanding disagreements by discussion and included reports on which all 3 of us agreed.

Other trials that seem to have addressed the question concerned in the index report were sought by searching the Cochrane Controlled Trials Register (CENTRAL). We did not systematically search for all such trials or judge whether there was sufficient similarity between the new trial and other trials to combine them in a formal meta-analysis.

We independently assessed the discussion section of each eligible report to decide whether an attempt had been made within it to integrate the results of the new trial within a systematic review, either qualitatively or quantitatively. Such a review could have been one done previously or one done especially by the authors of the report. We resolved any disagreements in our assessments by discussion.

Thirty-three reports of randomized trials, available in the online reference list (available in PDF format), were identified in the 19 issues of these 5 journals published in May 2001. In 4 reports, the authors claimed that their study was the first to have addressed the question concerned. Reports of apparently similar trials were found for 1 of these 4.

None of the 30 reports for which there appear to be similar trials contained a discussion of the trial's results in the context of an up-to-date systematic review of earlier trials. Relevant systematic reviews were mentioned in the discussion section of 3 reports,^4-6 but the results of the new trial had not been integrated either qualitatively or quantitatively into an update of these reviews. However, one trial⁶ was reported in the same issue of The Lancet as a systematic review of other relevant trials.⁷ In the remaining 27 reports, there was no evidence that any systematic attempt had been made to set the results of the new trials in context (Table 1).

More than 35 years ago, Austin Bradford Hill suggested that the structure of a scientific paper could usefully be conceptualized in terms of 4 questions: Why did you start? What did you do? What answer did you get? And what does it mean anyway?⁸ These questions are reflected in a common structure for scientific reports (sometimes called IMRaD) comprising an introduction, a description of the materials and methods, the results, and a discussion of the findings. The introduction should clarify why the study was worth starting. The discussion should indicate the contribution of the new findings to the evidence available at reporting.

The public would be served better if systematic reviews were always available before new clinical studies began, which would reduce unwanted duplication of research and help ensure that new research had been designed to build on lessons from earlier research. Another consequence would be the automatic establishment of the basis for preparing a more informative discussion section when the new study is reported.^9,10

In our 2 studies, separated by 4 years, we have been unable to detect any increase in the extent to which the results of new randomized trials published in 5 prestigous general medical journals have been presented within the context of updated systematic reviews of other relevant studies (Table 1). We have not identified any other reports of empirical research assessing the extent to which this issue has been addressed. Some articles have considered other issues about the quality of discussion sections,¹¹ but the fundamental and important issue addressed in our studies seems to have been ignored. Perhaps even more worrying, the CONSORT group appears to have weakened its initial guidance on this matter by deleting their earlier reference to the need to interpret the data from a trial "in the light of the totality of the available evidence."¹²

The expectation that the results of a new randomized trial will be reported in the context of an up-to-date systematic review of earlier trials does not imply that the discussion section of every report of a randomized trial should contain a full account of the material, methods, and findings of such a review. The technology already exists to enable a brief review to be included in the discussion section, with links to relevant, up-to-date, systematic reviews published elsewhere. The encouraging development by the BMJ of including a summary with each report of new research to show what is already known on a topic and what the new study adds could be extended to provide links to the evidence, such as systematic reviews, upon which these summaries are based.

Because our expectations imply radical changes in the way that research is done and reported, we expect that not all researchers, journal editors, or publishers will agree with them. However, science is cumulative, and everyone, including the public, has a right to expect that this principle will be reflected more effectively in the way that science is conducted and reported. We feel that this imposes a duty on researchers to present their results in proper context and on journal editors to require researchers to do so.