Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
Peer Review Congress
July 15, 1998

Discussion Sections in Reports of Controlled Trials Published in General Medical JournalsIslands in Search of Continents?

Author Affiliations

From the Clinical Trial Service Unit, Oxford University (Dr Clarke), and the United Kingdom Cochrane Centre (Drs Clarke and Chalmers), Oxford, England.

JAMA. 1998;280(3):280-282. doi:10.1001/jama.280.3.280

Context.— Several journals have adopted the Consolidated Standards of Reporting Trials (CONSORT) recommendations to make assessment of the quality of randomized controlled trials (RCTs) easier. One of these recommendations is that the trial's results be discussed in light of the totality of the available evidence.

Objective.— To assess the extent to which reports of RCTs published in 5 general medical journals have discussed new results in light of all available evidence.

Design.— Assessment of the discussion sections in all 26 reports of RCTs published during May 1997 in Annals of Internal Medicine, BMJ, JAMA , The Lancet, and The New England Journal of Medicine.

Main Outcome Measure.— The inclusion or mention of a systematic review in the discussion section of each article.

Results.— In only 2 articles were the RCT's results discussed in the context of an updated systematic review of earlier trials. In a further 4 articles, references were made to relevant systematic reviews, but no attempts were made to integrate the results of the new trials in updated versions of these reviews. One article was probably the first published trial to address the question studied. The remaining 19 articles included no evidence that any systematic attempt had been made to set the reported trial's results in the context of previous trials.

Conclusion.— There is little evidence that journals have adequately implemented the CONSORT recommendation that results of an RCT be discussed in light of the totality of the available evidence.

SEVERAL MAJOR health care journals have already adopted the Consolidated Standards of Reporting Trials (CONSORT) recommendations to make it easier for readers to assess the quality of controlled trials.1 This is the first joint attempt by biomedical journals to improve the quality of reports of controlled trials, a topic of research for 4 decades.2 No other category of biomedical report has received such sustained attention, and this reflects the practical importance of controlled trials in guiding decisions in health care.

Previous research has highlighted deficiencies in descriptions of the materials and methods used and the analysis and presentation of results.2 Like most similar articles before it, the CONSORT statement concentrates on these 2 elements of reports of controlled trials. By contrast, the quality of introduction and discussion sections in trial reports has received little systematic scrutiny. The typical discussion section usually addresses a number of dimensions, but, crucially, it is in this section that readers will look for an answer to Bradford Hill's "bottom line" question for any research article: "What does it mean, anyway?"3 This was recognized in the CONSORT statement, which included the recommendation that trialists should "state general interpretation of the data in light of the totality (our emphasis) of the available evidence."1

Other research has illustrated how selective citation of previous research in the discussion sections of research articles can be biased. Studies that have yielded relatively dramatic results are more likely to be cited in reports of subsequent similar studies than previous studies yielding unremarkable point estimates of effects.4 In addition, authors from a particular country or specialty have been shown to selectively cite material generated from within that country5,6 or specialty.7,8

Ideally, the discussion section of the report of a new trial should involve the presentation of an up-to-date systematic review,9 as was done, for example, in the 1986 article on the First International Study of Infarct Survival (ISIS-1).10 To show how closely reports of controlled trials reflect this ideal more than a decade after ISIS-1, we assessed the discussion sections of all reports of randomized trials published in May 1997 in 5 general medical journals. This month was chosen to allow as up-to-date an assessment as possible and without prior knowledge of thearticles to be published. We concentrated on how well the discussion sections in these articles had made a systematic attempt to set the results of the reported trial in the context of the totality of the other relevant existing evidence.


An article was eligible for inclusion as a trial if it met the following criteria: (1) It was published during May 1997 as a full article (ie, not in the editorials, news, or correspondence sections of the journals) in 1 of the following journals: Annals of Internal Medicine, BMJ, JAMA , The Lancet , and The New England Journal of Medicine (all except the last of these journals have adopted the CONSORT statement). (2) On the basis of the best available information, "the individuals (or other units) followed in the trial were assigned prospectively to one of 2 (or more) alternative forms of health care using random allocation or some quasi-random method of allocation (such as alternation, date of birth, or case record number)," that is, randomized and quasi-randomized trials, as defined by the Cochrane Collaboration.11

If the discussion section of an eligible article contained the results of an attempt to identify and consider all other similar trials (if an attempt either was or was not made to combine their results quantitatively with those of the newly reported trial), it was classified as a systematic review. Relevant issues of the 5 journals were ordered using a list of random numbers. One of the authors (M.C.) searched the issues in this order and the other (I.C.) searched them in reverse order. They independently identified articles that met the eligibility criteria.

The authors independently assessed the discussion section of each eligible article to decide whether an attempt had been made to integrate the results of the new trial into a systematic review, either qualitatively or quantitatively. Such a review could have been either done previously or done especially for the trial article by the authors of that article.

Disagreements on both assessments were resolved by discussion between the authors. Any eligible articles that were initially identified by only one of the authors were read and assessed for the inclusion of systematic reviews by the other before this aspect was discussed.

Reports of other trials that seem to have addressed the question concerned in the index article were sought by searching the Cochrane Controlled Trials Register.12 Advice was also sought from others with content expertise.


Table 1 shows the number of issues of each of the 5 selected journals in May 1997 and the distribution of the articles identified.1338 Twenty-six articles were identified that met the eligibility criteria, and all articles were identified by both of us independently. One additional article was identified by one of us (M.C.) as possibly eligible, but was rejected following discussion.

Number of Issues of Each Journal and Number of Randomized Controlled Trials Identified, May 1997
Number of Issues of Each Journal and Number of Randomized Controlled Trials Identified, May 1997
Image not available

Six of the 26 eligible articles claimed to report the first trial addressing a particular question. However, following a search of the Cochrane Controlled Trials Register and discussions with the relevant Cochrane Collaborative Review Groups, similar trials were identified for 5 of the 6 articles. It was not our intention to do a systematic search for trials or to make judgments about whether there was sufficient similarity between the reported trial and previous trials to combine them in a formal meta-analysis. Instead, we simply tried to identify studies that should be considered for inclusion in a systematic review. On this basis, we judged only 1 of the 26 trials identified as likely to genuinely be the first to address a particular question.37

Of the remaining 25 articles, 2 discussed the results of the reported trial in the context of a systematic review. Four others mentioned prior systematic reviews but did not attempt to integrate the new results into updated versions of these reviews. Many of the other articles, including some that claimed to be the first trial addressing a particular question, did contain citations to previous trials, but it is unclear whether these represented all similar trials, how they had been identified, or why they had been included.


We have attempted to find studies similar to our own in which others have assessed whether trial reports set their results in the context of the totality of the evidence; so far, we have been unsuccessful. We recognize that our study can provide nothing more than a snapshot of the current state of the discussion sections of trial reports. However, the articles we assessed were published in 5 widely respected journals, 4 of which have endorsed the CONSORT recommendations. It was not our aim to assess whether the discussion sections of trial reports have improved or worsened over time. Rather we sought to assess whether a problem existed at the time of our study. We suggest that our findings confirm a problem did exist in May 1997. We welcome other studies, since these would help to assess the extent of the problem.

We did not investigate how well the authors of a trial report fulfilled the other important role of the discussion section as an opportunity to summarize and critique the findings of their own trial.39 Others have done this within particular health care questions. For example, in a study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis, Rochon and colleagues40 found the data presented did not support conclusions about efficacy in nearly one fifth of the articles, and did not support claims about toxicity in about half the articles. We did not investigate whether each article's introduction section included details of a systematic review done in advance of the reported research. Any such review was likely to be out of date by the time the new trial was completed and published. If a review had been done in advance of the new trial, then it would seem sensible to update it in the discussion section.

More than 10 years after the ISIS-1 article, considerable scope remains for authors and journals to help readers set the results of new trials in the context of previous research. Very occasionally, a new study can be confidently regarded as unique and the manuscript can thus justifiably be regarded as providing the totality of evidence addressing a particular question. However, these circumstances are likely to be rare, and such claims need to be justified by a thorough search for earlier studies. Smith and Goodman41 concluded that the failure of authors to mention relevant previous work is probably a widespread form of research malpractice. They commended the suggestion that, when submitting reports for journal review, investigators should provide evidence that they have made a thorough search for relevant previous work.7

An editorial accompanying the 1986 ISIS-1 article referring to the updated systematic review of relevant data presented in its discussion section acknowledged that "there is a good case for such analyses." But it went on to state that if anyone suggested these should become a regular feature of clinical trial reports, The Lancet would "lead the opposition."42 There seems less justification for such a response 12 years later with the development of electronic publication. For example, 1 of the 2 articles identified in this study as containing an updated systematic review (both of which were published in The Lancet) summarized the findings of that review and referred readers to the electronically published versions of the relevant reviews.23,43,44

The best way to report the findings of a new trial in the context of earlier studies would be to include an updated systematic review in the discussion section. This would reduce the tendency for the new trial or a select group of previous trials to be viewed in isolation from the whole body of relevant previous research. If, in planning their study, the trialists had to identify and consider relevant prior research, then the production of a systematic review to incorporate their eventual results should be much easier. In addition, with the rapidly increasing number of systematic reviews included in the Cochrane Library,45 trialists should be increasingly able to find a ready source for the context into which to set their results.

The public is often confused by the conflicting messages it receives as a result of piecemeal reporting of research. To deserve the public's continued support and trust, researchers and journals need to ensure that reports of research end with scientifically defensible answers to Bradford Hill's question, "What does it mean, anyway?" Those who turn to reports of trials to help guide treatment deserve nothing less than a discussion of the totality of the relevant evidence, as rightly recommended by the CONSORT Group. To paraphrase John Donne, No trial is an island, entire of itself; every trial is a piece of the continent, a part of the main.46

Begg C, Cho M, Eastwood S.  et al.  Improving the quality of reporting of randomized controlled trials: the CONSORT statement.  JAMA.1996;276:637-639.
Hemminki E. Quality of clinical trials: a concern of three decades.  Methods Inf Med.1982;21:81-85.
Hill AB. The reasons for writing.  BMJ.1965;4:870.
Gotzsche PC. Reference bias in reports of drug trials.  Br Med J (Clin Res Ed).1987;295:654-656.
Campbell FM. National bias: a comparison of citation practices by health professionals.  Bull Med Libr Assoc.1990;78:376-382.
Gibbs WW. Lost science in the Third World.  Sci Am.1995;273:92-99.
Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias.  JAMA.1990;263:1392-1395.
Wessely S, Joyce J, Rabe-Hesketh S. Reviewing the reviews: the case of chronic fatigue syndrome. Paper presented at: Third International Congress on Peer Review in Biomedical Publication; September 19, 1997; Prague, Czech Republic.
Saunders MC, Dick JS, Brown IM, McPherson K, Chalmers I. The effects of hospital admission for bed rest on the duration of twin pregnancy: a randomised trial.  Lancet.1985;2:793-795.
First International Study of Infarct Survival Collaborative Group.  Randomised trial of intravenous atenolol among 16,027 cases of suspected acute myocardial infarction: ISIS-1.  Lancet.1986;2:57-66.
Mulrow CD, Oxman AD. Cochrane Collaboration Handbook: Definitions of RCTs and CCTS . Appendix 5b. In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update Software; 1996. Updated quarterly.
 Cochrane Controlled Trials Register . In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update Software; 1997. Issue 2.
Abraham E, Glauser MP, Butler T.  et al.  p55 Tumor necrosis factor receptor fusion protein in the treatment of patients with severe sepsis and septic shock: a randomized controlled multicenter trial.  JAMA.1997;277:1531-1538.
Bishop NJ, Morley R, Day JP, Lucas A. Aluminum neurotoxicity in preterm infants receiving intravenous-feeding solutions.  N Engl J Med.1997;336:1557-1561.
Bracken MB, Shepard MJ, Holford TR.  et al.  Administration of methylprednisolone for 24 or 48 hours or tirilazad mesylate for 48 hours in the treatment of acute spinal cord injury: results of the Third National Acute Spinal Cord Injury Randomized Controlled Trial.  JAMA.1997;277:1597-1604.
CAESAR Coordinating Committee.  Randomised trial of addition of lamivudine or lamivudine plus loviride to zidovudine-containing regimens for patients with HIV-1 infection: the CAESAR trial.  Lancet.1997;349:1413-1421.
Chung SS, Lau JY, Sung JJ.  et al.  Randomised comparison between adrenaline injection alone and adrenaline injection plus heat probe treatment for actively bleeding ulcers.  BMJ.1997;314:1307-1311.
Creed F, Mbaya P, Lancashire S, Tomenson B, Williams B, Holme S. Cost effectiveness of day and inpatient psychiatric treatment: results of a randomised controlled trial.  BMJ.1997;314:1381-1385.
Gasche C, Dejaco C, Waldhoer T.  et al.  Intravenous iron and erythropoietin for anemia associated with Crohn disease: a randomized, controlled trial.  Ann Intern Med.1997;126:782-787.
Gianni AM, Bregni M, Siena S.  et al.  High-dose chemotherapy and autologous bone marrow transplantation compared with MACOP-B in aggressive B-cell lymphoma.  N Engl J Med.1997;336:1290-1297.
Grossman DC, Neckerman HJ, Koepsell TD.  et al.  Effectiveness of a violence prevention curriculum among children in elementary school: a randomized controlled trial.  JAMA.1997;277:1605-1611.
Hall AS, Murray GD, Ball SG. Follow-up study of patients randomly allocated ramipril or placebo for heart failure after acute myocardial infarction: AIRE Extension (AIREX) Study.  Lancet.1997;349:1493-1497.
International Stroke Trial Collaborative Group.  The International Stroke Trial (IST): a randomised trial of aspirin, subcutaneous heparin, both, or neither among 19,435 patients with acute ischaemic stroke.  Lancet.1997;349:1569-1581.
Jacobson JM, Greenspan JS, Spritzler J.  et al.  Thalidomide for the treatment of oral aphthous ulcers in patients with human immunodeficiency virus infection.  N Engl J Med.1997;336:1487-1493.
Khan WA, Seas C, Dhar U, Salam MA, Bennish ML. Treatment of shigellosis, V: comparison of azithromycin and ciprofloxacin: a double-blind, randomized, controlled trial.  Ann Intern Med.1997;126:697-703.
Liem MS, van der Graaf Y, van Steensel CJ.  et al.  Comparison of conventional anterior surgery and laparoscopic surgery for inguinal-hernia repair.  N Engl J Med.1997;336:1541-1547.
Luzuriaga K, Bryson Y, Krogstad P.  et al.  Combination treatment with zidovudine, didanosine, and nevirapine in infants with human immunodeficiency virus type 1 infection.  N Engl J Med.1997;336:1343-1349.
Malmberg K.for the DIGAMI Study Group.  Prospective randomised study of intensive insulin treatment on long term survival after acute myocardial infarction in patients with diabetes mellitus.  BMJ.1997;314:1512-1515.
Meydani SN, Meydani M, Blumberg JB.  et al.  Vitamin E supplementation and in vivo immune response in healthy elderly subjects. A randomized controlled trial.  JAMA.1997;277:1380-1386.
Perez V, Gilaberte I, Faries D, Alvarez E, Artigas F. Randomised, double-blind, placebo-controlled trial of pindolol in combination with fluoxetine antidepressant treatment.  Lancet.1997;349:1594-1597.
Quinn J, Wells G, Sutcliffe T.  et al.  A randomized trial comparing octylcyanoacrylate tissue adhesive and sutures in the management of lacerations.  JAMA.1997;277:1527-1530.
Schuman P, Capps L, Peng G.  et al.  Weekly fluconazole for the prevention of mucosal candidiasis in women with HIV infection: a randomized, double-blind, placebo-controlled trial.  Ann Intern Med.1997;126:689-696.
Spruance SL, Rea TL, Thoming C, Tucker R, Saltzman R, Boon R. Penciclovir cream for the treatment of herpes simplex labialis: a randomized, multicenter, double-blind, placebo-controlled trial.  JAMA.1997;277:1374-1379.
The IMPACT-II Investigators.  Randomised placebo-controlled trial of effect of eptifibatide on complications of percutaneous coronary intervention: Integrilin to Minimise Platelet Aggregation and Coronary Thrombosis-II.  Lancet.1997;349:1422-1428.
The CAPTURE Investigators.  Randomised placebo-controlled trial of abciximab before and during coronary intervention in refractory unstable angina: the CAPTURE Study.  Lancet.1997;349:1429-1435.
Tingle AJ, Mitchell LA, Grace M.  et al.  Randomised double-blind placebo-controlled study on adverse effects of rubella immunisation in seronegative women.  Lancet.1997;349:1277-1281.
Wilding P, Clark M, Coon JT.  et al.  Effect of long-term treatment with salmeterol on asthma control: a double blind, randomised crossover study.  BMJ.1997;314:1441-1446.
Winters JC, Sobel JS, Groenier KH, Arendzen HJ, Meyboom de Jong B. Comparison of physiotherapy, manipulation, and corticosteroid injection for treating shoulder complaints in general practice: randomised, single blind study.  BMJ.1997;314:1320-1325.
Tyson JE, Furzan JA, Reisch JS, Mize SG. An evaluation of the quality of therapeutic studies in perinatal medicine.  J Pediatr.1983;102:10-13.
Rochon PA, Gurwitz JH, Simms RW.  et al.  A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis.  Arch Intern Med.1994;154:157-163.
Smith AJ, Goodman NW. The hypertensive response to intubation: do researchers acknowledge previous work?  Can J Anaesth.1997;44:9-13.
 Intravenous beta-blockade during acute myocardial infarction [editorial].  Lancet.1986;2:79-80.
Counsell C, Sandercock P. The efficacy and safety of anticoagulant therapy in patients with assumed ischaemic stroke. In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update Software; 1997. Issue 1.
Counsell C, Sandercock P. The efficacy and safety of antiplatelet therapy in patients with assumed ischaemic stroke. In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update Software; 1997. Issue 1.
 The Cochrane Library [database on CD-ROM]. Oxford, England: Update Software; 1997.
Donne J. Devotion Upon Emergent Occasions . Meditation 17.