Context.— Several journals have adopted the Consolidated Standards of Reporting
Trials (CONSORT) recommendations to make assessment of the quality of randomized
controlled trials (RCTs) easier. One of these recommendations is that the
trial's results be discussed in light of the totality of the available evidence.
Objective.— To assess the extent to which reports of RCTs published in 5 general
medical journals have discussed new results in light of all available evidence.
Design.— Assessment of the discussion sections in all 26 reports of RCTs published
during May 1997 in Annals of Internal Medicine, BMJ, JAMA , The Lancet,
and The New England Journal of Medicine.
Main Outcome Measure.— The inclusion or mention of a systematic review in the discussion section
of each article.
Results.— In only 2 articles were the RCT's results discussed in the context of
an updated systematic review of earlier trials. In a further 4 articles, references
were made to relevant systematic reviews, but no attempts were made to integrate
the results of the new trials in updated versions of these reviews. One article
was probably the first published trial to address the question studied. The
remaining 19 articles included no evidence that any systematic attempt had
been made to set the reported trial's results in the context of previous trials.
Conclusion.— There is little evidence that journals have adequately implemented the
CONSORT recommendation that results of an RCT be discussed in light of the
totality of the available evidence.
SEVERAL MAJOR health care journals have already adopted the Consolidated
Standards of Reporting Trials (CONSORT) recommendations to make it easier
for readers to assess the quality of controlled trials.1
This is the first joint attempt by biomedical journals to improve the quality
of reports of controlled trials, a topic of research for 4 decades.2 No other category of biomedical report has received
such sustained attention, and this reflects the practical importance of controlled
trials in guiding decisions in health care.
Previous research has highlighted deficiencies in descriptions of the
materials and methods used and the analysis and presentation of results.2 Like most similar articles before it, the CONSORT
statement concentrates on these 2 elements of reports of controlled trials.
By contrast, the quality of introduction and discussion sections in trial
reports has received little systematic scrutiny. The typical discussion section
usually addresses a number of dimensions, but, crucially, it is in this section
that readers will look for an answer to Bradford Hill's "bottom line" question
for any research article: "What does it mean, anyway?"3
This was recognized in the CONSORT statement, which included the recommendation
that trialists should "state general interpretation of the data in light of
the totality (our emphasis) of the available evidence."1
Other research has illustrated how selective citation of previous research
in the discussion sections of research articles can be biased. Studies that
have yielded relatively dramatic results are more likely to be cited in reports
of subsequent similar studies than previous studies yielding unremarkable
point estimates of effects.4 In addition, authors
from a particular country or specialty have been shown to selectively cite
material generated from within that country5,6
or specialty.7,8
Ideally, the discussion section of the report of a new trial should
involve the presentation of an up-to-date systematic review,9
as was done, for example, in the 1986 article on the First International Study
of Infarct Survival (ISIS-1).10 To show how
closely reports of controlled trials reflect this ideal more than a decade
after ISIS-1, we assessed the discussion sections of all reports of randomized
trials published in May 1997 in 5 general medical journals. This month was
chosen to allow as up-to-date an assessment as possible and without prior
knowledge of thearticles to be published. We concentrated on how well the
discussion sections in these articles had made a systematic attempt to set
the results of the reported trial in the context of the totality of the other
relevant existing evidence.
An article was eligible for inclusion as a trial if it met the following
criteria: (1) It was published during May 1997 as a full article (ie, not
in the editorials, news, or correspondence sections of the journals) in 1
of the following journals: Annals of Internal Medicine, BMJ, JAMA , The Lancet
, and The New England Journal of Medicine (all except
the last of these journals have adopted the CONSORT statement). (2) On the
basis of the best available information, "the individuals (or other units)
followed in the trial were assigned prospectively to one of 2 (or more) alternative
forms of health care using random allocation or some quasi-random method of
allocation (such as alternation, date of birth, or case record number)," that
is, randomized and quasi-randomized trials, as defined by the Cochrane Collaboration.11
If the discussion section of an eligible article contained the results
of an attempt to identify and consider all other similar trials (if an attempt
either was or was not made to combine their results quantitatively with those
of the newly reported trial), it was classified as a systematic review. Relevant
issues of the 5 journals were ordered using a list of random numbers. One
of the authors (M.C.) searched the issues in this order and the other (I.C.)
searched them in reverse order. They independently identified articles that
met the eligibility criteria.
The authors independently assessed the discussion section of each eligible
article to decide whether an attempt had been made to integrate the results
of the new trial into a systematic review, either qualitatively or quantitatively.
Such a review could have been either done previously or done especially for
the trial article by the authors of that article.
Disagreements on both assessments were resolved by discussion between
the authors. Any eligible articles that were initially identified by only
one of the authors were read and assessed for the inclusion of systematic
reviews by the other before this aspect was discussed.
Reports of other trials that seem to have addressed the question concerned
in the index article were sought by searching the Cochrane Controlled Trials
Register.12 Advice was also sought from others
with content expertise.
Table 1 shows the number
of issues of each of the 5 selected journals in May 1997 and the distribution
of the articles identified.13- 38
Twenty-six articles were identified that met the eligibility criteria, and
all articles were identified by both of us independently. One additional article
was identified by one of us (M.C.) as possibly eligible, but was rejected
following discussion.
Number of Issues of Each Journal and Number of Randomized Controlled Trials Identified, May 1997
Six of the 26 eligible articles claimed to report the first trial addressing
a particular question. However, following a search of the Cochrane Controlled Trials Register and discussions with the relevant
Cochrane Collaborative Review Groups, similar trials were identified for 5
of the 6 articles. It was not our intention to do a systematic search for
trials or to make judgments about whether there was sufficient similarity
between the reported trial and previous trials to combine them in a formal
meta-analysis. Instead, we simply tried to identify studies that should be
considered for inclusion in a systematic review. On this basis, we judged
only 1 of the 26 trials identified as likely to genuinely be the first to
address a particular question.37
Of the remaining 25 articles, 2 discussed the results of the reported
trial in the context of a systematic review. Four others mentioned prior systematic
reviews but did not attempt to integrate the new results into updated versions
of these reviews. Many of the other articles, including some that claimed
to be the first trial addressing a particular question, did contain citations
to previous trials, but it is unclear whether these represented all similar
trials, how they had been identified, or why they had been included.
We have attempted to find studies similar to our own in which others
have assessed whether trial reports set their results in the context of the
totality of the evidence; so far, we have been unsuccessful. We recognize
that our study can provide nothing more than a snapshot of the current state
of the discussion sections of trial reports. However, the articles we assessed
were published in 5 widely respected journals, 4 of which have endorsed the
CONSORT recommendations. It was not our aim to assess whether the discussion
sections of trial reports have improved or worsened over time. Rather we sought
to assess whether a problem existed at the time of our study. We suggest that
our findings confirm a problem did exist in May 1997. We welcome other studies,
since these would help to assess the extent of the problem.
We did not investigate how well the authors of a trial report fulfilled
the other important role of the discussion section as an opportunity to summarize
and critique the findings of their own trial.39
Others have done this within particular health care questions. For example,
in a study of manufacturer-supported trials of nonsteroidal anti-inflammatory
drugs in the treatment of arthritis, Rochon and colleagues40
found the data presented did not support conclusions about efficacy in nearly
one fifth of the articles, and did not support claims about toxicity in about
half the articles. We did not investigate whether each article's introduction
section included details of a systematic review done in advance of the reported
research. Any such review was likely to be out of date by the time the new
trial was completed and published. If a review had been done in advance of
the new trial, then it would seem sensible to update it in the discussion
section.
More than 10 years after the ISIS-1 article, considerable scope remains
for authors and journals to help readers set the results of new trials in
the context of previous research. Very occasionally, a new study can be confidently
regarded as unique and the manuscript can thus justifiably be regarded as
providing the totality of evidence addressing a particular question. However,
these circumstances are likely to be rare, and such claims need to be justified
by a thorough search for earlier studies. Smith and Goodman41
concluded that the failure of authors to mention relevant previous work is
probably a widespread form of research malpractice. They commended the suggestion
that, when submitting reports for journal review, investigators should provide
evidence that they have made a thorough search for relevant previous work.7
An editorial accompanying the 1986 ISIS-1 article referring to the updated
systematic review of relevant data presented in its discussion section acknowledged
that "there is a good case for such analyses." But it went on to state that
if anyone suggested these should become a regular feature of clinical trial
reports, The Lancet would "lead the opposition."42 There seems less justification for such a response
12 years later with the development of electronic publication. For example,
1 of the 2 articles identified in this study as containing an updated systematic
review (both of which were published in The Lancet)
summarized the findings of that review and referred readers to the electronically
published versions of the relevant reviews.23,43,44
The best way to report the findings of a new trial in the context of
earlier studies would be to include an updated systematic review in the discussion
section. This would reduce the tendency for the new trial or a select group
of previous trials to be viewed in isolation from the whole body of relevant
previous research. If, in planning their study, the trialists had to identify
and consider relevant prior research, then the production of a systematic
review to incorporate their eventual results should be much easier. In addition,
with the rapidly increasing number of systematic reviews included in the Cochrane
Library,45 trialists should be increasingly
able to find a ready source for the context into which to set their results.
The public is often confused by the conflicting messages it receives
as a result of piecemeal reporting of research. To deserve the public's continued
support and trust, researchers and journals need to ensure that reports of
research end with scientifically defensible answers to Bradford Hill's question,
"What does it mean, anyway?" Those who turn to reports of trials to help guide
treatment deserve nothing less than a discussion of the totality of the relevant
evidence, as rightly recommended by the CONSORT Group. To paraphrase John
Donne, No trial is an island, entire of itself; every trial is a piece of
the continent, a part of the main.46
1.Begg C, Cho M, Eastwood S.
et al. Improving the quality of reporting of randomized controlled trials:
the CONSORT statement. JAMA.1996;276:637-639.
2.Hemminki E. Quality of clinical trials: a concern of three decades. Methods Inf Med.1982;21:81-85.
3.Hill AB. The reasons for writing. BMJ.1965;4:870.
4.Gotzsche PC. Reference bias in reports of drug trials. Br Med J (Clin Res Ed).1987;295:654-656.
5.Campbell FM. National bias: a comparison of citation practices by health professionals. Bull Med Libr Assoc.1990;78:376-382.
6.Gibbs WW. Lost science in the Third World. Sci Am.1995;273:92-99.
7.Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias. JAMA.1990;263:1392-1395.
8.Wessely S, Joyce J, Rabe-Hesketh S. Reviewing the reviews: the case of chronic fatigue syndrome. Paper presented at: Third International Congress on Peer Review in
Biomedical Publication; September 19, 1997; Prague, Czech Republic.
9.Saunders MC, Dick JS, Brown IM, McPherson K, Chalmers I. The effects of hospital admission for bed rest on the duration of twin
pregnancy: a randomised trial. Lancet.1985;2:793-795.
10.First International Study of Infarct Survival Collaborative Group. Randomised trial of intravenous atenolol among 16,027 cases of suspected
acute myocardial infarction: ISIS-1. Lancet.1986;2:57-66.
11.Mulrow CD, Oxman AD. Cochrane Collaboration Handbook: Definitions of RCTs and CCTS . Appendix 5b. In: The Cochrane Library [database on CD-ROM]. Oxford,
England: Update Software; 1996. Updated quarterly.
12. Cochrane Controlled Trials Register . In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update
Software; 1997. Issue 2.
13.Abraham E, Glauser MP, Butler T.
et al. p55 Tumor necrosis factor receptor fusion protein in the treatment
of patients with severe sepsis and septic shock: a randomized controlled multicenter
trial. JAMA.1997;277:1531-1538.
14.Bishop NJ, Morley R, Day JP, Lucas A. Aluminum neurotoxicity in preterm infants receiving intravenous-feeding
solutions. N Engl J Med.1997;336:1557-1561.
15.Bracken MB, Shepard MJ, Holford TR.
et al. Administration of methylprednisolone for 24 or 48 hours or tirilazad
mesylate for 48 hours in the treatment of acute spinal cord injury: results
of the Third National Acute Spinal Cord Injury Randomized Controlled Trial. JAMA.1997;277:1597-1604.
16.CAESAR Coordinating Committee. Randomised trial of addition of lamivudine or lamivudine plus loviride
to zidovudine-containing regimens for patients with HIV-1 infection: the CAESAR
trial. Lancet.1997;349:1413-1421.
17.Chung SS, Lau JY, Sung JJ.
et al. Randomised comparison between adrenaline injection alone and adrenaline
injection plus heat probe treatment for actively bleeding ulcers. BMJ.1997;314:1307-1311.
18.Creed F, Mbaya P, Lancashire S, Tomenson B, Williams B, Holme S. Cost effectiveness of day and inpatient psychiatric treatment: results
of a randomised controlled trial. BMJ.1997;314:1381-1385.
19.Gasche C, Dejaco C, Waldhoer T.
et al. Intravenous iron and erythropoietin for anemia associated with Crohn
disease: a randomized, controlled trial. Ann Intern Med.1997;126:782-787.
20.Gianni AM, Bregni M, Siena S.
et al. High-dose chemotherapy and autologous bone marrow transplantation compared
with MACOP-B in aggressive B-cell lymphoma. N Engl J Med.1997;336:1290-1297.
21.Grossman DC, Neckerman HJ, Koepsell TD.
et al. Effectiveness of a violence prevention curriculum among children in
elementary school: a randomized controlled trial. JAMA.1997;277:1605-1611.
22.Hall AS, Murray GD, Ball SG. Follow-up study of patients randomly allocated ramipril or placebo
for heart failure after acute myocardial infarction: AIRE Extension (AIREX)
Study. Lancet.1997;349:1493-1497.
23.International Stroke Trial Collaborative Group. The International Stroke Trial (IST): a randomised trial of aspirin,
subcutaneous heparin, both, or neither among 19,435 patients with acute ischaemic
stroke. Lancet.1997;349:1569-1581.
24.Jacobson JM, Greenspan JS, Spritzler J.
et al. Thalidomide for the treatment of oral aphthous ulcers in patients with
human immunodeficiency virus infection. N Engl J Med.1997;336:1487-1493.
25.Khan WA, Seas C, Dhar U, Salam MA, Bennish ML. Treatment of shigellosis, V: comparison of azithromycin and ciprofloxacin:
a double-blind, randomized, controlled trial. Ann Intern Med.1997;126:697-703.
26.Liem MS, van der Graaf Y, van Steensel CJ.
et al. Comparison of conventional anterior surgery and laparoscopic surgery
for inguinal-hernia repair. N Engl J Med.1997;336:1541-1547.
27.Luzuriaga K, Bryson Y, Krogstad P.
et al. Combination treatment with zidovudine, didanosine, and nevirapine in
infants with human immunodeficiency virus type 1 infection. N Engl J Med.1997;336:1343-1349.
28.Malmberg K.for the DIGAMI Study Group. Prospective randomised study of intensive insulin treatment on long
term survival after acute myocardial infarction in patients with diabetes
mellitus. BMJ.1997;314:1512-1515.
29.Meydani SN, Meydani M, Blumberg JB.
et al. Vitamin E supplementation and in vivo immune response in healthy elderly
subjects. A randomized controlled trial. JAMA.1997;277:1380-1386.
30.Perez V, Gilaberte I, Faries D, Alvarez E, Artigas F. Randomised, double-blind, placebo-controlled trial of pindolol in combination
with fluoxetine antidepressant treatment. Lancet.1997;349:1594-1597.
31.Quinn J, Wells G, Sutcliffe T.
et al. A randomized trial comparing octylcyanoacrylate tissue adhesive and
sutures in the management of lacerations. JAMA.1997;277:1527-1530.
32.Schuman P, Capps L, Peng G.
et al. Weekly fluconazole for the prevention of mucosal candidiasis in women
with HIV infection: a randomized, double-blind, placebo-controlled trial. Ann Intern Med.1997;126:689-696.
33.Spruance SL, Rea TL, Thoming C, Tucker R, Saltzman R, Boon R. Penciclovir cream for the treatment of herpes simplex labialis: a randomized,
multicenter, double-blind, placebo-controlled trial. JAMA.1997;277:1374-1379.
34.The IMPACT-II Investigators. Randomised placebo-controlled trial of effect of eptifibatide on complications
of percutaneous coronary intervention: Integrilin to Minimise Platelet Aggregation
and Coronary Thrombosis-II. Lancet.1997;349:1422-1428.
35.The CAPTURE Investigators. Randomised placebo-controlled trial of abciximab before and during
coronary intervention in refractory unstable angina: the CAPTURE Study. Lancet.1997;349:1429-1435.
36.Tingle AJ, Mitchell LA, Grace M.
et al. Randomised double-blind placebo-controlled study on adverse effects
of rubella immunisation in seronegative women. Lancet.1997;349:1277-1281.
37.Wilding P, Clark M, Coon JT.
et al. Effect of long-term treatment with salmeterol on asthma control: a
double blind, randomised crossover study. BMJ.1997;314:1441-1446.
38.Winters JC, Sobel JS, Groenier KH, Arendzen HJ, Meyboom de Jong B. Comparison of physiotherapy, manipulation, and corticosteroid injection
for treating shoulder complaints in general practice: randomised, single blind
study. BMJ.1997;314:1320-1325.
39.Tyson JE, Furzan JA, Reisch JS, Mize SG. An evaluation of the quality of therapeutic studies in perinatal medicine. J Pediatr.1983;102:10-13.
40.Rochon PA, Gurwitz JH, Simms RW.
et al. A study of manufacturer-supported trials of nonsteroidal anti-inflammatory
drugs in the treatment of arthritis. Arch Intern Med.1994;154:157-163.
41.Smith AJ, Goodman NW. The hypertensive response to intubation: do researchers acknowledge
previous work? Can J Anaesth.1997;44:9-13.
42. Intravenous beta-blockade during acute myocardial infarction [editorial]. Lancet.1986;2:79-80.
43.Counsell C, Sandercock P. The efficacy and safety of anticoagulant therapy in patients with assumed
ischaemic stroke. In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update
Software; 1997. Issue 1.
44.Counsell C, Sandercock P. The efficacy and safety of antiplatelet therapy in patients with assumed
ischaemic stroke. In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update
Software; 1997. Issue 1.
45. The Cochrane Library [database on CD-ROM]. Oxford, England: Update Software; 1997.
46.Donne J. Devotion Upon Emergent Occasions . Meditation 17.