Moher D, Jones A, Lepage L, for the CONSORT Group . Use of the CONSORT Statement and Quality of Reports of Randomized TrialsA Comparative Before-and-After Evaluation. JAMA. 2001;285(15):1992-1995. doi:10.1001/jama.285.15.1992
Author Affiliations: University of Ottawa (Mr Moher) and Thomas C. Chalmers Centre for Systematic Reviews (Mr Moher, Ms Jones, and Dr Lepage), Ottawa, Ontario.
Context The Consolidated Standards for Reporting of Trials (CONSORT) statement
was developed to help improve the quality of reports of randomized controlled
trials (RCTs). To date, a paucity of data exists regarding whether it has
achieved this goal.
Objective To determine whether use of the CONSORT statement is associated with
improvement in the quality of reports of RCTs.
Design and Setting Comparative before-and-after evaluation in which reports of RCTs published
in 1994 (pre-CONSORT) were compared with RCT reports from the same journals
published in 1998 (post-CONSORT). We included 211 reports from BMJ, JAMA, and The Lancet (journals that
adopted CONSORT) as well as The New England Journal of Medicine (a journal that did not adopt CONSORT and was used as a comparator).
Main Outcome Measures Number of CONSORT items included in a report, frequency of unclear reporting
of allocation concealment, and overall trial quality score based on the Jadad
scale, a 5-point quality assessment instrument.
Results Compared with 1994, the number of CONSORT checklist items in reports
of RCTs increased in all 4 journals in 1998, and this increase was statistically
significant for the 3 adopter journals (pre-CONSORT, 23.4; mean change, 3.7;
95% confidence interval [CI], 2.1-5.3). The frequency of unclear reporting
of allocation concealment decreased for each of the 4 journals, and this change
was statistically significant for adopters (pre-CONSORT, 61%; mean change, −22%;
95% CI, −38% to −6%). Similarly, 3 of the 4 journals showed an
improvement in the quality score for reports of RCTs, and this increase was
statistically significant for adopter journals overall (pre-CONSORT, 2.7;
mean change, 0.4; 95% CI, 0.1-0.8).
Conclusion Use of the CONSORT statement is associated with improvements in the
quality of reports of RCTs.
Reports of randomized controlled trials (RCTs) are the "gold standard"
by which health care professionals and others make decisions about treatment
effectiveness. To assess the strengths and limitations of RCTs, readers need
and deserve to know the quality of their methods. Previous studies1,2 indicate that reports of low-quality
RCTs, compared with reports of higher-quality ones, overestimate the effectiveness
of interventions by about 30% across a variety of health care conditions.
The Consolidated Standards for Reporting of Trials (CONSORT) Group developed
the CONSORT statement,3 an evidence-based approach
to help improve the quality of reports of RCTs. Since its publication in 1996,
the CONSORT statement has been widely supported,4- 6
has been translated into several languages, and has an Internet presence (http://www.consort-statement.org) to facilitate awareness and dissemination.
However, there is a paucity of data7 regarding
whether the CONSORT statement has improved the quality of reports of RCTs.
We compared a sample of RCTs published in 1994 (pre-CONSORT) to a sample
published in 1998 (post-CONSORT). To be included, RCT reports had to be drawn
from journals classified by the Science Citation Index
as "general and/or internal medicine." Four journals were evaluated: BMJ, JAMA, The Lancet, and The New England Journal of Medicine (NEJM). Three of these journals (BMJ, JAMA, Lancet) were early adopters of the CONSORT statement,4- 6 whereas the fourth (NEJM) did not formally adopt the CONSORT statement and
was used as a comparator.
Reports were included only if they involved human subjects and if the
study design was identified as an RCT by examining the title and the abstract.
Two readers (A.J., L.L.) independently hand-searched the 4 journals for RCTs
published between January 1 and June 30 of both 1994 and 1998. Hard copies
of relevant articles were obtained but were not masked because evidence concerning
the effect of masking on assessments of trial quality is inconsistent.8,9
Three measures were used to assess the quality of reports of RCTs. First,
the CONSORT checklist was modified so that multiple items were listed separately,
which resulted in 40 items. Each item was assigned a yes or no response depending
on whether the authors had reported it. Second, the reporting of allocation
concealment was assessed as adequate, inadequate, or unclear.2
Third, the Jadad scale,9 which contains 2 questions
for randomization and masking and 1 question evaluating the reporting of withdrawals
and dropouts, was used to assess quality. Each question entails a yes or no
response option. In total, 5 points can be awarded, with higher scores indicating
superior quality. Two reviewers (A.J., L.L.) completed all of these evaluations.
Both reviewers underwent training in evaluating RCTs using the CONSORT
checklist. Before training, the definition of each checklist item was discussed.
To assess interobserver agreement, 5 items from the checklist were purposefully
selected (inclusion criteria, exclusion criteria, point estimate, deviation
from protocol, and general interpretation of study findings). A κ statistic
was calculated for each item based on a randomly selected set of 10 RCTs,
from 1994 and 1998, and these were not included in this study. Discrepancies
were resolved by consensus involving a third party (D.M.). A similar approach
was used to assess interobserver agreement in assessing reporting of allocation
concealment and using the Jadad scale.
Taking time (1994-1998) into consideration and using either unpaired t tests or χ2 tests, we compared the number
of checklist criteria included in each report and the mean number of criteria
included within each subheading specified in the CONSORT checklist. We also
assessed the percentage of studies that reported unclear allocation concealment
and the specific item and overall quality score derived from the Jadad scale.
The number of CONSORT checklist items reported was treated as the dependent
variable in an analysis of variance (ANOVA) with 2 factors: journal and year.
The difference between the adopter journals and the comparator journal was
derived from the fitted ANOVA using a contrast involving journal (ie, adopters
vs comparator) and year (ie, 1998 vs 1994). Least-square estimates and 95%
confidence intervals (CIs) also were computed; for significant level, P = .05 (2-sided).
Of the 221 RCTs identified, 211 met the inclusion criteria. Six studies
were excluded because they were animal studies and 4 studies because they
were quasi-randomized trials.
Substantial agreement was established for 4 items from the CONSORT checklist
and the Jadad instrument (inclusion and exclusion criteria, point estimate
and general interpretation [κ = 1.0], quality assessment, and overall
Jadad score [κ = 0.74]). Moderate agreement was established for allocation
concealment (κ = 0.53) and for 1 item from the CONSORT checklist (κ
= 0.54, deviation from protocol).
There was an increase over time in the number of CONSORT checklist items
included in the reports of RCTs in all 4 journals (Table 1). This increase was statistically significant for 2 individual
journals and overall for adopter journals (pre-CONSORT, 23.4; mean change,
3.7; 95% CI, 2.1-5.3). Over time, the increase in the reporting of CONSORT
items was significantly greater for adopter journals when evaluated against
the comparator journal (mean difference, 3.8; 95% CI, 1.0-6.5; 2-sided P = .007).
The proportion of RCTs with unclear reporting of allocation concealment
decreased over time in all 4 journals (Table 2) and was statistically significant for adopter journals
(pre-CONSORT, 61%; mean change, −22%; 95% CI, −38% to −6%).
Over time, 3 of the 4 journals improved the quality of reports of RCTs as
assessed by the Jadad scale (Table 2),
which was statistically significant for 1 journal (Lancet) and across the adopter journals (pre-CONSORT, 2.7; mean change, 0.4;
95% CI, 0.1-0.8).
The quality of reports of RCTs in all 4 journals included in this study
improved over time. This improvement appears to be greater for the journals
that adopted CONSORT. However, because of our study design, it is only possible
to suggest that the improvement may be associated with the implementation
of the CONSORT statement.
These results also suggest that aspects of reporting of RCTs still require
improvement. For example, the reporting of bias reduction methods, such as
masking, is less than optimal. Similarly, our results confirm a concern raised
by others10,11 regarding how the
discussion/comment sections of RCTs are reported. It is unclear whether these
deficiencies reflect difficulties in using CONSORT experienced by authors,
by journals, or by both. To help address these questions, it will be important
to obtain data from editors and authors as well as data on the readability
of CONSORT reports as a way to gauge their scientific content.
We used 1 journal with a high citation impact factor as the comparator.
This approach offers some control over more obvious forms of bias, such as
the passage of time, and is considered to be a stronger research design than
having no comparator.12 Ideally, we would have
liked to include more comparator journals, but we were unable to identify
them. Because of this, we limited our analysis in all cases except 1 to a
comparison over time rather than between adopter and nonadopter journals.
To strengthen these findings, we recommend that this evaluation be replicated
and expanded to include more nonadopter journals. To facilitate such a study,
we encourage all journals to indicate, perhaps in their information for authors,
whether they support the CONSORT statement. Moreover, to increase generalizability
of these results, future evaluations should also include specialty journals.13
Another limitation of our study is the time frame in which we completed
the evaluation. We chose reports of RCTs published during the first half of
1998, only 12 to 18 months after the endorsement of CONSORT by journals included
in this evaluation. It is possible, even likely, that effective dissemination
is a slow process and that to estimate the true influence of CONSORT requires
more time. In addition, our results pertain to the CONSORT checklist and do
not evaluate the use of the CONSORT flow diagram. There are limitations to
the present version of the flow diagram14 in
terms of information requested of authors. Additionally, interpretation of
the terms used and the sensibility of some of the criteria across RCTs has
been inconsistent. Egger and colleagues15 report
the results of an evaluation of the CONSORT statement flow diagram in an accompanying
In summary, these findings suggest that use of the CONSORT checklist
may be associated with improving the quality of reports of RCTs. Higher-quality
reports are likely to improve RCT interpretation, minimize biased conclusions,
and ultimately facilitate decision making about treatment effectiveness.