From the Department of Social and Preventive Medicine, University of Berne, Berne, Switzerland.
Context.— Although standards of reporting randomized controlled trials are well
established internationally, essential study elements continue to be omitted,
which hampers interpretation and systematic review of randomized controlled
Objective.— To identify deficiencies in the quality of reporting of placebo-controlled
randomized trials published in German or English.
Design.— Observational study comparing 32 German- and 89 English-language reports
of placebo-controlled trials with parallel design, published by the same group
of authors between 1985 and 1994.
Main Outcome Measure.— High reporting quality, defined as adherence to published standards
and measured by an 18-item scale based on 2 standard guidelines.
Results.— The mean quality score was 8.4 (SD, 3.0; range, 1-16) of 18. The difference
of the mean quality scores between English-language reports compared with
German-language reports was small (0.27; 95% confidence interval, −0.97
to 1.52). More articles reported clinical aspects than trial methods or statistics.
Conclusion.— There is room for improvement in the reporting of placebo-controlled
randomized trials for both English and German reports.
QUALITY of scientific work is judged at several stages from the initiation
of research projects to the application of their results in clinical practice.
Recent research has greatly contributed to improving quality through the process
of peer review in publishing1,2
and the systematic appraisal of trial quality in systematic reviews.3- 7
Within systematic reviews8- 11
and studies assessing publication quality,12- 17
various checklists and scales have been used. However, missing definitions
of "trial quality" have been identified as a basic conceptual problem in most
studies.18,19 In addition, in
methodologic quality assessment, the quality of reporting should be distinguished
from the methodologic quality.5
Defined standards in an accepted form are basic for quality assessment20 and have existed as Uniform Requirements for Manuscripts
Submitted to Biomedical Journals for 20 years.21
The Uniform Requirements are now accepted by over 500 journals.22
The objective of the present study was to improve reporting quality by measuring
the reporting quality of articles on randomized controlled trials (RCTs) and
to identify those aspects of trial reports with the largest deficiencies.
At the same time, this study compared the reporting quality of articles written
in English or German by the same group of authors.
All 224 trial reports published from 1985 through 1994 in 5 leading
German-language general medical journals were identified in a manual search.
Subsequently, 559 reports published in English by the same key authors (first,
second, and last authors) during the same 10-year period were identified through
MEDLINE. This sample of studies had been used previously to study aspects
of language bias23,24 and of trial
To create a homogeneous group of trials, all 131 reports of placebo-controlled
trials with parallel design were selected from these 783 articles. Ten duplicates
reporting the same study with the same outcomes were excluded, using a random
procedure. This resulted in a final sample of 32 German and 89 English trial
reports in 5 and 58 journals, respectively.
Outcome Measures. We defined report quality as "adherence to published standards on trial
reporting." A scale for the assessment of reporting quality items was constructed
from the checklist for statistical review of articles on clinical trials for BMJ26 and from the third
edition of the Uniform Requirements.27 Items
were interpreted according to references.13,28,29
We used 14 of the 24 items of the BMJ checklist
and 11 of 12 aspects of the Uniform Requirements (Table 1). Items were not used when they were not applicable to the
selected sample or when their assessment required clinical specialty knowledge.
Three items in the BMJ checklist on blinding and
completeness of follow-up reflected trial quality and could not be used in
the scale of reporting quality. The 18 items used could be answered with a
yes or no. Unclear situations were treated as not meeting the criteria. The
quality score was calculated as the sum of met criteria. Data were analyzed
with SAS for Windows 6.10, Proc Freq, and Proc Univariate (SAS Institute,
The mean quality score was 8.4 (SD, 3.0; range 1-16). Fifty percent
of the 121 reports met 9 or more of the 18 criteria (Table 1). Most frequently, information was missing in articles for
confidence interval (96%), method of randomization (93%), reference of the
statistical software used (90%), and sample size calculation (89%). However,
a sufficient description of the experimental treatment was missing in only
7% of articles. Criteria were met more often for clinical aspects than for
trial methodology and statistical analysis.
Differences by Language. The difference of the mean quality score between English- and German-language
reports was 0.27 (95% confidence interval, −0.97 to 1.52). Seventeen
of the items showed no difference between German-language and English-language
reports (P>.1 in χ2 or Fisher exact
tests). Side effects were reported in 63% of German and 39% of English reports
(P=.04 in χ2 test).
Duplicate Publications. There were 11 sets with 2 to 4 papers in each set concerning the same
study, altogether 26 articles reporting on 12 studies (in 1 case, 2 studies
were reported in 3 papers). Previous publication was handled and referenced
according to standard rules22 in only 4 cases.
Four studies had been reported in identical or similar reports and in 1 case
4 papers had increasing numbers of patients. In another case, 2 papers reported
on different subgroups of the same study without cross reference.
Three such sets were in English only, 1 in German only, and 7 with articles
in both languages. In addition, 5 articles gave a reference to a previous
publication or stated that they were preliminary reports, where the counterpart
was not in the sample. Six English and 4 German papers were excluded from
the analysis because they were identical with another paper.
Using a sample of reports from German-speaking Europe, the present study
confirms the existence of large deficiencies in the reporting of placebo-controlled
randomized trials with parallel designs. Missing sample size calculations
and missing descriptions of the method of randomization are major deficits.
These have been reported earlier.8,13,15
The results suggest that it may be easier for investigators to describe clinical
features (inclusion criteria and therapeutic regimens) than to report trial
methodology and statistical features of their studies. Yet, incomplete reporting
of trial methodology impedes reading and systematic reviewing of articles.
No difference between German and English articles was found. The size
of the sample would have allowed the discovery of a difference in the scores
of 2 with a power of .9 on a 2-sided test at the α level of .05.
There are a number of limitations to the present study. The concept
of quality poses some problems. The basic assumption underlying the measurement
of quality in a single score is that items of quality are correlated with
each other. This can be questioned because 1 fatal error can invalidate the
results of otherwise excellent work. However, we are often not aware of the
particular excluding criteria. Also, only papers already published were investigated.
Within the present study, scoring of report quality was not duplicated and
interrater reliability could therefore not be evaluated. External validity
was aimed at by using accepted standards. Finally, only articles of authors
from German-speaking Europe were assessed. A further study is under way to
measure the reliability of the scale used and to compare the present sample
with articles from English-language countries.
Standardized assessment of papers submitted for publication can improve
the adherence of authors to these standards.14,16
Adherence could be promoted by better dissemination of content and meaning
of the standards such as the Uniform Requirements. The Uniform Requirements
are also known as "The Vancouver Style," suggesting that the Uniform Requirements
only refer to the style of manuscripts, but not to methodological and statistical
reporting. MEDLINE references 58 publications of the 5 editions of the Uniform
Requirements in 8 languages. A German translation of the fourth edition was
published in 1993, but only by an ophthalmology journal.30
The Consolidated Standards of Reporting Trials (CONSORT) statement is
an important step toward improving reporting quality of RCTs since it requires
that authors themselves complete the checklist on reporting quality.31 It might be recommended that all journals, including
those not following the CONSORT statement strictly, request their authors
to fill out a checklist on reporting quality.
Junker CA. Adherence to Published Standards of ReportingA Comparison of Placebo-Controlled Trials Published in English or
German. JAMA. 1998;280(3):247–249. doi:10.1001/jama.280.3.247