[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Peer Review Congress
July 15, 1998

Adherence to Published Standards of ReportingA Comparison of Placebo-Controlled Trials Published in English or German

Author Affiliations

From the Department of Social and Preventive Medicine, University of Berne, Berne, Switzerland.

JAMA. 1998;280(3):247-249. doi:10.1001/jama.280.3.247
Context.—

Context.— Although standards of reporting randomized controlled trials are well established internationally, essential study elements continue to be omitted, which hampers interpretation and systematic review of randomized controlled trials.

Objective.— To identify deficiencies in the quality of reporting of placebo-controlled randomized trials published in German or English.

Design.— Observational study comparing 32 German- and 89 English-language reports of placebo-controlled trials with parallel design, published by the same group of authors between 1985 and 1994.

Main Outcome Measure.— High reporting quality, defined as adherence to published standards and measured by an 18-item scale based on 2 standard guidelines.

Results.— The mean quality score was 8.4 (SD, 3.0; range, 1-16) of 18. The difference of the mean quality scores between English-language reports compared with German-language reports was small (0.27; 95% confidence interval, −0.97 to 1.52). More articles reported clinical aspects than trial methods or statistics.

Conclusion.— There is room for improvement in the reporting of placebo-controlled randomized trials for both English and German reports.

QUALITY of scientific work is judged at several stages from the initiation of research projects to the application of their results in clinical practice. Recent research has greatly contributed to improving quality through the process of peer review in publishing1,2 and the systematic appraisal of trial quality in systematic reviews.37 Within systematic reviews811 and studies assessing publication quality,1217 various checklists and scales have been used. However, missing definitions of "trial quality" have been identified as a basic conceptual problem in most studies.18,19 In addition, in methodologic quality assessment, the quality of reporting should be distinguished from the methodologic quality.5

Defined standards in an accepted form are basic for quality assessment20 and have existed as Uniform Requirements for Manuscripts Submitted to Biomedical Journals for 20 years.21 The Uniform Requirements are now accepted by over 500 journals.22 The objective of the present study was to improve reporting quality by measuring the reporting quality of articles on randomized controlled trials (RCTs) and to identify those aspects of trial reports with the largest deficiencies. At the same time, this study compared the reporting quality of articles written in English or German by the same group of authors.

METHODS
Selection of Reports

All 224 trial reports published from 1985 through 1994 in 5 leading German-language general medical journals were identified in a manual search. Subsequently, 559 reports published in English by the same key authors (first, second, and last authors) during the same 10-year period were identified through MEDLINE. This sample of studies had been used previously to study aspects of language bias23,24 and of trial quality.25

To create a homogeneous group of trials, all 131 reports of placebo-controlled trials with parallel design were selected from these 783 articles. Ten duplicates reporting the same study with the same outcomes were excluded, using a random procedure. This resulted in a final sample of 32 German and 89 English trial reports in 5 and 58 journals, respectively.

Outcome Measures. We defined report quality as "adherence to published standards on trial reporting." A scale for the assessment of reporting quality items was constructed from the checklist for statistical review of articles on clinical trials for BMJ26 and from the third edition of the Uniform Requirements.27 Items were interpreted according to references.13,28,29

We used 14 of the 24 items of the BMJ checklist and 11 of 12 aspects of the Uniform Requirements (Table 1). Items were not used when they were not applicable to the selected sample or when their assessment required clinical specialty knowledge. Three items in the BMJ checklist on blinding and completeness of follow-up reflected trial quality and could not be used in the scale of reporting quality. The 18 items used could be answered with a yes or no. Unclear situations were treated as not meeting the criteria. The quality score was calculated as the sum of met criteria. Data were analyzed with SAS for Windows 6.10, Proc Freq, and Proc Univariate (SAS Institute, Cary, NC).

Reporting Quality Listed by Item and by Language of Publication
Reporting Quality Listed by Item and by Language of Publication
Image description not available.
RESULTS
Reporting Deficiencies

The mean quality score was 8.4 (SD, 3.0; range 1-16). Fifty percent of the 121 reports met 9 or more of the 18 criteria (Table 1). Most frequently, information was missing in articles for confidence interval (96%), method of randomization (93%), reference of the statistical software used (90%), and sample size calculation (89%). However, a sufficient description of the experimental treatment was missing in only 7% of articles. Criteria were met more often for clinical aspects than for trial methodology and statistical analysis.

Differences by Language. The difference of the mean quality score between English- and German-language reports was 0.27 (95% confidence interval, −0.97 to 1.52). Seventeen of the items showed no difference between German-language and English-language reports (P>.1 in χ2 or Fisher exact tests). Side effects were reported in 63% of German and 39% of English reports (P=.04 in χ2 test).

Duplicate Publications. There were 11 sets with 2 to 4 papers in each set concerning the same study, altogether 26 articles reporting on 12 studies (in 1 case, 2 studies were reported in 3 papers). Previous publication was handled and referenced according to standard rules22 in only 4 cases. Four studies had been reported in identical or similar reports and in 1 case 4 papers had increasing numbers of patients. In another case, 2 papers reported on different subgroups of the same study without cross reference.

Three such sets were in English only, 1 in German only, and 7 with articles in both languages. In addition, 5 articles gave a reference to a previous publication or stated that they were preliminary reports, where the counterpart was not in the sample. Six English and 4 German papers were excluded from the analysis because they were identical with another paper.

COMMENT

Using a sample of reports from German-speaking Europe, the present study confirms the existence of large deficiencies in the reporting of placebo-controlled randomized trials with parallel designs. Missing sample size calculations and missing descriptions of the method of randomization are major deficits. These have been reported earlier.8,13,15 The results suggest that it may be easier for investigators to describe clinical features (inclusion criteria and therapeutic regimens) than to report trial methodology and statistical features of their studies. Yet, incomplete reporting of trial methodology impedes reading and systematic reviewing of articles.

No difference between German and English articles was found. The size of the sample would have allowed the discovery of a difference in the scores of 2 with a power of .9 on a 2-sided test at the α level of .05.

There are a number of limitations to the present study. The concept of quality poses some problems. The basic assumption underlying the measurement of quality in a single score is that items of quality are correlated with each other. This can be questioned because 1 fatal error can invalidate the results of otherwise excellent work. However, we are often not aware of the particular excluding criteria. Also, only papers already published were investigated. Within the present study, scoring of report quality was not duplicated and interrater reliability could therefore not be evaluated. External validity was aimed at by using accepted standards. Finally, only articles of authors from German-speaking Europe were assessed. A further study is under way to measure the reliability of the scale used and to compare the present sample with articles from English-language countries.

Standardized assessment of papers submitted for publication can improve the adherence of authors to these standards.14,16 Adherence could be promoted by better dissemination of content and meaning of the standards such as the Uniform Requirements. The Uniform Requirements are also known as "The Vancouver Style," suggesting that the Uniform Requirements only refer to the style of manuscripts, but not to methodological and statistical reporting. MEDLINE references 58 publications of the 5 editions of the Uniform Requirements in 8 languages. A German translation of the fourth edition was published in 1993, but only by an ophthalmology journal.30

The Consolidated Standards of Reporting Trials (CONSORT) statement is an important step toward improving reporting quality of RCTs since it requires that authors themselves complete the checklist on reporting quality.31 It might be recommended that all journals, including those not following the CONSORT statement strictly, request their authors to fill out a checklist on reporting quality.

References
1.
Rennie D. Guarding the guardians: research on editorial peer review.  JAMA.1990;263(theme issue):1317-1441.
2.
Rennie D, Flanagin A. The second international congress on peer review in biomedical publication.  JAMA.1994;272:91.
3.
Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbé KA. Incorporating variations in the quality of individual randomized trials into meta-analysis.  J Clin Epidemiol.1992;45:255-265.
4.
Moher D, Olkin I. Meta-analysis of randomized controlled trials: a concern for standards.  JAMA.1995;274:1962-1964.
5.
Cook DJ, Sackett DL, Spitzer WO. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on meta-analysis.  J Clin Epidemiol.1995;48:167-171.
6.
Mosteller F, Colditz GA. Understanding research synthesis (meta-analysis).  Annu Rev Public Health.1996;17:1-23.
7.
Meade MO, Richardson WS. Selecting and appraising studies for a systematic review.  Ann Intern Med.1997;127:531-537.
8.
Evans M, Pollock AV. Trials on trial: a review of trials of antibiotic prophylaxis.  Arch Surg.1984;119:109-113.
9.
Imperiale TF, McCullough AJ. Do corticosteriods reduce mortality from alcoholic hepatitis? a meta-analysis of the randomized trials.  Ann Intern Med.1990;113:299-307.
10.
Koes BW, Assendelft WJJ, van der Heiden GJMG, Bouter LM, Knipschild PG. Spinal manipulation and mobilisation for back and neck pain: a blinded review.  BMJ.1991;303:1298-1303.
11.
Nurmohamed MT, Rosendaal FR, Büller HR.  et al.  Low-molecular weight heparin versus standard heparin in general and orthopaedic surgery: a meta-analysis.  Lancet.1992;340:152-156.
12.
DerSimonian R, Charette LJ, McPeek B, Mosteller F. Reporting on methods in clinical trials.  N Engl J Med.1982;306:1332-1337.
13.
Pocock SJ. Statistical problems in the reporting of clinical trials.  N Engl J Med.1987;317:426-432.
14.
Gardner MJ, Bond J. An explanatory study of statistical assessment of papers published in the British Medical Journal JAMA.1990;263:1355-1357.
15.
Altman DG, Doré CJ. Randomisation and baseline comparisons in clinical trials.  Lancet.1990;335:149-153.
16.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine Ann Intern Med.1994;121:11-21.
17.
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials.  JAMA.1995;273:408-412.
18.
Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists.  Control Clin Trials.1995;16:62-73.
19.
Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials: current issues and future directions.  Int J Technol Assess Health Care.1996;12:195-208.
20.
Crosby PB. Quality is Free: The Art of Making Quality Certain . New York, NY: McGraw-Hill Book Co; 1979.
21.
International Steering Committee of Medical Editors.  Uniform Requirements for Manuscripts Submitted to Biomedical Journals.  BMJ.1978;1:1334-1336.
22.
International Committee of Medical Journal Editors.  Uniform Requirements for Manuscripts Submitted to Biomedical Journals.  JAMA.1997;277:927-934.
23.
Egger M, Zellweger T, Antes G. Randomised trials in German-language journals [letter].  Lancet.1996;347:1047-1048.
24.
Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German.  Lancet.1997;350:326-329.
25.
Junker C, Egger M, Schneider M, Zellweger T, Antes G. The CONSORT statement [letter].  JAMA.1996;276:1876-1877.
26.
Gardner MJ, Machin D, Campbell MJ. Use of checklists in assessing the statistical content of medical studies.  BMJ.1986;292:810-812. Reprinted in: Gardner MJ, Altman DG, eds. Statistics With Confidence . London, England: BMJ Books; 1989:101-108.
27.
International Committee of Medical Journal Editors.  Uniform Requirements for Manuscripts Submitted to Biomedical Journals.  Ann Intern Med.1988;108:258-265.
28.
Bailar JC, Mosteller F. Guidelines for statistical reporting in articles for medical journals: amplifications and explanations.  Ann Intern Med.1988;108:266-273. Reprinted in: Bailar JC, Mosteller F, eds. Medical Uses of Statistics . Boston, Mass: N Engl J Med Books; 1992.
29.
Altman DG, Gore SM, Gardner MJ, Pocock SJ. Statistical guidelines for contributors to medical journals.  BMJ.1983;286:1489-1493. Reprinted in: Gardner MJ, Altman DG, eds. Statistics With Confidence . London, England: BMJ Books; 1989:83-100.
30.
International Committee of Medical Journal Editors.  Einheitliche Richtlinien für Manuskripte zur Publikation in biomedizinischen Zeitschriften.  Klin Monatsbl Augenheilkd.1993;203(suppl 1):1-6; 1997;211(suppl 3):1-12.
31.
Begg C, Cho M, Eastwood S.  et al.  Improving the quality of reporting of randomized controlled trials: the CONSORT statement.  JAMA.1996;276:637-639.
×