Context.— Structured abstracts, that is, abstracts that describe a study using
requisite content headings, provide more informative content. Concomitant
reporting in the text of the report might improve with structured abstract
use because of increased awareness by authors or editors of important study
areas associated with content headings.
Objective.— To assess whether structured abstract use is associated with improved
reporting of randomized clinical trials.
Design and Setting.— Survey of trial reports published the year preceding, of, and following
new use of structured abstracts, found by hand searching Archives of Ophthalmology (1992-1994) and Ophthalmology (1991-1993), as well as trial reports published concurrently without
change in abstract format (American Journal of Ophthalmology, 1991-1994).
Main Outcome Measures.— We measured the inclusion of 56 criteria derived from Consolidated Standards
of Reporting Trials (CONSORT) descriptors (JAMA 1996;276:637-639) in the
text of each report and calculated the number of criteria included per report
and the proportion of reports including individual criteria. Reports with
structured abstracts were compared with those without, and reports published
in 1993 and 1994 in the American Journal of Ophthalmology were compared with those published in 1991 and 1992.
Results.— The mean (SEM) number of criteria included by authors was 15.8 (0.4)
per report in 125 trial reports. We found no difference in the mean number
of criteria included or the proportion of reports that included specific criteria
by journal. Following structured abstract use, there was no difference in
either the mean number of criteria per report or the proportion of reports
including a majority of criteria within each CONSORT subheading. Four criteria
were included more often and 2 less often following structured abstract use
in individual journals.
Conclusion.— Using CONSORT descriptor criteria to evaluate reporting quality, we
found no difference in text reporting associated with structured abstract
use in the journals examined.
STRUCTURED ABSTRACTS, that is, abstracts that describe a study using
specified content headings rather that paragraph format, were suggested by
the Ad Hoc Working Group for Critical Appraisal of the Medical Literature
in 19871 "to provide more information . . .
for articles reporting original research . . . of medical care."2
The original objectives for using structured abstracts were to help health
professionals quickly assess the reliability and content of a clinical report,
to facilitate peer review, and to aid accurate indexing and retrieval of reports
from computerized databases such as MEDLINE and EMBASE. In structured abstracts
essential elements are concisely reported using content headings (eg, objective,
design, setting, participants, intervention, results, conclusions).1 Rapid adoption of structured abstracts by journals
resulted in an annual doubling of reports with structured abstracts published
from 1989 through 1991 appearing in MEDLINE; 15% of these were reports of
clinical trials.3
Froom and Froom4 evaluated the quality
of structured abstracts and found important deficiencies in reporting of patient
demographics, follow-up, and statistical methods. Taddio et al5
compared structured with unstructured abstracts and found an improvement in
reporting using the structured format, albeit with similar deficiencies. They
could not rule out that improvements were due to changes over time. Harcourt
et al3 reported that indexers at the National
Library of Medicine applied an average of 3 more Medical Subject Headings
to reports with structured abstracts compared with concurrent unstructured
abstracts, but could not determine whether this finding was related to better
abstract or text reporting or to differential indexing of the journals that
first incorporated structured abstracts.
To our knowledge, no investigator has directly examined whether structured
abstract use affects the reporting quality of report text. In writing about
structured abstract merits, Rennie and Glass6
raise the possibility that "[abstract] structure reminds authors . . . of
the necessity of providing each category of information." By focusing on the
content headings required by a structured abstract, authors or editors might
incorporate descriptors more consistently in the manuscript text. Our objective
was to assess whether the use of structured abstracts is associated with an
improvement in the overall reporting of ophthalmology randomized controlled
trials (RCTs).
We selected 2 US ophthalmology journals that we previously found to
regularly report large numbers of RCTs7 and
that recently revised their Instructions to Authors to require structured
abstracts, the Archives of Ophthalmology and Ophthalmology.8,9
We manually searched for reports describing RCTs the year preceding, of, and
following the first appearance of structured abstracts (1992, 1993, and 1994,
respectively, in the Archives of Ophthalmology and
1991, 1992, and 1993, respectively, in Ophthalmology).
To monitor changes over time, we searched the American Journal
of Ophthalmology (a US ophthalmology journal that publishes similar
numbers of RCTs7 and that did not require structured
abstracts until late 1994)10 for RCT reports
from 1991 through 1994.
Hand searching was carried out by 2 independent, trained readers who
examined each full-length report for RCT status in the selected journals for
the specified years.11 We defined RCT as a
controlled experiment designed to evaluate an intervention or diagnostic tool,
using a random method to assign individuals, eyes, or some other unit to a
test or comparison group. We included quasi-randomized clinical trials, ie,
those employing a method of assignment (eg, alternation) designed to avoid
bias. Reports of RCTs that did not include data by randomized treatment group
(eg, validation of a method used for measuring an outcome) were excluded.
We found 154 reports of RCTs. Five papers were excluded; 1 had no abstract,
2 presented data on subsets of patients, and 2 examined methods to measure
outcomes. We also excluded 24 reports because the authors stated that an abbreviated
methodological description was provided since the methods had previously been
described.
Extraction of RCT Design and Operational Characteristics. The Consolidated Standards of Reporting Trials (CONSORT) statement describes
RCT descriptors and provides a flowchart showing patient entry and follow-up
that are recommended for inclusion in every RCT report.12
Using CONSORT descriptors as a "gold standard" to evaluate reporting quality
in each article, we scored the presence of 56 criteria (each flowchart block
or descriptor, Table 1) in individual
reports as yes, no, or not applicable. We selected 9 criteria (with daggers
in Table 1) corresponding to common
abstract content headings to measure inclusion in each abstract.
Table 1.—CONSORT Descriptor Criteria*
Because some study- or journal-specific characteristics could influence
the inclusion of CONSORT statement descriptors, we extracted information by
journal about study and report characteristics, including purpose of intervention,
multicenter status, type of test intervention, sample size, group or individual
authorship, length of report (number of pages), and length of methods section
(number of pages).
Analyses. Data were entered into a database (Paradox, Version 4.0, Borland International
Inc, Scotts Valley, Calif), and exported to a statistical program (SAS, Version
6.2, SAS Institute Inc, Cary, NC). We calculated the number of criteria included
in each report, the proportion of reports that included specific criteria,
and the number of reports that included more than the majority of criteria
within each CONSORT subheading (introduction, protocol, etc). We compared
structured with unstructured abstracts, and reports published in 1991 and
1992 with those published in 1993 and 1994 (American Journal
of Ophthalmology), using the Student t test
or χ2 tests. Odds ratios (ORs) with 95% confidence intervals
(CIs) are presented.
Description of RCTs by Journal
We found that RCTs reported in the Archives of Ophthalmology, compared with Ophthalmology and the American Journal of Ophthalmology, were more often multicentered
(10 [38%] of 26 vs 12 [24%] of 51 and 13 [27%] of 48, respectively; P=.07), evaluated surgical or laser trials more often (7
[27%] of 26 vs 6 [12%] of 51 and 3 [6%] of 48, respectively; P=.001), more frequently devoted more than a single page to the methods
section (10 [38%] of 26 vs 9 [18%] of 51 and 12 [25%] of 48, respectively; P=.005), and published more reports with study group authorship
(5 [19%] of 26 vs 2 [4%] of 51 and 3 [6%] of 48, respectively; P=.001). Thus, we examined all results separately by journal.
Reporting of Descriptors in Text by Journal.
Reporting of CONSORT criteria in the text was unimpressive. The mean
(SEM) number of criteria included was 15.8 (0.4) of a possible 56; there was
little difference among journals (Table
1). Journals were also remarkably similar in the proportion of reports
that included specific criteria (Figure 1). Criteria reported in a low proportion of reports in all 3 journals
were often associated with CONSORT subheadings associated with RCT methods,
such as assignment and masking.
Comparison by Structured Abstract Use or Over Time. We found no difference in the mean number of criteria that were included
in reports with structured abstracts compared with those without (Table 2). Including a majority of criteria
within a single CONSORT subheading was positively associated with structured
abstracts for "protocol" in the Archives of Ophthalmology (OR, 2.16; 95% CI, 1.12-4.16), negatively associated for "introduction"
in Ophthalmology (OR, 0.57; 95% CI, 0.33-0.95), and
not associated with later year of publication. Thus, we found no evidence
for improvement in inclusion of criteria associated with structured abstracts.
Table 2.—Criteria Included in Text by Journal and Structured Abstract Use or Year of Publication
We then calculated the proportion of reports in which a specific criterion
was included to see if there were improved reporting by criterion associated
with structured abstract use or year of publication. Individual criteria were
included more or less often following structured abstract use or with later
publication years, but by individual journal (Table 3).
Table 3.—Criteria Included in a Significantly Larger or Smaller Proportion of Reports Associated With Structured Abstracts or Later Year of Publication
Reporting of Descriptors in Abstract of Report. Of the 9 criteria used to evaluate abstract reporting, a mean of 5.0
(0.2) were included in all abstracts from all journals. Structured abstracts
were more often associated with inclusion of criteria in the Archives of Ophthalmology (Table
2).
Specific criteria included infrequently in structured abstracts for
the Archives of Ophthalmology, Ophthalmology, and American Journal of Ophthalmology were description study population (6 [67%] of 9, 18 [57%] of 28, and
16 [64%] of 25, respectively); primary outcome (6 [67%] of 9, 8 [29%] of 28,
and 12 [48%] of 25, respectively); and number of patients followed up (2 [22%]
of 9, 5 [18%] of 28, and 3 [12%] of 25, respectively).
Our results do not support an association of improved text reporting
with structured abstracts or later publication in the journals examined. These
3 journals tended to be more alike than different in overall reporting of
individual criteria, and there was no consistent pattern of change in reporting
associated with structured abstract use or later publication. Possibly, the
time period we examined was transitional or represented a lag time while editors
or authors were incorporating use of structured abstracts. Haynes et al2 reported that initial use of structured abstracts
found some authors writing abstracts concurrently with manuscript preparation,
whereas others did so only at submission or on editor request. Also, no special
emphasis was placed on use of structured abstracts for RCTs initially, even
with recognized importance for trial reports. Finally, it is possible that
our sample size was insufficient to detect subtle changes in reporting, as
our results are based on 125 reports from 3 ophthalmology journals; they may
not be generalizable to other journals or areas of medicine.
Although we did not detect improvement in overall text reporting, we
thought there might be individual criteria reported more frequently when structured
abstracts were used, but found no consistent reporting pattern. For example,
use of random or trial in
the title and rationale for statistical tests were reported more often, but
in separate journals. Some criteria were reported less frequently with structured
abstracts, and perhaps were viewed as less important when space constraints
limited text length. However, any changes we report in inclusion of criteria
associated with structured abstract use or later year of publication may be
due to chance, given the number of observations.
Although checklists for assessing reporting quality of RCTs were available,13-15 we chose to use CONSORT
descriptors as a "gold standard" since it comprises a comprehensive list of
criteria. We did not intend to evaluate trial quality. It has been argued,
however, that "a well-designed but poorly reported trial could be judged as
having low quality,"16 so assessing reporting
is an important first step in assessing trial quality.
Our initial search yielded 24 RCT reports that included abbreviated
methodological descriptions because methods had previously been reported.
Editors are faced with a tension between space limitations and inclusion of
all CONSORT descriptors in subsequent RCT reports. Since readers may not have
previous reports available, we believe each RCT report should include all
CONSORT descriptors to allow independent report evaluation.
Finally, we found a significant improvement in abstract reporting quality
when structured abstracts were used in the Archives of Ophthalmology, and some improvement in Ophthalmology. Consistent
with findings of others,4,5 we
found abstract reporting deficiencies with authors frequently omitting a description
of study population, primary outcome, or number of patients followed up.
In summary, we found no improvement in text reporting when structured
abstracts were used. Nevertheless, structured abstract use should not be abandoned
since abstract reporting itself is improved using this format. Rather, our
results highlight the need for a standard such as the CONSORT statement to
enhance RCT text reporting.
1.Ad Hoc Working Group for Critical Appraisal of the Medical Literature. A proposal for more informative abstracts of clinical articles.
Ann Intern Med.1987;106:595-604.Google Scholar 2.Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ. More informative abstracts revisited.
Ann Intern Med.1990;113:69-76.Google Scholar 3.Harcourt AM, Knecht LS, Humphreys BL. Structured abstracts in MEDLINE, 1989-1991.
Bull Med Libr Assoc.1995;83:190-195.Google Scholar 4.Froom P, Froom J. Deficiencies in structured medical abstracts.
J Clin Epidemiol.1993;46:591-594.Google Scholar 5.Taddio A, Pain T, Fassos FF, Boon H, Ilersich AL, Elnarson TR. Quality of nonstructured and structured abstracts of original research
articles in the
British Medical Journal, the
Canadian Medical Association Journal and the
Journal of the American Medical Association.
Can Med Assoc J.1994;150:1611-1615.Google Scholar 6.Rennie D, Glass RM. Structuring abstracts to make them more informative.
JAMA.1991;266:116-117.Google Scholar 7.Dickersin K, Scherer RW, Lefebvre C. Identification of relevant studies for systematic reviews.
BMJ.1994;309:1286-1291.Google Scholar 8.Goldberg MF. Changes in the
Archives.
Arch Ophthalmol.1993;111:39-40.Google Scholar 9.Lichter PR. Structured abstract now required for all submissions to the journal.
Ophthalmology.1991;98:1611-1612.Google Scholar 10.Straastsma BR. Information for authors and benefits to readers.
Am J Ophthalmol.1994;117:104-105.Google Scholar 12.Begg C, Eastwood S, Horton R.
et al. Improving the quality of reporting of randomized controlled trials.
JAMA.1996;276:637-639.Google Scholar 13.Mahon WA, Daniel EE. A method for the assessment of reports of drug trials.
Can Med Assoc J.1964;90:565-569.Google Scholar 14.DerSimonian R, Charette LJ, McPeek B, Mosteller F. Reporting on methods in clinical trials.
N Engl J Med.1982;306:1332-1337.Google Scholar 15.Grant A. Reporting clinical trials.
Br J Obstet Gynaecol.1989;96:397-400.Google Scholar 16.Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials: current issues
and future directions.
Int J Technol Assess Health Care.1996;12:195-208.Google Scholar