Context The section of a research article most likely to be
read is the abstract, and therefore it is particularly important that
the abstract reflect the article faithfully.
Objective To assess abstracts accompanying research articles
published in 6 medical journals with respect to whether data in the
abstract could be verified in the article itself.
Design Analysis of simple random samples of 44 articles and
their accompanying abstracts published during 1 year (July 1, 1996-June
30, 1997) in each of 5 major general medical journals (Annals of
Internal Medicine, BMJ, JAMA, Lancet,
and New England Journal of Medicine) and a consecutive sample
of 44 articles published during 15 months (July 1, 1996-August 15,
1997) in the CMAJ.
Main Outcome Measure Abstracts were considered deficient if they
contained data that were either inconsistent with corresponding data in
the article's body (including tables and figures) or not found in the
body at all.
Results The proportion of deficient abstracts varied widely
(18%-68%) and to a statistically significant degree
(P<.001) among the 6 journals studied.
Conclusions Data in the abstract that are inconsistent with or
absent from the article's body are common, even in large-circulation
general medical journals.
The
abstract accompanying a research article, because it is often the only
part of the article that will be read, should reflect fully and
accurately the work reported. We observed in 1 medical specialty
journal that a quarter or more of manuscripts returned after revision
contained data in the abstract that could not be verified in the body
of the paper.1 If this problem were to persist in published
articles, then a potential for misinterpretation would exist. In the
present study, we surveyed research articles and their accompanying
abstracts published recently in 6 medical journals to verify data in
the abstract by relating them to corresponding data in the body of the
report.
Articles studied included simple random samples of reports of
original research (including meta-analyses but not other types of
reviews) appearing in 5 medical journals between July 1, 1996, and June
30, 1997 (Annals of Internal Medicine, BMJ,
JAMA, Lancet, and New England Journal of
Medicine); all articles appearing in a sixth journal CMAJ
(Canadian Medical Association Journal), between July 1, 1996,
and August 15, 1997, were also studied. Additional inclusion criteria
were (1) the article was accompanied by an abstract and (2) the article
occupied at least 2 full journal pages.
To estimate the sample sizes, we used some preliminary
observations1 that 25% to 50% of articles published in 2
of the journals studied contained 1 or more deficiencies in abstracts.
We assumed this rate would range from 10% to 40% across the 6
journals studied and that α was .05 and power was 0.8, yielding a
projected sample size of 44 from each journal. From each of the 5
journals that published more than 44 research articles in the 2 volumes
studied (July 1, 1996-June 30, 1997), we selected a computer-generated
simple random sample of 44. From the CMAJ, we analyzed a
consecutive cohort of all 44 articles published from July 1, 1996,
through August 15, 1997.
For each selected article, the abstract was scrutinized by 1 of 3
examiners who identified each datum or other piece of information in
the abstract and then sought to relate it to its source in the body of
the article, including tables and figures. Two types of discrepancies
were sought: (1) data given differently in the abstract and the body
and (2) data given in the abstract but not in the body. If either was
identified, the abstract was considered deficient. Discrepancies
attributable to rounding were not considered to be deficiencies as long
as the rounding was done appropriately, and the rounded value appeared
in the abstract and the more detailed value in the body.
The proportions of articles containing deficiencies were compared
across journals by χ2 analysis. On the basis of normal
approximation, 95% confidence intervals (CIs) were calculated for each
proportion. We also performed a validation study by randomly selecting
(using another computer-generated random number sequence) 7 of each set
of 44 articles and having these examined by a second (and different)
examiner.
Table 1 contains the proportions of
deficient abstracts and 95% CIs for each journal, tabulated
considering the abstract as the unit, as well as the types of
deficiencies found in the 6 journals. The proportion of deficient
abstracts ranged from a low of 18% to a high of 68%. Inconsistency
between abstract and body was generally more common than omitted data
(ie, data in the abstract not found in the body). A substantial
proportion of deficient abstracts contained both kinds of defects
(25/104; 24%).
In the validation study, 38 of the 42 paired comparisons were
concordant with respect to identification of deficiencies. The κ
value for agreement between the 2 evaluators was 0.81 (z =
5.22; P<.001).
The frequency with which we found abstracts to be inaccurate, in the
sense of containing information not verifiable in the article's main
body (including tables and figures, as well as text) was surprisingly
large, ranging from 18% to 68% in the 6 journals surveyed. The more
common type of the 2 deficiencies was inconsistency between data in the
abstract and those in the body. Giving data or other information in the
abstract but not in the body was somewhat less common. These findings
are all the more surprising considering that the journals studied are
all prominent and highly regarded general medical publications whose
editors were founding members of the International Committee of
Medical Journal Editors, a respected standard-setting body. These
journals have full-time professional staffs who can be presumed to
devote a good deal of time and energy to editorial and production
processes.
Many of the discrepancies identified were quite minor and not likely to
cause serious misinterpretation. For example, 1 abstract2
reported the population to consist of "42 consecutive patients,"
whereas the body indicated it to be "44 consecutive patients of which
42 agreed to participate." Sometimes, however, discrepancies were
more serious; for example, 1 abstract3 gave the estimated
15-year survival as 48%, whereas the body of the text indicated it to
be 58%.
The specific question we asked in this study—Can the data and
other information in the abstract be verified in the body of the
article?—does not seem to have been examined before. Previous
studies4,5 of abstract quality generally involved overall
or global assessment. Most of the recent literature on abstracts has
concerned structured abstracts, introduced in 19876 with
the goal of making abstracts more informative. Several
investigations7-9 indicated that structured abstracts are
actually better in quality, more informative, more readable, and a more
efficient use of readers' time. Structured abstracts may well offer
all of these advantages, but there is little reason to expect them to
reduce the types of deficiencies assessed in this study. Indeed, if
structured abstracts are more informative (ie, if they provide more
information), they might be more likely to be subject to deficiencies
we assessed. In the present study, we could not discern any
relationship between various structured formats and the deficiencies
assessed.
It is important to acknowledge that we addressed only 1 aspect of
abstract accuracy in asking if what is in the abstract is consistent
with the body of the article. There is another, at least equally
important question: Is the important information in the article found
in the abstract? Our study was not designed to address this question.
We found previously1 that providing authors with specific
instructions about abstract accuracy when they are revising manuscripts
is ineffective in preventing the types of defects assessed in this
study. If it is important that abstracts be as accurate as
possible—and it can hardly be argued otherwise—and if authors cannot
be counted on to provide this level of accuracy, the responsibility
must be taken by journals' editorial staffs. As part of the
copyediting process, the abstract needs to be scrutinized painstakingly
on a line-by-line or even word-by-word basis and each bit of
information verified individually and specifically.
1.Pitkin RM, Branagan MA. Can the accuracy of
abstracts be improved by providing specific instructions? a randomized
controlled trial.
JAMA.1998;280:267-269.Google Scholar 2.Rothwell PM, McDowell Z, Wong CK, Dorman PJ. Doctors and
patients don't agree: cross-sectional study of patients' and
doctors' perceptions and assessments of disability in multiple
sclerosis.
BMJ.1997;314:1580-1583.Google Scholar 3.Socié G, Mary J-Y, de Gramont A.
et al. , for the
French Society of Haematology. Paroxysmal nocturnal haemoglobinuria:
long-term follow-up and prognostic factors.
Lancet.1996;348:573-577.Google Scholar 4.Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at
Annals of Internal Medicine.
Ann Intern Med.1994;121:11-21.Google Scholar 5.Froom P, Froom J. Deficiencies in structured medical
abstracts.
J Clin Epidemiol.1993;46:591-594.Google Scholar 6.Ad Hoc Working Group for Critical Appraisal of the
Medical Literature. A proposal for more informative abstracts of
clinical articles.
Ann Intern Med.1987;106:598-604.Google Scholar 7.Comans ML, Overbeke AJ. The structural summary: a tool
for reader and author.
Ned Tijdschr Geneeskd.1990;134:2338-2343.Google Scholar 8.Narine L, Yee DX, Einarson TR, Ilersich AL. The quality
of journal abstracts for research articles.
CMAJ.1991;144:449-453.Google Scholar 9.Taddio A, Pain T, Fassos FF, Boon H, Ilersich AL, Einarson TR. Quality of nonstructured and structured abstracts of
original research articles in the
British Medical Journal, the
Canadian Medical Association Journal and the
Journal of
the American Medical Association.
CMAJ.1994;150:1611-1615.Google Scholar