Reporting of Effect Direction and Size in Abstracts of Systematic Reviews

To the Editor: Clinicians commonly misinterpret systematic review abstracts: a recent study showed many arrived at incorrect conclusions, and only 62% correctly identified the direction of the main effect.¹

Interpreting numerical results requires statistical knowledge that many clinicians lack. To ensure correct interpretation, abstracts should give the direction and size of effects both in words and numerically. Because systematic reviews are important and widely used summaries of primary research, we decided to examine a sample of systematic review abstracts to assess the nature and extent of any deficiencies in reporting.

The systematic reviews selected were all new reviews of interventions published in issue 4, 2009, of the Cochrane Library² and from a search (based on Moher et al³) of the National Library of Medicine's 119 Core Clinical Journals (Abridged Index Medicus)⁴ during 2009. Reviews were eligible if they included 1 or more meta-analyses comparing interventions. Eligibility was broad, as there is no widely agreed definition of a systematic review. We used a definition by Moher et al³: “ . . . the authors' stated objective was to summarize evidence from multiple studies, and the article described explicit methods, regardless of the details provided.”

One author (E.M.B.) screened all titles and abstracts with a second author (S.H.) independently checking citations classified as possible systematic reviews based on full-text review. All Cochrane reviews and a random sample of eligible non-Cochrane reviews were selected for data extraction.

From abstracts, 2 authors (E.M.B., P.P.G.) independently extracted the description in words of the direction and size of effect, statistical significance, numerical estimates of effect size (relative and absolute), P value, and confidence interval.

We included 64 Cochrane and 125 of 275 non-Cochrane systematic reviews (Figure); 7 of the non-Cochrane reviews did not include a meta-analysis, leaving 182 abstracts.

Of 182 abstracts, 77 (42%) did not describe the direction of intervention effect in words (Figure), and only 22 (12%) described the size in words. In 34 (19%), the direction of effect could only be determined by interpreting numerical results.

In 43 (24%) of the abstracts, we could not reliably determine the direction of effect (Figure). Sometimes a risk or odds ratio was stated, and if it was less than 1, the direction might be assumed to favor the intervention (for example, “The pooled relative risk was 0.62 [95% CI, 0.45 to 0.86] . . . ”). However, we classified these instances as ambiguous.

Although all included reviews contained a meta-analysis, 45 (25%) gave no numerical effect measure and 44 (24%) gave no measure of uncertainty (Table).

Besides Cochrane reviews less frequently providing P values (6% vs 28%), reporting of effects in Cochrane and non-Cochrane reviews were similar.

For 42% of abstracts of systematic reviews, the direction of the main effect either could not be determined or needed to be inferred. Statistical uncertainty was also poorly reported: 24% of abstracts reported neither a confidence interval nor a P value.

Because many readers can only, or will only, read a systematic review's abstract, clear presentation of the main results is vital. Although guidance exists for clinical trial abstracts,⁵ guidance for reporting systematic review abstracts does not, other than general guidelines in the PRISMA statement⁶ and the Cochrane Handbook for Systematic Reviews of Interventions. Guidelines for writing abstracts are needed and should include not only which items are presented, but how.

Although abstracts should present estimates of effect and confidence intervals, interpretation of the results should not require statistical knowledge. Given the high level of innumeracy among journal readers, the main results should be presented in both words and numbers. Although replication in a wider sample of journals is desirable, the apparent poor quality of systematic review abstracts deserves attention from authors, reviewers, and journal editors.

Author Contributions: Ms Beller had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Beller, Glasziou, Hopewell, Altman.

Acquisition of data: Beller, Glasziou, Hopewell.

Analysis and interpretation of data: Beller, Glasziou, Hopewell, Altman.

Drafting of the manuscript: Beller.

Critical revision of the manuscript for important intellectual content: Beller, Glasziou, Hopewell, Altman.

Statistical analysis: Beller, Glasziou, Hopewell.

Obtained funding: Beller, Glasziou, Hopewell, Altman.

Administrative, technical, or material support: Beller.

Funding/Support: This study was funded in part by the Australian National Health and Medical Research Council grant (527500). Dr Hopewell is supported by a grant from the National Institute for Health Research (United Kingdom). Dr Altman is supported by Cancer Research UK.

Role of the Sponsors: The funding sources had no role in the design and conduct of the study; in the collection, analysis, and interpretation of the data; or in the preparation, review, or approval of the manuscript.

Additional Contributions: We thank Chris Del Mar, MD, Health Sciences and Medicine, Bond University, for critical and gratis review of a draft of this article.