[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.161.130.145. Please contact the publisher to request reinstatement.
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Download PDF
Figure. Flow Diagram of Selection of Systematic Reviews and Primary Results of Analysis
Figure. Flow Diagram of Selection of Systematic Reviews and Primary Results of Analysis

NLM indicates National Library of Medicine.

Table. Reporting of Effect Size in Numerical Format and Measures of Statistical Uncertainty (N=182)
Table. Reporting of Effect Size in Numerical Format and Measures of Statistical Uncertainty (N=182)
1.
Lai NM, Teng CL, Lee ML. Interpreting systematic reviews: are we ready to make our own conclusions? a cross-sectional study.  BMC Med. 2011;9:30PubMedArticle
2.
 The Cochrane Library. Cochrane Collaboration. http://www.thecochranelibrary.com/view/0/index.html. Accessed February 21, 2011
3.
Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews.  PLoS Med. 2007;4(3):e78PubMedArticleArticle
4.
 Abridged Index Medicus (AIM or “Core Clinical”) journal titles. US National Library of Medicine. http://www.nlm.nih.gov/bsd/aim.html. Accessed February 21, 2011
5.
Hopewell S, Clarke M, Moher D,  et al; CONSORT Group.  CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration.  PLoS Med. 2008;5(1):e20PubMedArticleArticle
6.
Liberati A, Altman DG, Tetzlaff J,  et al.  The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.  PLoS Med. 2009;6(7):e1000100PubMedArticleArticle
Research Letter
November 9, 2011

Reporting of Effect Direction and Size in Abstracts of Systematic Reviews

Author Affiliations

Author Affiliations: Centre for Research in Evidence-Based Practice, Bond University, Gold Coast, Australia (Ms Beller and Dr Glasziou) (ebeller@bond.edu.au); and Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (Drs Hopewell and Altman).

JAMA. 2011;306(18):1981-1982. doi:10.1001/jama.2011.1620

To the Editor: Clinicians commonly misinterpret systematic review abstracts: a recent study showed many arrived at incorrect conclusions, and only 62% correctly identified the direction of the main effect.1

Interpreting numerical results requires statistical knowledge that many clinicians lack. To ensure correct interpretation, abstracts should give the direction and size of effects both in words and numerically. Because systematic reviews are important and widely used summaries of primary research, we decided to examine a sample of systematic review abstracts to assess the nature and extent of any deficiencies in reporting.

Methods

The systematic reviews selected were all new reviews of interventions published in issue 4, 2009, of the Cochrane Library2 and from a search (based on Moher et al3) of the National Library of Medicine's 119 Core Clinical Journals (Abridged Index Medicus)4 during 2009. Reviews were eligible if they included 1 or more meta-analyses comparing interventions. Eligibility was broad, as there is no widely agreed definition of a systematic review. We used a definition by Moher et al3: “ . . . the authors' stated objective was to summarize evidence from multiple studies, and the article described explicit methods, regardless of the details provided.”

One author (E.M.B.) screened all titles and abstracts with a second author (S.H.) independently checking citations classified as possible systematic reviews based on full-text review. All Cochrane reviews and a random sample of eligible non-Cochrane reviews were selected for data extraction.

From abstracts, 2 authors (E.M.B., P.P.G.) independently extracted the description in words of the direction and size of effect, statistical significance, numerical estimates of effect size (relative and absolute), P value, and confidence interval.

Results

We included 64 Cochrane and 125 of 275 non-Cochrane systematic reviews (Figure); 7 of the non-Cochrane reviews did not include a meta-analysis, leaving 182 abstracts.

Of 182 abstracts, 77 (42%) did not describe the direction of intervention effect in words (Figure), and only 22 (12%) described the size in words. In 34 (19%), the direction of effect could only be determined by interpreting numerical results.

In 43 (24%) of the abstracts, we could not reliably determine the direction of effect (Figure). Sometimes a risk or odds ratio was stated, and if it was less than 1, the direction might be assumed to favor the intervention (for example, “The pooled relative risk was 0.62 [95% CI, 0.45 to 0.86] . . . ”). However, we classified these instances as ambiguous.

Although all included reviews contained a meta-analysis, 45 (25%) gave no numerical effect measure and 44 (24%) gave no measure of uncertainty (Table).

Besides Cochrane reviews less frequently providing P values (6% vs 28%), reporting of effects in Cochrane and non-Cochrane reviews were similar.

Comment

For 42% of abstracts of systematic reviews, the direction of the main effect either could not be determined or needed to be inferred. Statistical uncertainty was also poorly reported: 24% of abstracts reported neither a confidence interval nor a P value.

Because many readers can only, or will only, read a systematic review's abstract, clear presentation of the main results is vital. Although guidance exists for clinical trial abstracts,5 guidance for reporting systematic review abstracts does not, other than general guidelines in the PRISMA statement6 and the Cochrane Handbook for Systematic Reviews of Interventions. Guidelines for writing abstracts are needed and should include not only which items are presented, but how.

Although abstracts should present estimates of effect and confidence intervals, interpretation of the results should not require statistical knowledge. Given the high level of innumeracy among journal readers, the main results should be presented in both words and numbers. Although replication in a wider sample of journals is desirable, the apparent poor quality of systematic review abstracts deserves attention from authors, reviewers, and journal editors.

Back to top
Article Information

Author Contributions: Ms Beller had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Beller, Glasziou, Hopewell, Altman.

Acquisition of data: Beller, Glasziou, Hopewell.

Analysis and interpretation of data: Beller, Glasziou, Hopewell, Altman.

Drafting of the manuscript: Beller.

Critical revision of the manuscript for important intellectual content: Beller, Glasziou, Hopewell, Altman.

Statistical analysis: Beller, Glasziou, Hopewell.

Obtained funding: Beller, Glasziou, Hopewell, Altman.

Administrative, technical, or material support: Beller.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: This study was funded in part by the Australian National Health and Medical Research Council grant (527500). Dr Hopewell is supported by a grant from the National Institute for Health Research (United Kingdom). Dr Altman is supported by Cancer Research UK.

Role of the Sponsors: The funding sources had no role in the design and conduct of the study; in the collection, analysis, and interpretation of the data; or in the preparation, review, or approval of the manuscript.

Additional Contributions: We thank Chris Del Mar, MD, Health Sciences and Medicine, Bond University, for critical and gratis review of a draft of this article.

References
1.
Lai NM, Teng CL, Lee ML. Interpreting systematic reviews: are we ready to make our own conclusions? a cross-sectional study.  BMC Med. 2011;9:30PubMedArticle
2.
 The Cochrane Library. Cochrane Collaboration. http://www.thecochranelibrary.com/view/0/index.html. Accessed February 21, 2011
3.
Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews.  PLoS Med. 2007;4(3):e78PubMedArticleArticle
4.
 Abridged Index Medicus (AIM or “Core Clinical”) journal titles. US National Library of Medicine. http://www.nlm.nih.gov/bsd/aim.html. Accessed February 21, 2011
5.
Hopewell S, Clarke M, Moher D,  et al; CONSORT Group.  CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration.  PLoS Med. 2008;5(1):e20PubMedArticleArticle
6.
Liberati A, Altman DG, Tetzlaff J,  et al.  The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.  PLoS Med. 2009;6(7):e1000100PubMedArticleArticle
×