[Skip to Content]
[Skip to Content Landing]
Download PDF
Figure.
Reporting of Limitations of Observational Research
Reporting of Limitations of Observational Research

Reporting of limitations of observational research published in 7 major internal medicine journals (New England Journal of Medicine [NEJM], Lancet, JAMA, BMJ, PLoS Med, Annals of Internal Medicine, and JAMA Internal Medicine) from January 1, 2013, to June 30, 2013. Data are proportions of the indicated journal documents and associated news stories that mention any study limitation (dark blue bars) or contain an explicit statement that causality cannot be inferred (light blue bars). NEJM Journal Watch articles were categorized as press releases for NEJM articles.

aNumber of abstracts that report any limitation: Annals of Internal Medicine, 9 of 9; BMJ, 2 of 13; JAMA, 2 of 17; JAMA Internal Medicine, 2 of 21; PLoS Medicine, 1 of 7; NEJM, 0 of 10; and Lancet, 0 of 4. Number of abstracts that report a causality limitation: Annals of Internal Medicine, 1 of 9; BMJ, 1 of 13; JAMA, 1 of 17; JAMA Internal Medicine, 0 of 21; PLoS Medicine, 0 of 7; NEJM, 0 of 10; and Lancet, 0 of 4.

bNumber of journal press releases that report any limitation: Annals of Internal Medicine, 0 of 2; BMJ, 0 of 8; JAMA, 3 of 16; JAMA Internal Medicine, 4 of 12; PLoS Medicine, 2 of 6; NEJM, 4 of 7; and Lancet, 0 of 3. Number of journal press releases that report a causality limitation: Annals of Internal Medicine, 0 of 2; BMJ, 0 of 8; JAMA, 1 of 16; JAMA Internal Medicine, 0 of 12; PLoS Medicine, 1 of 6; NEJM, 0 of 7; and Lancet, 0 of 3.

1.
Young  SS, Karr  A.  Deming, data and observational studies: a process out of control and needing fixing. Significance. 2011;8(3):116-120. doi:10.1111/j.1740-9713.2011.00506.x.
2.
Ioannidis  JPA.  Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol. 2007;60(4):324-329.
PubMedArticle
3.
Yavchitz  A, Boutron  I, Bafeta  A,  et al.  Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med. 2012;9(9):e1001308.
PubMedArticle
4.
Prasad  V, Jorgenson  J, Ioannidis  JPA, Cifu  A.  Observational studies often make clinical practice recommendations: an empirical evaluation of authors’ attitudes. J Clin Epidemiol. 2013;66(4):361-366.e4.
PubMedArticle
5.
Sumner  P, Vivian-Griffiths  S, Boivin  J,  et al.  The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ. 2014;349:g7015.
PubMedArticle
6.
Prasad  V, Cifu  A, Ioannidis  JPA.  Reversals of established medical practices: evidence to abandon ship. JAMA. 2012;307(1):37-38.
PubMedArticle
Research Letter
September 2015

Reporting of Limitations of Observational Research

Author Affiliations
  • 1Department of Medicine, University of Auckland, Auckland, New Zealand
JAMA Intern Med. 2015;175(9):1571-1572. doi:10.1001/jamainternmed.2015.2147

Observational research is abundant and influences clinical practice, in part via publication in high-impact journals and dissemination by news media. However, it frequently generates unreliable findings.1Quiz Ref ID Inherent methodologic limitations that generate bias and confounding mean that causal inferences cannot reliably be drawn. Study limitations may be inadequately acknowledged and accompanied by disclaimers that diminish their importance.2 We assess the reporting of limitations of observational studies published in major internal medicine journals and associated news stories, specifically focusing on inference of causality.

Methods

Quiz Ref IDUsing MEDLINE, journal websites, Eurekalert!, and Factiva, we collated 81 prospective cohort and case-control studies with clinical outcomes published between January 1, 2013, and June 30, 2013, in the Annals of Internal Medicine, BMJ, JAMA, JAMA Internal Medicine, Lancet, New England Journal of Medicine, and PLoS Medicine; 48 accompanying editorials; 54 journal press releases; and 319 news stories generated within 2 months of publication. We analyzed the Abstract and Discussion sections of the source articles as separate documents. For each of the resulting 583 documents, we determined whether any study limitation was reported and whether there was an explicit statement that causality could not be inferred. If a causality limitation was reported, we determined whether it was accompanied by a disclaimer, defined as a statement that undermines or downplays the limitation. Data were extracted independently by 2 of us (M.T.M.W. and A.G.), and differences were resolved by consensus.

Results

Quiz Ref IDAny study limitation was mentioned in 70 of 81 (86%) source article Discussion sections, 26 of 48 (54%) accompanying editorials, 13 of 54 (24%) journal press releases, 16 of 81 (20%) source article abstracts (of which 9 were published in the Annals of Internal Medicine), and 61 of 319 (19%) associated news stories.Quiz Ref ID An explicit statement that causality could not be inferred was infrequently present: 8 of 81 (10%) source article Discussion sections, 7 of 48 (15%) editorials, 2 of 54 (4%) press releases, 3 of 81 (4%) source article abstracts, and 31 of 319 (10%) news stories contained such statements (Figure). Among the 51 source documents that included a causality limitation, 23 (45%) were accompanied by a disclaimer.

Of the 13 source articles that generated at least 1 news story containing a causality limitation, 8 (62%) contained the limitation in the Abstract or Discussion, editorial, or journal press release. In comparison, only 10 of 68 (15%) source articles that did not generate at least 1 news story with a causality limitation contained a causality limitation in the Abstract or Discussion, editorial, or journal press release (P < .001). However, for the 18 articles that contained the causality limitation in the Abstract or discussion, editorial, or journal press release, the median proportion of associated news stories that reported the causality limitation was only 9% (range, 0%-100%).

Among 31 news stories that reported a causality limitation, 16 (52%) included a disclaimer. Study authors were responsible for 18 (58%) of these limitation statements but also for 14 (88%) of the disclaimers.

Discussion

Limitations of observational research published in high-impact journals were infrequently mentioned in associated news stories. Inadequate acknowledgment of limitations in the journal sources might contribute to the low proportion of news stories that mentioned limitations of observational research. Limitations were rarely mentioned in the study abstracts or journal press releases, the content of which is associated with that of news stories,3 but were commonly “buried” in lengthy Discussion sections. A fundamental limitation of observational research—the inability to attribute causation—was rarely mentioned in journal documents or news stories and was often accompanied by a disclaimer. In news stories, disclaimers were frequently attributable to study investigators, consistent with evidence that authors of observational studies often make clinical practice recommendations based on their work4 and that academic press releases exaggerate research outcomes.5

A possible consequence of inadequate reporting of limitations of observational research is that readers consider the reported associations to be causal, promoting health practices based on evidence of modest quality. Quiz Ref IDUp to 50% of such practices prove ineffective when tested in randomized clinical trials.6 Giving greater prominence to the limitations of observational research, particularly in the publication abstract and journal press releases, might temper this enthusiasm and reduce the need for subsequent reversals of practice.

Back to top
Article Information

Corresponding Author: Andrew Grey, MD, Department of Medicine, University of Auckland, Private Bag 92019, Auckland, New Zealand (a.grey@auckland.ac.nz).

Published Online: June 8, 2015. doi:10.1001/jamainternmed.2015.2147.

Author Contributions: Dr Grey had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Grey.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Wang.

Study supervision: Grey.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study received funding support from the Health Research Council of New Zealand and the University of Auckland.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.

Additional Contributions: Greg Gamble, MSc, University of Auckland, provided advice about data presentation and analysis.

References
1.
Young  SS, Karr  A.  Deming, data and observational studies: a process out of control and needing fixing. Significance. 2011;8(3):116-120. doi:10.1111/j.1740-9713.2011.00506.x.
2.
Ioannidis  JPA.  Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol. 2007;60(4):324-329.
PubMedArticle
3.
Yavchitz  A, Boutron  I, Bafeta  A,  et al.  Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med. 2012;9(9):e1001308.
PubMedArticle
4.
Prasad  V, Jorgenson  J, Ioannidis  JPA, Cifu  A.  Observational studies often make clinical practice recommendations: an empirical evaluation of authors’ attitudes. J Clin Epidemiol. 2013;66(4):361-366.e4.
PubMedArticle
5.
Sumner  P, Vivian-Griffiths  S, Boivin  J,  et al.  The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ. 2014;349:g7015.
PubMedArticle
6.
Prasad  V, Cifu  A, Ioannidis  JPA.  Reversals of established medical practices: evidence to abandon ship. JAMA. 2012;307(1):37-38.
PubMedArticle
×