Reporting of Limitations of Observational Research

Michael T. M. Wang; Mark J. Bolland; Andrew Grey

doi:10.1001/jamainternmed.2015.2147

Comment

Figure. Reporting of Limitations of Observational Research

View Large Download

Reporting of limitations of observational research published in 7 major internal medicine journals (New England Journal of Medicine [NEJM], Lancet, JAMA, BMJ, PLoS Med, Annals of Internal Medicine, and JAMA Internal Medicine) from January 1, 2013, to June 30, 2013. Data are proportions of the indicated journal documents and associated news stories that mention any study limitation (dark blue bars) or contain an explicit statement that causality cannot be inferred (light blue bars). NEJM Journal Watch articles were categorized as press releases for NEJM articles.

^aNumber of abstracts that report any limitation: Annals of Internal Medicine, 9 of 9; BMJ, 2 of 13; JAMA, 2 of 17; JAMA Internal Medicine, 2 of 21; PLoS Medicine, 1 of 7; NEJM, 0 of 10; and Lancet, 0 of 4. Number of abstracts that report a causality limitation: Annals of Internal Medicine, 1 of 9; BMJ, 1 of 13; JAMA, 1 of 17; JAMA Internal Medicine, 0 of 21; PLoS Medicine, 0 of 7; NEJM, 0 of 10; and Lancet, 0 of 4.

^bNumber of journal press releases that report any limitation: Annals of Internal Medicine, 0 of 2; BMJ, 0 of 8; JAMA, 3 of 16; JAMA Internal Medicine, 4 of 12; PLoS Medicine, 2 of 6; NEJM, 4 of 7; and Lancet, 0 of 3. Number of journal press releases that report a causality limitation: Annals of Internal Medicine, 0 of 2; BMJ, 0 of 8; JAMA, 1 of 16; JAMA Internal Medicine, 0 of 12; PLoS Medicine, 1 of 6; NEJM, 0 of 7; and Lancet, 0 of 3.

Young SS, Karr A. Deming, data and observational studies: a process out of control and needing fixing. Significance. 2011;8(3):116-120. doi:10.1111/j.1740-9713.2011.00506.x.Google Scholar

Ioannidis JPA. Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol. 2007;60(4):324-329.PubMed Google Scholar Crossref

Yavchitz A, Boutron I, Bafeta A, et al. Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med. 2012;9(9):e1001308.PubMed Google Scholar Crossref

Prasad V, Jorgenson J, Ioannidis JPA, Cifu A. Observational studies often make clinical practice recommendations: an empirical evaluation of authors’ attitudes. J Clin Epidemiol. 2013;66(4):361-366.e4.PubMed Google Scholar Crossref

Sumner P, Vivian-Griffiths S, Boivin J, et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ. 2014;349:g7015.PubMed Google Scholar Crossref

Prasad V, Cifu A, Ioannidis JPA. Reversals of established medical practices: evidence to abandon ship. JAMA. 2012;307(1):37-38.PubMed Google Scholar Crossref

Select Your Interests

Save Preferences

Others Also Liked

1 Comment for this article

EXPAND ALL

July 7, 2015

Response to Limitations of Observational Research

Sunder M. Lal | Boone Hospital Center, Columbia, MO

In general, I concur with the research letter by Wang et al (1). Observational studies from established investigators are usually published in high-impact journals. However, important clinical observations and/or results from randomized clinical trials conducted by lesser known clinical researchers are frequently rejected by scientific journals. Some of our observations, which later turned out to have clinical impact, are being presented.
(I). In early 1991, while preparing for a presentation on newer immunosuppressive drugs, it became apparent that Mycophenolate Mofetil (MM) exhibited specific effects on lymphocytes. Its potential role in lupus nephritis was discussed with a colleague, and he conducted research experiment in a mouse model, and mycophenolate was found to be effective in the treatment of lupus nephritis in that animal model (2). Thereafter, clinical trials have confirmed MM to be an effective agent in treating lupus nephritis.
(II). Several randomized trials had shown superiority of mycophenolate mofetil compared to azathioprine or a placebo treated kidney transplant patients (3-4). My colleagues and I conducted a prospective randomized clinical trial comparing safety and efficacy of mycophenolate mofetil and azathioprine in renal transplant recipients. In a small group of patients, our conclusions did not support superiority of MM over azathioprine treated patients. Therefore, our abstracts were published but not accepted for presentation in scientific meetings (5-6). Although, our results were subsequently confirmed in both short and long term, neither the researchers nor the editorial board of those journals credited us as original researchers on that subject (7-8). One of those journals has championed the concept of “Research In Context,” where researchers are now required to relate their results to previous research findings (7).
(III). Our manuscript describing the presence of IgM deposits in a transplanted kidney was rejected because the reviewers of that journal assumed that the deposits were a result of non specific trappings. A decade later, Gough et al (9), observed presence of IgM deposits in 7.6% of renal transplant biopsies.
(IV). In kidney transplant patients, compared to other Statins (lovastatin, simvastatin, fluvastatin, pravastatin) we found atorvastatin was most effective in attaining the NCEP targeted LDL-cholesterol values [33% vs 67%]. Those data were presented at a grand round in 1995. Almost nine years later, atorvastatin was shown to be superior to pravastatin in attaining a LDL level <70 mg/dl in the general population (10).
In the prevailing research environment, it is challenging for a practicing clinician to interpret the available data and treat their patients due to major drawbacks in the design and conduct of prospective randomized multi-center trials. The studies are frequently short term, the inclusion criteria are restrictive, and data on comparative effectiveness [for example with various Statin/ACE/ARB/CCB] are lacking.
References:
1).Wang MTM, Bolland MJ, Grey A: Reporting of Limitations of Observational Research. JAMA Internal Medicine 215; online June 6: E1-E2.

2). McMurray RW, Elbourne KB, Lagoo A, Lal S. Mycophenolate mofetil suppresses autoimmunity and mortality in the female NZB x NZW F1 mouse model of systemic lupus erythematosus. J Rheumatol 1998;25:2364-70

3). Sollinger HW: Mycophenolate mofetil for the prevention of acute rejection in primary cadaveric renal allograft recipients. U.S. Renal Transplant Mycophenolate Mofetil Study Group. Transplantation 1995;60:225-32.2).

4). European Mycophenolate Mofetil Cooperative Study Group. Placebo-controlled study of mycophenolate mofetil combined with cyclosporin and corticosteroids for prevention of acute rejection. Lancet 1995; 345: 1321–25.

5). Lal SM, Gupta,Nilakshi, Habib S : Effects of Mycophenolate based immunosuppression on renal function in cadaveric renal transplant recipients. J Am Soc Nephrol 1998;(9):page 683A

6). Lal SM, Habib S, Samuel J: Comparative effects of Azathioprine and Mycophenolate (Short vs Long term) based immunosuppression in cadaveric renal transplant patients. Am Soc for Artif Int Organs J, 1999;45(2):177

7). Remuzzi G, Lesti M, Gotti E, Ganeva M, Dimitrov BD, Ene-Iordache B, et al. Mycophenolate mofetil versus azathioprine for prevention of acute rejection in renal transplantation (MYSS): a randomized trial. Lancet 2004;364:503-12.

8). Remuzzi G, Cravedi P, Costantini M, Lesti M, Ganeva M, Gherardi G, et al. Mycophenolate mofetil versus azathioprine for prevention of chronic allograft dysfunction in renal transplantation: the MYSS follow-up randomized, controlled clinical trial. J Am Soc Nephrol 2007;18:1973-85.

9). Gough J, Yilmaz A, Yilmaz S, Benediktsson H. Recurrent and de novo glomerular immune-complex deposits in renal transplant biopsies. Arch Pathol Lab Med 2005. 129:231-233.

10). Cannon CP, Braunwald E, McCabe CH, Rader DJ, Rouleau JL, Belder R, Joyal SV, Hill KA, Pfeffer MA, and Skene AM, for the Pravastatin or Atorvastatin Evaluation and Infection Therapy–Thrombolysis in Myocardial Infarction 22 Investigators. Intensive versus Moderate Lipid Lowering with Statins after Acute Coronary Syndromes. N Engl J Med 2004; 350:1495-1504.

CONFLICT OF INTEREST: None Reported

Research Letter

September 2015

Reporting of Limitations of Observational Research

Michael T. M. Wang¹; Mark J. Bolland, MBChB, PhD¹; Andrew Grey, MD¹

Author Affiliations Article Information

¹Department of Medicine, University of Auckland, Auckland, New Zealand

JAMA Intern Med. 2015;175(9):1571-1572. doi:10.1001/jamainternmed.2015.2147

Observational research is abundant and influences clinical practice, in part via publication in high-impact journals and dissemination by news media. However, it frequently generates unreliable findings.¹Quiz Ref ID Inherent methodologic limitations that generate bias and confounding mean that causal inferences cannot reliably be drawn. Study limitations may be inadequately acknowledged and accompanied by disclaimers that diminish their importance.² We assess the reporting of limitations of observational studies published in major internal medicine journals and associated news stories, specifically focusing on inference of causality.

Methods

Quiz Ref IDUsing MEDLINE, journal websites, Eurekalert!, and Factiva, we collated 81 prospective cohort and case-control studies with clinical outcomes published between January 1, 2013, and June 30, 2013, in the Annals of Internal Medicine, BMJ, JAMA, JAMA Internal Medicine, Lancet, New England Journal of Medicine, and PLoS Medicine; 48 accompanying editorials; 54 journal press releases; and 319 news stories generated within 2 months of publication. We analyzed the Abstract and Discussion sections of the source articles as separate documents. For each of the resulting 583 documents, we determined whether any study limitation was reported and whether there was an explicit statement that causality could not be inferred. If a causality limitation was reported, we determined whether it was accompanied by a disclaimer, defined as a statement that undermines or downplays the limitation. Data were extracted independently by 2 of us (M.T.M.W. and A.G.), and differences were resolved by consensus.

Results

Quiz Ref IDAny study limitation was mentioned in 70 of 81 (86%) source article Discussion sections, 26 of 48 (54%) accompanying editorials, 13 of 54 (24%) journal press releases, 16 of 81 (20%) source article abstracts (of which 9 were published in the Annals of Internal Medicine), and 61 of 319 (19%) associated news stories.Quiz Ref ID An explicit statement that causality could not be inferred was infrequently present: 8 of 81 (10%) source article Discussion sections, 7 of 48 (15%) editorials, 2 of 54 (4%) press releases, 3 of 81 (4%) source article abstracts, and 31 of 319 (10%) news stories contained such statements (Figure). Among the 51 source documents that included a causality limitation, 23 (45%) were accompanied by a disclaimer.

Of the 13 source articles that generated at least 1 news story containing a causality limitation, 8 (62%) contained the limitation in the Abstract or Discussion, editorial, or journal press release. In comparison, only 10 of 68 (15%) source articles that did not generate at least 1 news story with a causality limitation contained a causality limitation in the Abstract or Discussion, editorial, or journal press release (P < .001). However, for the 18 articles that contained the causality limitation in the Abstract or discussion, editorial, or journal press release, the median proportion of associated news stories that reported the causality limitation was only 9% (range, 0%-100%).

Among 31 news stories that reported a causality limitation, 16 (52%) included a disclaimer. Study authors were responsible for 18 (58%) of these limitation statements but also for 14 (88%) of the disclaimers.

Discussion

Limitations of observational research published in high-impact journals were infrequently mentioned in associated news stories. Inadequate acknowledgment of limitations in the journal sources might contribute to the low proportion of news stories that mentioned limitations of observational research. Limitations were rarely mentioned in the study abstracts or journal press releases, the content of which is associated with that of news stories,³ but were commonly “buried” in lengthy Discussion sections. A fundamental limitation of observational research—the inability to attribute causation—was rarely mentioned in journal documents or news stories and was often accompanied by a disclaimer. In news stories, disclaimers were frequently attributable to study investigators, consistent with evidence that authors of observational studies often make clinical practice recommendations based on their work⁴ and that academic press releases exaggerate research outcomes.⁵

A possible consequence of inadequate reporting of limitations of observational research is that readers consider the reported associations to be causal, promoting health practices based on evidence of modest quality. Quiz Ref IDUp to 50% of such practices prove ineffective when tested in randomized clinical trials.⁶ Giving greater prominence to the limitations of observational research, particularly in the publication abstract and journal press releases, might temper this enthusiasm and reduce the need for subsequent reversals of practice.

Article Information

Corresponding Author: Andrew Grey, MD, Department of Medicine, University of Auckland, Private Bag 92019, Auckland, New Zealand (a.grey@auckland.ac.nz).

Published Online: June 8, 2015. doi:10.1001/jamainternmed.2015.2147.

Author Contributions: Dr Grey had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Grey.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Wang.

Study supervision: Grey.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study received funding support from the Health Research Council of New Zealand and the University of Auckland.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.

Additional Contributions: Greg Gamble, MSc, University of Auckland, provided advice about data presentation and analysis.

References

Young SS, Karr A. Deming, data and observational studies: a process out of control and needing fixing. Significance. 2011;8(3):116-120. doi:10.1111/j.1740-9713.2011.00506.x.Google Scholar

Ioannidis JPA. Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol. 2007;60(4):324-329.PubMed Google Scholar Crossref

Prasad V, Cifu A, Ioannidis JPA. Reversals of established medical practices: evidence to abandon ship. JAMA. 2012;307(1):37-38.PubMed Google Scholar Crossref

Reporting of Limitations of Observational Research

Featured Articles

USPSTF Recommendation Statements

Blogs

See More About

Select Your Interests

Select Your Interests

Others Also Liked

Citation

Manage citations:

Reporting of Limitations of Observational Research