Systematic reviews are the highest level of evidence in otolaryngology clinical practice guidelines.1 However, they generally present only formally published data, which may lead to an inherent problem caused by publication bias, that is, a strong bias to publish only studies that show significant results.2,3 This bias can be mitigated through both the search methods selected and the statistical methods used.
In the present study, we address the following 4 specific research topics: (1) the techniques systematic reviewers used to mitigate publication bias during the search process; (2) the statistical methods used to evaluate the existence of publication bias during data synthesis; (3) whether a difference existed in the frequency of publication bias evaluation among systematic reviews that did or did not endorse the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline; and (4) for systematic reviews not reporting a publication bias evaluation, whether there was sufficient information contained within the published report to conduct an independent evaluation, and if so, the findings of such evaluations.
The following journals were identified through the use of Google Scholar h5-index scores: American Journal of Otolaryngology, Otolaryngology–Head and Neck Surgery, JAMA Otolaryngology–Head & Neck Surgery, International Journal of Otolaryngology, Clinical Otolaryngology, Journal of Otolaryngology–Head and Neck Surgery, and The Laryngoscope. A PubMed search was conducted on October 26, 2016, to identify all relevant systematic reviews within these top-ranked otolaryngology journals. To be eligible for inclusion as a systematic review, an article must have reported summarized evidence across numerous studies and must have provided information on the search strategy, such as search terms, databases, and inclusion and exclusion criteria. Specific elements were retrieved from each review, including the title, authors, year of publication, journal name, type of systematic review or meta-analysis, primary study type, whether the systematic reviewers manually searched reference lists, methods used to evaluate publication bias, and additional elements, which are given in the Table.
After data extraction, evidence for publication bias was assessed among the systematic reviews that contained at least 1 meta-analysis with 10 or more primary studies that did not evaluate for publication bias. The following methods were used in the present study to assess publication bias: the Egger regression test,4 Duval and Tweedie trim-and-fill method,5 and Begg rank correlation test.6 Data were analyzed using Stata, version 13.1 (StataCorp) and Comprehensive Meta-Analysis (Biostat).
In total, 324 systematic reviews were included in the present study. Extracted data elements are given in the Table. Of the 324 reviews, 67 (21%) searched the gray literature and 145 (45%) mentioned the use of a reporting guideline. Of these 145 reviews, 133 (92%) used the PRISMA reporting guideline. Only 69 of the 324 reviews (22%) could appropriately evaluate for publication bias (they contained at least 1 meta-analysis with at least 10 primary studies). Of these 69 reviews, 38 (55%) formally assessed for publication bias. There was no significant association between the evaluation of publication bias and the use of the PRISMA reporting guideline (odds ratio, 1.67; 95% CI, 0.58-4.85). The most common method used by these 38 systematic reviews for evaluating publication bias was visualization of a funnel plot (35 [92%]). Only 9 of the 31 eligible reviews that did not assess for publication bias contained sufficient data for analysis. The present study found that all 9 of these reviews likely had publication bias. These 9 reviews included 22 meta-analyses. Using the Duval and Tweedie trim-and-fill test, we found that 18 of the 22 meta-analyses had evidence of publication bias. Using the Egger regression test, Duval and Tweedie trim-and-fill method, and Begg rank correlation test, we found that 10 of the meta-analyses showed evidence of publication bias.
Our findings indicate that researchers who conducted systematic reviews that were published in high-ranking otolaryngology journals did not often mention, plan for, or formally evaluate for the presence of publication bias. This conclusion was reached on the basis of 3 findings: most of the SRs included in the present study did not search the gray literature, did not report the use of a reporting guideline, and did not assess for publication bias. The limitation to this study was that we restricted our search to high-ranking journals in otolaryngology. Therefore, our findings may not be applicable to lower-ranking journals in the field. Publication bias appears to be a pervasive problem in medical research, and interventions may be needed to encourage the publication of high-quality research regardless of the strength or direction of the study findings.
Accepted for Publication: September 23, 2018.
Corresponding Author: Andrew Ross, BS, Oklahoma State University Center for Health Sciences, 1111 W 17th St, Tulsa, OK 74107 (aeross@okstate.edu).
Published Online: December 6, 2018. doi:10.1001/jamaoto.2018.3301
Author Contributions: Mr Ross and Dr Vassar had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Umberham, Vassar.
Acquisition, analysis, or interpretation of data: Ross, Cooper, Gray, Umberham.
Drafting of the manuscript: Ross, Cooper, Gray, Umberham.
Critical revision of the manuscript for important intellectual content: Ross, Vassar.
Statistical analysis: Ross, Gray, Umberham.
Administrative, technical, or material support: Ross.
Supervision: Umberham, Vassar.
Conflict of Interest Disclosures: None reported.